Text clustering method, apparatus, and electronic device

Abstract

The application discloses a text clustering method and device and electronic equipment. It is related to the field of financial technology or other fields, and the method comprises the following steps: obtaining a plurality of digital vectors of a to-be-processed text, wherein each digital vector corresponds to part of the to-be-processed text; determining a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is the maximum limit value of the clustering range, and the second distance threshold is the minimum limit value of the clustering range; performing first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold to obtain a clustering result; obtaining the number of clusters in the clustering result; performing second clustering processing on the plurality of digital vectors based on the number of clusters to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector. The application solves the technical problem of poor text clustering effect caused by the fact that the number of clusters cannot be accurately determined in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology, in particular to a text clustering method and device and electronic equipment. BACKGROUND

[0002] Automatic clustering of text is a key technology in classification processing of text data and natural language processing, and is also an important application in automatic classification of text literature. With the explosive development of modern information technology, product reviews have important research value, which can not only help potential consumers to evaluate product quality and make wise purchase decisions, but also provide feedback information for businesses as important electronic testimonials. In the face of massive text, it is of great significance to efficiently obtain similar text and classify it.

[0003] At present, the related clustering algorithm (for example, K-means clustering algorithm) for text clustering often needs to specify the number of clusters in advance when clustering. In the existing related technology, the selection of this value is generally based on empirical values, which is prone to inaccurate judgment, resulting in a large error in clustering, and even falling into local optimization.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a text clustering method, device and electronic equipment to at least solve the technical problem of poor text clustering effect caused by the inability to accurately determine the number of clusters in the prior art.

[0006] According to an aspect of the embodiments of the present application, a text clustering method is provided, comprising: obtaining a plurality of digital vectors of a to-be-processed text, wherein each digital vector corresponds to part of the to-be-processed text; determining a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is the maximum limit value of the clustering range, and the second distance threshold is the minimum limit value of the clustering range; performing first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold to obtain a clustering result; obtaining the number of clusters in the clustering result; performing second clustering processing on the plurality of digital vectors based on the number of clusters to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector.

[0007] Further, the text clustering method further comprises: calculating the ratio of the vector sum of the plurality of digital vectors to the number of vectors corresponding to the plurality of digital vectors to obtain a mean vector; determining the farthest distance and the nearest distance between the plurality of digital vectors and the mean vector; determining the first distance threshold and the second distance threshold based on the farthest distance and the nearest distance.

[0008] Further, the text clustering method further comprises: calculating a difference between the farthest distance and the nearest distance; in a case that a ratio of the difference to the farthest distance is greater than a first preset value, determining the difference as the first distance threshold and a product of the farthest distance and the first preset value as the second distance threshold; in a case that the ratio of the difference to the farthest distance is less than or equal to the first preset value, determining a product of the nearest distance and a second preset value as the first distance threshold and a product of the farthest distance and the first preset value as the second distance threshold.

[0009] Further, the text clustering method further comprises: obtaining at least one center vector from a center list, wherein the center list is used to store the at least one center vector, and each center vector corresponds to a digital vector in the plurality of digital vectors; selecting a target digital vector in a vector list and determining distances between the target digital vector and each center vector, wherein the vector list is used to store the plurality of digital vectors, and the target digital vector is a digital vector different from any center vector; determining whether the target digital vector and each center vector belong to different clusters based on the distances between the target digital vector and each center vector, the first distance threshold and the second distance threshold; in a case that the target digital vector and each center vector belong to different clusters, updating the center list based on the target digital vector and removing the target digital vector from the vector list; in a case that the target digital vector and any center vector belong to the same cluster, adding the target digital vector to the corresponding cluster and removing the target digital vector from the vector list.

[0010] Further, the text clustering method further comprises: after determining whether the target digital vector and each center vector belong to different clusters based on the distances between the target digital vector and each center vector, the first distance threshold and the second distance threshold, in a case that the target digital vector is in a pending state, adding the target digital vector to the corresponding cluster and retaining the target digital vector in the vector list, wherein the pending state represents that the distance between the target digital vector and any center vector is between the first distance threshold and the second distance threshold.

[0011] Further, the text clustering method further comprises: before obtaining the at least one center vector from the center list, detecting the center vectors in the center list to obtain a detection result; in a case that the detection result represents that there is no center vector in the center list, selecting a first digital vector from the plurality of digital vectors as the center vector.

[0012] Further, the text clustering method further comprises: determining a vector quantity of the target digital vectors in the pending state among the preset quantity of target digital vectors after determining whether the target digital vector and each center vector belong to different clusters based on the distance between the target digital vector and each center vector, the first distance threshold and the second distance threshold; and adjusting the second distance threshold to a target second distance threshold in a case where the vector quantity of the target digital vectors in the pending state is greater than a first preset threshold, wherein the target second distance threshold is greater than the second distance threshold.

[0013] Further, the text clustering method further comprises: determining a plurality of initial centroid vectors based on the quantity of clusters and the plurality of digital vectors; performing first mapping processing on each digital vector in the plurality of digital vectors based on the plurality of initial centroid vectors to obtain a corresponding relationship between each digital vector and an initial centroid vector; determining a digital vector corresponding to each initial centroid vector based on the corresponding relationship; and performing first reduction processing on the digital vector corresponding to each initial centroid vector to obtain a target centroid vector corresponding to each initial centroid vector.

[0014] Further, the text clustering method further comprises: determining a plurality of second digital vectors from the plurality of digital vectors based on the quantity of clusters, wherein a vector quantity of the plurality of second digital vectors is associated with the quantity of clusters; performing cosine similarity calculation on the plurality of second digital vectors to obtain a cosine angle; and determining a plurality of initial centroid vectors based on the plurality of second digital vectors, the quantity of clusters and the cosine angle, wherein a vector quantity of the initial centroid vectors is the same as the quantity of clusters.

[0015] Further, the text clustering method further comprises: determining an error vector between each initial centroid vector and a target centroid vector corresponding thereto after obtaining the plurality of target centroid vectors; determining a maximum error vector from the error vectors; and updating the initial centroid vector based on the target centroid vector to obtain a target initial centroid vector in a case where the maximum error vector is greater than a second preset threshold.

[0016] Further, the text clustering method further comprises: dividing the text to be processed into a plurality of subtexts; performing second mapping processing and second reduction processing on each subtext to obtain a plurality of segmented words; and performing feature extraction processing on the plurality of segmented words to obtain a digital vector corresponding to each subtext.

[0017] According to another aspect of the embodiments of the present application, there is also provided a text clustering apparatus, comprising: a first obtaining module configured to obtain a plurality of digital vectors of a text to be processed, wherein each digital vector corresponds to a part of the text to be processed; a determining module configured to determine a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is a maximum limit value of a clustering range, and the second distance threshold is a minimum limit value of the clustering range; a first processing module configured to perform a first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold, to obtain a clustering result; a second obtaining module configured to obtain a number of clusters in the clustering result; and a second processing module configured to perform a second clustering processing on the plurality of digital vectors based on the number of clusters, to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents a feature of the cluster corresponding to the target centroid vector.

[0018] According to another aspect of the embodiments of the present application, there is also provided a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is configured to execute the text clustering method when the computer program is run.

[0019] According to another aspect of the embodiments of the present application, there is also provided an electronic device, comprising one or more processors; and a memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a program for running, wherein the program is configured to execute the text clustering method when the program is run.

[0020] According to another aspect of the embodiments of the present application, there is also provided a computer program product, wherein the computer program / instruction is executed by a processor to implement the text clustering method.

[0021] In the embodiments of the present application, the limit value of the clustering range is determined based on the plurality of digital vectors, and then the number of clusters is determined based on the limit value of the clustering range. The plurality of digital vectors of the text to be processed are obtained, and then the first distance threshold and the second distance threshold are determined based on the plurality of digital vectors. Then, the first clustering processing is performed on the plurality of digital vectors based on the first distance threshold and the second distance threshold, to obtain the clustering result. The number of clusters in the clustering result is obtained, and then the second clustering processing is performed on the plurality of digital vectors based on the number of clusters, to obtain the target centroid vector corresponding to each cluster. Each digital vector corresponds to a part of the text to be processed. The first distance threshold is the maximum limit value of the clustering range, and the second distance threshold is the minimum limit value of the clustering range. The target centroid vector represents the feature of the cluster corresponding to the target centroid vector.

[0022] In the foregoing process, the maximum limit value and the minimum limit value of the clustering range are determined based on the plurality of digital vectors, accurate determination of the maximum limit value and the minimum limit value of the clustering range is achieved, and the problem of inaccurate judgment caused by manual setting of the limit value of the clustering range is avoided. Therefore, the first clustering processing of the digital vectors based on the foregoing first distance threshold and the second distance threshold can obtain a more accurate number of clusters, the problem of inaccurate judgment caused by manual setting of the number of clusters is avoided, and a more accurate target centroid vector can be obtained, thereby achieving a better text clustering effect.

[0023] Therefore, the scheme provided in the present application achieves the purpose of determining the limit value of the clustering range based on the plurality of digital vectors, and then determining the number of clusters based on the limit value of the clustering range, thereby achieving the technical effect of improving the text clustering effect, and further solving the technical problem of poor text clustering effect caused by the inability to accurately determine the number of clusters in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application in any way. In the drawings:

[0025] Figure 1 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of an optional text clustering method according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of an optional text clustering method according to an embodiment of the present application; DETAILED DESCRIPTION

[0032] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work shall fall into the protection scope of the present application.

[0033] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0035] Embodiment 1

[0036] According to the embodiments of the present application, an embodiment of a text clustering method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that herein.

[0037] Figure 1 is a schematic diagram of an optional text clustering method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0038] Step S101, obtaining a plurality of digital vectors of the text to be processed, wherein each digital vector corresponds to a part of the text to be processed.

[0039] In step S101, a plurality of digital vectors of the to-be-processed text can be acquired by an electronic device, an application system, a processor or the like. In this embodiment, the plurality of digital vectors of the to-be-processed text are acquired by a text clustering system. The text clustering system can directly acquire the digital vectors from a storage device, or can acquire the to-be-processed text, and then extract features of the to-be-processed text to obtain the digital vectors. The to-be-processed text is a collection of multiple texts. Each text can correspond to a line of text or a paragraph of text in the to-be-processed text. Each digital vector can correspond to one text or multiple texts in the to-be-processed text. In this embodiment, the to-be-processed text includes multiple lines of text. Each line of text corresponds to one text. Each digital vector corresponds to one line of text, that is, one text. The digital vector is data that can be recognized by a computer or the like, and is used to represent the features of each text.

[0040] It should be noted that the plurality of digital vectors of the to-be-processed text are acquired, so that the features of the to-be-processed text can be acquired by the related device, and subsequent text clustering can be implemented.

[0041] In step S102, a first distance threshold and a second distance threshold are determined based on the plurality of digital vectors. The first distance threshold is a maximum limit value of a clustering range, and the second distance threshold is a minimum limit value of the clustering range.

[0042] Optionally, the text clustering system can determine the number of clusters based on an improved Canopy clustering algorithm, so as to avoid the problem of easy error judgment caused by determining the number of clusters based on an empirical value. In step S102, the text clustering system can determine the first distance threshold and the second distance threshold in the improved Canopy clustering algorithm based on the plurality of digital vectors. The first distance threshold is greater than the second distance threshold. The first distance threshold can be used to determine whether a digital vector does not belong to a certain cluster. The second distance threshold can be used to determine whether a digital vector belongs to a certain cluster.

[0043] It should be noted that the first distance threshold and the second distance threshold are determined based on the plurality of digital vectors, so that the first distance threshold and the second distance threshold can be accurately determined, and better clustering effect can be achieved in subsequent first clustering processing, so that more accurate clustering results can be obtained.

[0044] In step S103, the first clustering processing is performed on the plurality of digital vectors based on the first distance threshold and the second distance threshold, and a clustering result is obtained.

[0045] In step S103, the text clustering system can cluster the plurality of digital vectors based on the first distance threshold and the second distance threshold by using the improved Canopy clustering algorithm to obtain a clustering result, wherein the clustering result is used to represent the clusters to which the digital vectors belong, and the digital vectors corresponding to the same cluster have high text similarity.

[0046] It should be noted that the first clustering processing of the plurality of digital vectors based on the first distance threshold and the second distance threshold can obtain more accurate clustering results, thereby facilitating the subsequent acquisition of the accurate number of clusters.

[0047] In step S104, the number of clusters in the clustering result is obtained.

[0048] In step S104, the text clustering system can identify each cluster in the clustering result to determine the number of clusters, thereby achieving the acquisition of the accurate number of clusters.

[0049] In step S105, the plurality of digital vectors are subjected to second clustering processing based on the number of clusters to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector.

[0050] In step S105, as shown in Figure 2 The text clustering system can first determine an initial centroid vector of each cluster based on the number of clusters, and then perform second clustering processing on the plurality of digital vectors based on the initial centroid vector to cluster the plurality of digital vectors into each cluster and calculate a target centroid vector corresponding to each cluster. The second clustering processing is a clustering method different from the first clustering processing. The initial centroid vector represents the characteristics of the cluster corresponding to the initial centroid vector, and the target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector.

[0051] Further, after obtaining the target centroid vector, the text clustering system can directly label the digital vector corresponding to the target centroid vector based on the target centroid vector to obtain the labeled text, thereby achieving the text clustering effect. Alternatively, the text clustering system can perform second clustering processing or other clustering processing on the plurality of digital vectors based on the target centroid vector to re-determine the digital vector corresponding to each target centroid vector, and then label the digital vector corresponding to the target centroid vector based on the target centroid vector to obtain the labeled text, thereby achieving the text clustering effect.

[0052] It should be noted that by determining the target centroid vector corresponding to each cluster based on the accurate number of clusters, the obtained target centroid vector is more accurate, i.e., better reflects the characteristics of each cluster, thereby achieving a better text clustering effect.

[0053] Based on the scheme defined in steps S101-S105, it can be known that in the embodiment of the application, the limit value of the clustering range is determined based on a plurality of digital vectors, and then the number of clusters is determined based on the limit value of the clustering range. The number of clusters is determined based on the plurality of digital vectors, and then the first distance threshold and the second distance threshold are determined based on the plurality of digital vectors. Then, the plurality of digital vectors are subjected to first clustering processing based on the first distance threshold and the second distance threshold, and the clustering result is obtained. The number of clusters in the clustering result is obtained, and then the plurality of digital vectors are subjected to second clustering processing based on the number of clusters, and the target centroid vector corresponding to each cluster is obtained. Each digital vector corresponds to part of the text to be processed. The first distance threshold is the maximum limit value of the clustering range, and the second distance threshold is the minimum limit value of the clustering range. The target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector.

[0054] It is easy to note that in the above process, the maximum limit value and the minimum limit value of the clustering range are determined based on a plurality of digital vectors, which realizes accurate determination of the maximum limit value and the minimum limit value of the clustering range, and avoids the problem of inaccurate judgment caused by manual setting of the limit value of the clustering range. Therefore, the first clustering processing of the digital vector based on the aforementioned first distance threshold and second distance threshold can obtain a more accurate number of clusters, avoiding the problem of inaccurate judgment caused by manual setting of the number of clusters, and further obtaining a more accurate target centroid vector, realizing a better text clustering effect.

[0055] Therefore, the scheme provided in the present application achieves the purpose of determining the limit value of the clustering range based on a plurality of digital vectors, and then determining the number of clusters based on the limit value of the clustering range, thereby realizing the technical effect of improving the text clustering effect, and further solving the technical problem of poor text clustering effect caused by the fact that the number of clusters cannot be accurately determined in the prior art.

[0056] In an optional embodiment, after obtaining the plurality of digital vectors of the text to be processed, the text clustering system can divide the text to be processed into a plurality of subtexts, and perform second mapping processing and second reduction processing on each subtext to obtain a plurality of segmented words. Then, the plurality of segmented words are subjected to feature extraction processing to obtain a digital vector corresponding to each subtext.

[0057] Optionally, in the embodiment, the text clustering system is used for text clustering of mass texts, and the text clustering system can rely on a cloud computing platform of a Hadoop framework (a distributed system infrastructure), and each clustering algorithm is put into a MapReduce framework (a programming model) for mass texts in the texts to be processed. The design advantage of the MapReduce is that the distributed computing can be realized by simply modifying the traditional single-threaded program through a Map process and a Reduce process, the parallel coding difficulty is reduced, and the development speed of the parallel program is greatly improved. Therefore, the distributed storage and parallel computing in the text clustering process based on the Hadoop technology can improve the computing efficiency.

[0058] Specifically, as shown in Figure 2 , the text clustering system can divide each row of the texts to be processed into a subtext based on the Hadoop distributed environment, then in the process of the second mapping processing, the jieba word segmentation tool is used for word segmentation processing of each row (that is, each subtext), the obtained word segmentation string is removed from the stop words, and a key-value pair is output, the key is the text number, and the value is the word segmentation string after removing the stop words. In the process of the second reduction processing, the key-value pair is directly output, so that a word segmentation string file (that is, the plurality of word segmentation strings) composed of the word segmentation string of each text of each row is obtained. Then, the text clustering system can use a term frequency-inverse document frequency (TF-IDF) method to extract features from the word segmentation string file in the Hadoop distributed environment, so that each subtext forms a digital vector that can be recognized by a computer.

[0059] In the process of extracting features from the word segmentation string file, as shown in Figure 3 , the text clustering system first divides the word segmentation string file into a plurality of small files (such as small file 1, small file 2, …, and small file n in Figure 3 ) by rows, each row in the small file is composed of a text number and a corresponding word segmentation string, and then the mapping and reduction processing is performed on each small file. In the process of mapping processing, the text clustering system removes the repeated word segmentation from the word segmentation in each row, and outputs a key-value pair, the key is the text number, and the value is the word segmentation after taking the set. Then, the text clustering system performs reduction processing, combines each row of word segmentation together, takes the set to form a bag of words, outputs a key-value pair, the key is 1, and the value is the bag of words composed of the word segmentation set, and a bag of words file is obtained.

[0060] Further, after the mapping-reducing processing is performed on each small file, the text clustering system performs feature extraction on the aforementioned segmented string file and bag of words based on TF-IDF to obtain the weight value of each word, and the formula is as follows:

[0061] W = TF w x IDF w

[0062] wherein, TF w represents the word frequency of the segmented word w, and IDF w represents the inverse text frequency index of the segmented word w.

[0063] Specifically, the formula for calculating the word frequency of the segmented word w is as follows:

[0064]

[0065] wherein, TF w represents the word frequency of the segmented word w, N w represents the number of occurrences of the segmented word w in a subtext, and N represents the total number of segmented words corresponding to the segmented word w in the segmented string file.

[0066] Specifically, the formula for calculating the inverse text frequency index of each segmented word is as follows:

[0067]

[0068] wherein, IDF w represents the inverse text frequency index of the segmented word w, Y represents the total number of subtexts, and Y w represents the number of texts corresponding to the subtext containing the segmented word w.

[0069] Further, after obtaining the weight value of each segmented word, the digital vector of each subtext is the collection of the weight of each segmented word in the segmented string corresponding to the subtext, thereby realizing the acquisition of the digital vector corresponding to each subtext. The collection of the digital vector corresponding to each subtext can form a digital vector file.

[0070] It should be noted that by performing the second mapping processing and the second reducing processing on each subtext and performing feature extraction processing on the obtained multiple segmented words, the obtained digital vector can be more accurate.

[0071] In an optional embodiment, in the process of determining the first distance threshold and the second distance threshold based on the plurality of digital vectors, the text clustering system can calculate the ratio of the vector sum of the plurality of digital vectors to the number of vectors corresponding to the plurality of digital vectors to obtain a mean vector, and then determine the farthest distance and the nearest distance between the plurality of digital vectors and the mean vector, thereby determining the first distance threshold and the second distance threshold based on the farthest distance and the nearest distance.

[0072] Optional, such as Figure 4 As shown, the text clustering system can traverse the numeric vectors, sum all numeric vectors, and divide by the total number of numeric vectors to obtain the mean vector. Then, it calculates the distances between the mean vector and all numeric vectors, determining the farthest distance L1 and the nearest distance L2. Based on these distances, a first distance threshold and a second distance threshold are determined. Figure 4 In this context, T1 represents the first distance threshold, and T2 represents the second distance threshold.

[0073] It should be noted that by determining the first and second distance thresholds based on the farthest and nearest distances, the first and second distance thresholds can be determined quickly and accurately, avoiding the problem of inaccurate judgment caused by manually setting distance thresholds. This improves the accuracy of the number of clusters and increases computational efficiency.

[0074] In one optional embodiment, during the process of determining the first distance threshold and the second distance threshold based on the farthest distance and the nearest distance, the text clustering system can calculate the difference between the farthest distance and the nearest distance, and if the ratio of the difference to the farthest distance is greater than a first preset value, determine the difference as the first distance threshold, and determine the product of the farthest distance and the first preset value as the second distance threshold; if the ratio of the difference to the farthest distance is less than or equal to the first preset value, determine the product of the nearest distance and the second preset value as the first distance threshold, and determine the product of the farthest distance and the first preset value as the second distance threshold.

[0075] Optionally, the first and second preset values ​​can be manually input. In this embodiment, the first preset value is preferably 1 / 2, and the second preset value is preferably 2 / 3. Specifically, after determining the farthest distance L1 and the nearest distance L2, if L1-L2>L1 / 2, the text clustering system uses L1-L2 as the first distance threshold and L1 / 2 as the second distance threshold; conversely, if L1-L2≤L1 / 2, the text clustering system uses 3L2 / 2 as the first distance threshold and L1 / 2 as the second distance threshold.

[0076] It should be noted that by comparing the difference with the farthest distance to determine the specific values ​​corresponding to the first and second distance thresholds, a more accurate determination of the first and second distance thresholds is achieved.

[0077] In an optional embodiment, during the first clustering process based on a first distance threshold and a second distance threshold to obtain clustering results for multiple numeric vectors, the text clustering system can obtain at least one center vector from the center list, then select a target numeric vector from the vector list, and determine the distance between the target numeric vector and each center vector. Next, based on the distance between the target numeric vector and each center vector, the first distance threshold, and the second distance threshold, it determines whether the target numeric vector and each center vector belong to different clusters. If the target numeric vector and each center vector belong to different clusters, the center list is updated based on the target numeric vector, and the target numeric vector is removed from the vector list. If the target numeric vector and any center vector belong to the same cluster, the target numeric vector is added to the corresponding cluster, and the target numeric vector is removed from the vector list. The center list stores at least one center vector, each center vector corresponding to a numeric vector among multiple numeric vectors. The vector list stores multiple numeric vectors, and the target numeric vector is a numeric vector that is different from any center vector.

[0078] Furthermore, after determining whether the target digit vector and each center vector belong to different clusters based on the distance between the target digit vector and each center vector, the first distance threshold, and the second distance threshold, when the target digit vector is in an undetermined state, the target digit vector is added to the corresponding cluster and retained in the vector list. The undetermined state indicates that the distance between the target digit vector and any center vector is between the first distance threshold and the second distance threshold.

[0079] Optional, such as Figure 4 As shown, after determining the first and second distance thresholds, the text clustering system randomly selects a digit vector from the dataset (i.e., the vector list) as the target digit vector and calculates its distance to all Canopy centers (i.e., center vectors) in the Canopy list (i.e., the center list). If the distance from the target digit vector to the current Canopy center is less than or equal to the second distance threshold, the target digit vector is determined to belong to the same cluster as the current Canopy center, and the target digit vector is added to the current Canopy (i.e., added to the corresponding cluster) and simultaneously removed from the dataset. If the distance from the target digit vector to any Canopy center is greater than the first distance threshold, the target digit vector does not belong to any Canopy center, the text clustering system adds it to the Canopy list as a new Canopy center, and removes it from the dataset. Figure 4 In this context, S represents the distance from the target digit vector to the current Canopy center, T1 represents the first distance threshold, and T2 represents the second distance threshold.

[0080] Further, if the target digital vector is to a certain Canopy center distance greater than the second distance threshold and less than or equal to the first distance threshold, it is determined that the target digital vector is in a pending state, the text clustering system adds the target digital vector to the current Canopy, and retains it in the data set, so that the target digital vector continues to participate in the next round of clustering.

[0081] Further, the text clustering system continuously selects digital vectors from the vector list as target digital vectors to perform the foregoing clustering process until the data set is empty, at which time the clustering result is obtained, and the number of clusters, i.e., the K value, can be obtained from the clustering result.

[0082] It should be noted that the improved Canopy clustering algorithm is used to cluster the digital vectors to obtain the clustering result, so as to quickly and accurately determine the number of clusters.

[0083] In an optional embodiment, before obtaining the at least one center vector from the center list, the text clustering system can detect the center vector in the center list to obtain a detection result, and in a case where the detection result indicates that there is no center vector in the center list, select a first digital vector from the plurality of digital vectors as a center vector, wherein the first digital vector is any digital vector of the plurality of digital vectors.

[0084] Optionally, before obtaining the at least one center vector from the center list, the text clustering system can first confirm whether there is a Canopy center (i.e., a center vector), as shown in the following table: Figure 4 If there is no Canopy center, the text clustering system can select any digital vector from the foregoing data set as a Canopy center and remove it from the data set.

[0085] It should be noted that by selecting any digital vector from the plurality of digital vectors as a center vector in the case where there is no center vector in the center list, the normal progress of the first clustering process is ensured.

[0086] In an optional embodiment, after determining whether the target digital vector and each center vector belong to different clusters based on the distance between the target digital vector and each center vector, the first distance threshold, and the second distance threshold, the text clustering system can determine the number of target digital vectors in a pending state among a preset number of target digital vectors, and in a case where the number of target digital vectors in a pending state is greater than a first preset threshold, adjust the second distance threshold to a target second distance threshold, wherein the target second distance threshold is greater than the second distance threshold.

[0087] Optionally, in the first clustering process, the text clustering system can confirm the removal rate of the data set in the previous first clustering process (each time the first clustering process is performed on a target digital vector is counted as a first clustering process). Since the target digital vector is in a pending state and is not removed from the data set, it can also be considered as confirming the number of vectors of the target digital vector in a pending state in the previous first clustering process. The predetermined number is used to determine when the text clustering system performs the above-mentioned confirmation after performing the first clustering process for the predetermined number of times. Optionally, the text clustering system can also extract the first clustering results corresponding to the predetermined number of target digital vectors from the first clustering process after performing at least one first clustering process, and determine the removal rate or the number of vectors of the target digital vector in a pending state from the first clustering results.

[0088] Further, if the number of vectors of the target digital vector in a pending state is greater than the first predetermined threshold, the text clustering system can increase the second distance threshold. Specifically, in this embodiment, if the number of vectors of the removed digital vector and the number of vectors of the digital vector in the Canopy list is less than 5% of the number of vectors in the data set when the first clustering process is performed for the predetermined number of times, and increasing T2 is more effective, then T2 is increased. The operator can also manually adjust the second distance threshold based on the clustering effect diagram or the removal rate.

[0089] It should be noted that by adjusting the second distance threshold based on the clustering effect of the target digital vector, better clustering effect can be achieved and the clustering efficiency can be improved.

[0090] In an optional embodiment, in the process of performing the second clustering on the plurality of digital vectors based on the number of clusters to obtain the target centroid vector corresponding to each cluster, the text clustering system can determine a plurality of initial centroid vectors based on the number of clusters and the plurality of digital vectors, then perform a first mapping process on each digital vector in the plurality of digital vectors based on the plurality of initial centroid vectors to obtain the correspondence between each digital vector and the initial centroid vector, and determine the digital vector corresponding to each initial centroid vector based on the correspondence, thereby performing a first reduction process on the digital vector corresponding to each initial centroid vector to obtain the target centroid vector corresponding to each initial centroid vector.

[0091] Optionally, as Figure 5As shown, the text clustering system can employ an improved K-means initialization algorithm to determine K initial centroid vectors based on the number of clusters (i.e., the K value) and the plurality of digital vectors, and generate a centroid vector file based on the K initial centroid vectors, where each row of the centroid vector file consists of an initial centroid vector sequence number and the initial centroid vector. Thereafter, the aforementioned digital vector file is split into a plurality of small files by row, where each row in the small file is a digital vector of a subtext, and then the digital vectors are subjected to a second clustering process based on the centroid vector file in a Hadoop distributed environment.

[0092] Specifically, in the second clustering process, a first mapping process is performed on the digital vectors, i.e., the cosine similarity is calculated between each row of digital vectors in the small file and each initial centroid vector in the centroid vector file, the initial centroid vector corresponding to the maximum cosine similarity calculated with the digital vector is found, a class centroid vector corresponding to the digital vector is generated, and a key-value pair is output, where the sequence number of the class centroid vector is the key and the row of digital vector is the value. The sequence number of the class centroid vector corresponds to the sequence number of the centroid vector, and the first mapping process can be the same as or different from the second mapping process.

[0093] Further, a first reduction process is performed on the digital vectors, i.e., the digital vectors with the same key of the corresponding class centroid vector are added up and divided by their vector quantity to obtain their average value, and the corresponding key and the obtained average value are output as a target centroid vector file, where the aforementioned average value is the target centroid vector, thereby achieving the acquisition of the target centroid vector. The first reduction process can be the same as or different from the second reduction process.

[0094] It should be noted that since the plurality of initial centroid vectors is determined based on the aforementioned accurate number of clusters, the target centroid vector is obtained based on the initial centroid vector and the plurality of digital vectors, which can achieve accurate acquisition of the target centroid vector and thus improve the text clustering effect.

[0095] In an alternative embodiment, in the process of determining the plurality of initial centroid vectors based on the number of clusters and the plurality of digital vectors, the text clustering system can randomly determine a plurality of second digital vectors from the plurality of digital vectors based on the number of clusters, then perform cosine similarity calculation on the plurality of second digital vectors to obtain a cosine angle, and thus determine the plurality of initial centroid vectors based on the plurality of second digital vectors, the number of clusters, and the cosine angle. The vector quantity of the plurality of second digital vectors is associated with the number of clusters, and the vector quantity of the initial centroid vector is the same as the number of clusters.

[0096] Optionally, as Figure 4As shown, after the number of clusters is determined, the text clustering system determines the initial centroid vectors based on the improved K-means initialization algorithm. Specifically, the text clustering system can randomly extract a portion of the plurality of digital vectors as sample vectors (i.e., second digital vectors), and the number of the sample vectors is a multiple of the value of K. In this embodiment, preferably, the number of the sample vectors is three to five times the value of K. Then, the text clustering system randomly selects a sample vector from the sample vectors, and names it as C1, and calculates the cosine similarity between C1 and all the remaining sample vectors. The calculation formula is as follows:

[0097]

[0098] wherein x1 represents one of the two digital vectors for which the cosine similarity needs to be calculated, x2 represents the other of the two digital vectors for which the cosine similarity needs to be calculated, and sim(x1, x2) represents the cosine similarity between the two digital vectors.

[0099] Further, based on the calculation results of the cosine similarity between C1 and all the remaining sample vectors, the sample vector with the largest cosine similarity with C1 is found, and named as Ca. The cosine similarity between Ca and all the remaining sample vectors is calculated, the sample vector with the largest cosine similarity with Ca is found, and named as Cmax, and the sample vector with the smallest cosine similarity with Ca is found, and named as Cmin. Then, the cosine angle corresponding to the cosine similarity value between Ca and Cmin is calculated, and the formula is as follows:

[0100] θ = arc cos(sim(Ca, Cmin))

[0101] wherein θ represents the cosine angle, and sim(Ca, Cmin) represents the cosine similarity value between Ca and Cmin.

[0102] Further, after the cosine angle is obtained, the cosine angle is divided by the number of clusters, i.e., the value of K, to obtain an angle interval θ1, so as to divide the cosine angle into [0, θ1], [0, 2θ1], ∧ [(K-1)θ1, Kθ1], i.e., K intervals. Then, the vector Ca is taken as a reference vector, and the first interval of the aforementioned K intervals is selected first, and the cosine similarity between Ca and the other remaining sample vectors is calculated one by one, until a sample vector that satisfies the cosine similarity in the interval is found. Specifically, the text clustering system selects the first interval first, and when the vector C1 satisfies 0 < arc cos(sim(Ca, C1))≤ θ1, the text clustering system stops the calculation in the interval, and enters the calculation of the next interval, and thus the sample vectors that satisfy the cosine similarity in each interval are found one by one, and K sample vectors, i.e., (C1, C2, ∧ CK) are obtained. K), and take the K sample vectors as initial centroid vectors. Wherein, if the number of initial centroid vectors calculated based on the foregoing sample vectors is less than K, additional sample vectors are randomly selected from the digital vectors for calculation until K initial centroid vectors are obtained, thereby achieving the acquisition of the initial centroid vectors.

[0103] It should be noted that the initial centroid vectors are determined based on the number of clusters and the cosine similarity calculation, which achieves the fast and accurate determination of the initial centroid vectors.

[0104] In an alternative embodiment, after obtaining the target centroid vector corresponding to each cluster, the text clustering system can determine the error vector between each initial centroid vector and the target centroid vector corresponding thereto, and then determine the maximum error vector from the error vectors, so that the initial centroid vector is updated based on the target centroid vector in the case that the maximum error vector is greater than the second preset threshold, to obtain the target initial centroid vector.

[0105] Optionally, as shown in Figure 5 The text clustering system subtracts the initial centroid vector with the same key from the target centroid vector to obtain the error vector, and takes the vector value with the largest absolute value in the error vector as the error value of the error vector. In the error values of the error vectors corresponding to all keys, the maximum error value is found as the total error W (i.e. the maximum error vector) of the centroid vector, and the total error W is compared with the second preset threshold Y. If the total error W is less than the second preset threshold Y, it is determined that the second clustering processing is completed, and the target centroid vector is the final result. If the total error W is greater than or equal to the second preset threshold Y, the foregoing target centroid vector is taken as the new initial centroid vector (i.e. the target initial centroid vector), and the second clustering processing is performed on each digital vector based on the new initial centroid vector in the Hadoop distributed environment until the total error W between the new target centroid vector and the new initial centroid vector is less than the second preset threshold Y.

[0106] It should be noted that by determining the error vector between each initial centroid vector and the target centroid vector corresponding thereto, and determining whether to adjust the initial centroid vector based on the error vector, the accuracy of the result of the text clustering can be effectively controlled, thereby improving the text clustering effect.

[0107] It should be noted that, since the traditional K-means algorithm needs to specify the number of clusters (i.e. K value) and the initial center point of the cluster in advance, and the number of clusters (i.e. K value) and the initial center point of the cluster cannot be accurately determined, it is easy to cause clustering error and even fall into local optimum. Therefore, the application proposes an improved Canopy+K-means initialization clustering algorithm to realize text clustering in view of the problems existing in the clustering of the traditional K-means algorithm under the condition of big data. The algorithm adopts the improved Canopy clustering algorithm to perform "coarse" clustering on the digital vector in the first stage, determines the K value in view of the problem that the K-means algorithm cannot determine the number of clusters in advance, adopts the improved K-means initialization algorithm to perform "fine" clustering on the digital vector in the second stage, determines the initial centroid vector of the K-means algorithm by using the cosine similarity according to the obtained K value, and then uses the parallel computing method by means of the Hadoop distributed framework, thereby improving the efficiency and accuracy of mass text clustering.

[0108] It can be seen that the scheme provided by the application achieves the purpose of determining the limit value of the clustering range based on multiple digital vectors, and then determining the number of clusters based on the limit value of the clustering range, thereby realizing the technical effect of improving the text clustering effect, and further solving the technical problem of poor text clustering effect caused by the fact that the number of clusters cannot be accurately determined in the prior art. And it should be emphasized that the application can be applied to the scene of clustering text in consumption information in the field of financial technology, can also be applied to other scenes in the field of financial technology, and can also be applied to other fields.

[0109] Embodiment 2

[0110] According to the embodiment of the application, an embodiment of a text clustering device is provided, wherein, Figure 6 is a schematic diagram of an optional text clustering device according to an embodiment of the application, as Figure 6 shown, the device comprises:

[0111] The first acquisition module 601 is configured to acquire a plurality of digital vectors of a to-be-processed text, wherein each digital vector corresponds to part of the text in the to-be-processed text.

[0112] The determination module 602 is configured to determine a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is the maximum limit value of the clustering range, and the second distance threshold is the minimum limit value of the clustering range.

[0113] The first processing module 603 is configured to perform first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold, and obtain a clustering result.

[0114] The second acquisition module 604 is configured to acquire the number of clusters in the clustering result.

[0115] The second processing module 605 is configured to perform a second clustering processing on the plurality of digital vectors based on the number of clusters, to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents a feature of the cluster corresponding to the target centroid vector.

[0116] It should be noted that the first obtaining module 601, the determining module 602, the first processing module 603, the second obtaining module 604 and the second processing module 605 correspond to steps S101 to S105 in the above embodiment, and the five modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment 1.

[0117] Optionally, the determining module further includes: a first calculation module configured to calculate a ratio of a vector sum of the plurality of digital vectors to a number of vectors corresponding to the plurality of digital vectors, to obtain a mean vector; a first sub-determining module configured to determine a farthest distance and a nearest distance between the plurality of digital vectors and the mean vector; and a second sub-determining module configured to determine a first distance threshold and a second distance threshold based on the farthest distance and the nearest distance.

[0118] Optionally, the second sub-determining module further includes: a second calculation module configured to calculate a difference between the farthest distance and the nearest distance; a third sub-determining module configured to, in a case where a ratio of the difference to the farthest distance is greater than a first preset value, determine the difference as the first distance threshold and a product of the farthest distance and the first preset value as the second distance threshold; and a fourth sub-determining module configured to, in a case where the ratio of the difference to the farthest distance is less than or equal to the first preset value, determine a product of the nearest distance and a second preset value as the first distance threshold and a product of the farthest distance and the first preset value as the second distance threshold.

[0119] Optionally, the first processing module comprises: a sub-acquisition module, configured to acquire at least one center vector from a center list, wherein the center list is configured to store at least one center vector, and each center vector corresponds to a digital vector in a plurality of digital vectors; a first sub-processing module, configured to select a target digital vector from a vector list and determine a distance between the target digital vector and each center vector, wherein the vector list is configured to store the plurality of digital vectors, and the target digital vector is a digital vector different from any center vector; a second sub-processing module, configured to determine whether the target digital vector and each center vector belong to different clusters based on the distance between the target digital vector and each center vector, a first distance threshold and a second distance threshold; a first updating module, configured to update the center list based on the target digital vector and remove the target digital vector from the vector list in a case where the target digital vector and each center vector belong to different clusters; and a third sub-processing module, configured to add the target digital vector to a corresponding cluster and remove the target digital vector from the vector list in a case where the target digital vector and any center vector belong to the same cluster.

[0120] Optionally, the text clustering apparatus further comprises a fourth sub-processing module, configured to add the target digital vector to a corresponding cluster and retain the target digital vector in the vector list in a case where the target digital vector is in a pending state, wherein the pending state represents that the distance between the target digital vector and any center vector is between the first distance threshold and the second distance threshold.

[0121] Optionally, the text clustering apparatus further comprises a detection module, configured to detect the center vectors in the center list to obtain a detection result; and a selection module, configured to select a first digital vector as a center vector from the plurality of digital vectors in a case where the detection result represents that there is no center vector in the center list.

[0122] Optionally, the text clustering apparatus further comprises a fifth sub-determining module, configured to determine a vector quantity of target digital vectors in a pending state in a preset quantity of target digital vectors; and an adjusting module, configured to adjust the second distance threshold to a target second distance threshold in a case where the vector quantity of target digital vectors in the pending state is greater than a first preset threshold, wherein the target second distance threshold is greater than the second distance threshold.

[0123] Optionally, the second processing module comprises: a sixth sub-determining module, configured to determine a plurality of initial centroid vectors based on the number of clusters and the plurality of digital vectors; a mapping module, configured to perform first mapping processing on each of the plurality of digital vectors based on the plurality of initial centroid vectors to obtain a corresponding relationship between each digital vector and an initial centroid vector; a seventh sub-determining module, configured to determine a digital vector corresponding to each initial centroid vector based on the corresponding relationship; and a reduction module, configured to perform first reduction processing on the digital vector corresponding to each initial centroid vector to obtain a target centroid vector corresponding to each initial centroid vector.

[0124] Optionally, the sixth sub-determining module comprises: an eighth sub-determining module, configured to randomly determine a plurality of second digital vectors from the plurality of digital vectors based on the number of clusters, wherein the number of vectors of the plurality of second digital vectors is associated with the number of clusters; a third calculation module, configured to perform cosine similarity calculation on the plurality of second digital vectors to obtain a cosine angle; and a ninth sub-determining module, configured to determine the plurality of initial centroid vectors based on the plurality of second digital vectors, the number of clusters and the cosine angle, wherein the number of vectors of the initial centroid vectors is the same as the number of clusters.

[0125] Optionally, the text clustering apparatus further comprises: a tenth sub-determining module, configured to determine an error vector between each initial centroid vector and a target centroid vector corresponding thereto; an eleventh sub-determining module, configured to determine a maximum error vector from the error vectors; and a second updating module, configured to update the initial centroid vector based on the target centroid vector to obtain a target initial centroid vector in a case where the maximum error vector is greater than a second preset threshold.

[0126] Optionally, the first obtaining module comprises: a division module, configured to divide the to-be-processed text into a plurality of subtexts; a fifth sub-processing module, configured to perform second mapping processing and second reduction processing on each subtext to obtain a plurality of segmented words; and a feature extraction module, configured to perform feature extraction processing on the plurality of segmented words to obtain a digital vector corresponding to each subtext.

[0127] Embodiment 3

[0128] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is set to execute the above-mentioned text clustering method when running.

[0129] Embodiment 4

[0130] According to another aspect of the embodiments of the present application, an electronic device is also provided, and the electronic device comprises the above-mentioned text clustering apparatus. Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present application, as Figure 7As shown, the electronic device includes one or more processors; memory for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement a program for running, wherein the program is configured to perform the text clustering method described above when running.

[0131] Embodiment 5

[0132] According to another aspect of the embodiments of the present application, there is also provided a computer program product comprising computer programs / instructions which, when executed by a processor, implement the text clustering method described above.

[0133] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0134] In the above-mentioned embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0135] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0136] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0137] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0138] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0139] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method of text clustering, characterized by, The method comprises: obtaining a plurality of digital vectors of a text to be processed, wherein each digital vector corresponds to a part of the text to be processed; determining a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is a maximum limit value of a clustering range, and the second distance threshold is a minimum limit value of the clustering range; performing first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold to obtain a clustering result; obtaining a number of clusters in the clustering result; performing second clustering processing on the plurality of digital vectors based on the number of clusters to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents a feature of the cluster corresponding to the target centroid vector; wherein determining the first distance threshold and the second distance threshold based on the plurality of digital vectors comprises: determining a farthest distance and a nearest distance between the plurality of digital vectors and a mean vector of the plurality of digital vectors; calculating a difference value between the farthest distance and the nearest distance; in a case where a ratio of the difference value to the farthest distance is greater than a first preset value, determining the difference value as the first distance threshold, and determining a product of the farthest distance and the first preset value as the second distance threshold; in a case where the ratio of the difference value to the farthest distance is less than or equal to the first preset value, determining a product of the nearest distance and a second preset value as the first distance threshold, and determining a product of the farthest distance and the first preset value as the second distance threshold; wherein the method further comprises: selecting a target digital vector in a vector list; if a distance between the target digital vector and any center vector is between the first distance threshold and the second distance threshold, determining that the target digital vector is in a pending state, and storing the target digital vector in a data set.

2. The method of claim 1, wherein, Determining the first distance threshold and the second distance threshold based on the plurality of digital vectors comprises: calculating a ratio of a vector sum of the plurality of digital vectors to a number of vectors corresponding to the plurality of digital vectors to obtain a mean vector; determining a farthest distance and a nearest distance between the plurality of digital vectors and the mean vector; determining the first distance threshold and the second distance threshold based on the farthest distance and the nearest distance.

3. The method of claim 2, wherein, Performing first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold to obtain a clustering result comprises: obtaining at least one center vector from a center list, wherein the center list is used to store the at least one center vector, and each center vector corresponds to a digital vector in the plurality of digital vectors; selecting a target digital vector in a vector list, and determining a distance between the target digital vector and each center vector, wherein the vector list is used to store the plurality of digital vectors, and the target digital vector is a digital vector different from any center vector. determining whether the target digital vector belongs to different clusters with each center vector based on distances between the target digital vector and the center vectors, the first distance threshold and the second distance threshold; in a case where the target digital vector belongs to different clusters with each center vector, updating the center list based on the target digital vector and removing the target digital vector from the vector list; in a case where the target digital vector belongs to the same cluster with any center vector, adding the target digital vector to the corresponding cluster and removing the target digital vector from the vector list.

4. The method of claim 3, wherein, After determining whether the target digital vector belongs to different clusters with each center vector based on distances between the target digital vector and the center vectors, the first distance threshold and the second distance threshold, the method further comprises: in a case where the target digital vector is in a pending state, adding the target digital vector to the corresponding cluster and keeping the target digital vector in the vector list, wherein the pending state represents that a distance between the target digital vector and any center vector is between the first distance threshold and the second distance threshold.

5. The method of claim 3, wherein, Before obtaining at least one center vector from the center list, the method further comprises: detecting the center vectors in the center list to obtain a detection result; in a case where the detection result represents that there is no center vector in the center list, selecting a first digital vector from the plurality of digital vectors as a center vector.

6. The method of claim 4, wherein, After determining whether the target digital vector belongs to different clusters with each center vector based on distances between the target digital vector and the center vectors, the first distance threshold and the second distance threshold, the method further comprises: determining a vector quantity of target digital vectors in the pending state among a preset quantity of target digital vectors; in a case where the vector quantity of target digital vectors in the pending state is greater than a first preset threshold, adjusting the second distance threshold to a target second distance threshold, wherein the target second distance threshold is greater than the second distance threshold.

7. The method of claim 1, wherein, performing a second clustering processing on the plurality of digital vectors based on the quantity of clusters to obtain a target centroid vector corresponding to each cluster, comprising: determining a plurality of initial centroid vectors based on the quantity of clusters and the plurality of digital vectors; performing a first mapping processing on each digital vector in the plurality of digital vectors based on the plurality of initial centroid vectors to obtain a corresponding relationship between each digital vector and an initial centroid vector; determining a digital vector corresponding to each initial centroid vector based on the corresponding relationship; performing a first reduction processing on the digital vector corresponding to each initial centroid vector to obtain a target centroid vector corresponding to each initial centroid vector.

8. The method of claim 7, wherein, determining a plurality of initial centroid vectors based on the quantity of clusters and the plurality of digital vectors, comprising: randomly determining a plurality of second digital vectors from the plurality of digital vectors based on the quantity of clusters, wherein a vector quantity of the plurality of second digital vectors is associated with the quantity of clusters; performing cosine similarity calculation on the plurality of second digital vectors to obtain a cosine angle; determining a plurality of initial centroid vectors based on the plurality of second digital vectors, the number of clusters and the cosine angle, wherein the number of initial centroid vectors is the same as the number of clusters.

9. The method of claim 7, wherein, After obtaining the target centroid vector corresponding to each cluster, the method further comprises: determining an error vector between each initial centroid vector and the target centroid vector corresponding thereto; determining a maximum error vector from the error vectors; in a case where the maximum error vector is greater than a second preset threshold, updating the initial centroid vector based on the target centroid vector to obtain a target initial centroid vector.

10. The method of claim 1, wherein, obtaining a plurality of digital vectors of a text to be processed, comprising: dividing the text to be processed into a plurality of subtexts; performing second mapping processing and second reduction processing on each subtext to obtain a plurality of segmented words; performing feature extraction processing on the plurality of segmented words to obtain a digital vector corresponding to each subtext.

11. A text clustering apparatus characterized by comprising: comprising: a first obtaining module, configured to obtain a plurality of digital vectors of a text to be processed, wherein each digital vector corresponds to part of the text to be processed; a determining module, configured to determine a first distance threshold and a second distance threshold based on the plurality of digital vectors, wherein the first distance threshold is a maximum limit value of a clustering range, and the second distance threshold is a minimum limit value of the clustering range; a first processing module, configured to perform first clustering processing on the plurality of digital vectors based on the first distance threshold and the second distance threshold to obtain a clustering result; a second obtaining module, configured to obtain the number of clusters in the clustering result; a second processing module, configured to perform second clustering processing on the plurality of digital vectors based on the number of clusters to obtain a target centroid vector corresponding to each cluster, wherein the target centroid vector represents the characteristics of the cluster corresponding to the target centroid vector; wherein the device is further configured to determine the farthest distance and the nearest distance between the plurality of digital vectors and a mean vector of the plurality of digital vectors; calculate a difference value between the farthest distance and the nearest distance; in a case where the ratio of the difference value to the farthest distance is greater than a first preset value, determine the difference value as the first distance threshold and the product of the farthest distance and the first preset value as the second distance threshold; in a case where the ratio of the difference value to the farthest distance is less than or equal to a first preset value, determine the product of the nearest distance and a second preset value as the first distance threshold and the product of the farthest distance and the first preset value as the second distance threshold; wherein the device is further configured to select a target digital vector in a vector list; if the distance between the target digital vector and any center vector is between the first distance threshold and the second distance threshold, determine that the target digital vector is in a pending state, and place the target digital vector in a data set.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is configured to execute the text clustering method in any one of claims 1 to 10 when running.

13. An electronic device, comprising: The electronic device comprises one or more processors; The memory is configured to store one or more programs, and the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a program running method, wherein the program is configured to execute the text clustering method in any one of claims 1 to 10 when running.

14. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the text clustering method in any one of claims 1 to 10.

Images