Data clustering method, system, electronic device, and storage medium

Abstract

The application relates to the computer technical field and provides a data clustering method, a data clustering system, electronic equipment and a storage medium, the method comprises the following steps: initializing an original corpus based on a clustering center point and user features to obtain a first target corpus; similarity matrixes suitable for the term frequency inverse document matrix are calculated according to segmentation adopted in term frequency statistics in the process of constructing the term frequency inverse document matrix; the similarity matrixes are input into the first target corpus, the vector cosine similarity of each query term and all non-query terms is calculated and arranged in descending order to obtain a recommended result of an extended word; a sorting result is obtained by combining the recommended result and user information; the first target corpus is updated based on the sorting result to obtain a second target corpus. The application realizes quick clustering of new data of the same type, improves the clustering efficiency and precision, solves the problem of too many iteration numbers caused by randomly selecting a clustering center, and improves the clustering effect of data clustering.

Description

Data clustering methods, systems, electronic devices and storage media Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data clustering method, system, electronic device, and storage medium. Background Technology

[0002] In existing technologies, there is a lack of methods for automatically clustering and labeling internal enterprise data, including structured text and unstructured data, according to different user dimensions. The main methods include manually labeled clustering and K-means clustering. Manually labeled clustering relies on manual labeling, which cannot quickly cluster newly added data of the same type, resulting in low efficiency and accuracy. K-means clustering suffers from poor global search capabilities and a high dependence on initial cluster centers. It randomly selects K objects as initial cluster centers, leading to numerous clustering iterations and poor clustering results. Summary of the Invention

[0003] This invention provides a data clustering method, system, electronic device, and storage medium, aiming to improve the clustering efficiency, clustering accuracy, and clustering effect of data clustering.

[0004] In a first aspect, the present invention provides a data clustering method, comprising:

[0005] The original corpus is initialized based on cluster centroids and user features to obtain the first target corpus; the cluster centroids are the K-means algorithm centroids obtained from the labeled words in the dataset of the original corpus.

[0006] Based on the word segmentation used in the word frequency statistics during the construction of the inverse word frequency document matrix, a similarity matrix that fits the inverse word frequency document matrix is ​​calculated.

[0007] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of the extended terms.

[0008] Combining the recommendation results and user information, a ranking result is obtained, and the first target corpus is clustered and updated based on the ranking result to obtain the second target corpus.

[0009] In one embodiment, obtaining the ranking result by combining the recommendation result and the user information includes:

[0010] Obtain the user feature tag token weight and the historical total active token weight;

[0011] Based on the time factor, the weight of the user feature tag token, the weight of the historical total active token, and the recommendation results, the user interest score for each extended word is calculated;

[0012] Sort the user interest scores of each extended word from high to low, and output the ranking results for each extended word;

[0013] Output a preset number of target expanded words based on the sorting results, and receive the determination results returned based on the target expanded words;

[0014] The sorting result is determined from the target expanded words based on the determination result.

[0015] After performing clustering updates on the first target corpus based on the ranking results to obtain the second target corpus, the process further includes:

[0016] Respond to the operation request and obtain input data based on the operation request;

[0017] Extract clustering annotations from the second target corpus;

[0018] The input data is labeled with similar data based on the clustering annotation.

[0019] The initialization of the original corpus based on cluster centroids and user features to obtain the first target corpus includes:

[0020] The text corpus in the original corpus is segmented, stop words are removed, and part-of-speech tags are applied to obtain the dataset of the original corpus;

[0021] Keywords were extracted from the dataset using the TF-IDF algorithm; the keywords were words that appeared more than a preset number of times in the dataset.

[0022] The keywords are segmented and labeled, and the labeled content of the keywords is determined as the cluster centers of the initial K-means clustering.

[0023] The user features are categorized into clusters with the nearest cluster center by using the Minkowski distance, and the original corpus is initialized to obtain the first target corpus.

[0024] The formula for calculating the cluster center is:

[0025]

[0026] Among them, u kis the clustering center of the k-th group, C(k) is the feature words contained in the k-th word cluster, d(x) is the Minkowski distance from the k-th feature word x to the current clustering center, avg(x) is the average value of the Minkowski distances from all feature words within the k-th group to the current clustering center, m represents the number of feature words within the k-th group that satisfy the condition d(x) < avg(x), and I[d(x) < avg(x)] is an exponential function that has a value of 1 when d(x) < avg(x) and a value of 0 when d(x) ≥ avg(x).

[0027] Inputting the similarity matrix into the first target corpus, calculating the vector cosine similarity between each query term and all non-query terms and arranging them in descending order to obtain the recommended result of the extended words, includes:

[0028] Inputting the similarity matrix into the first target corpus and calculating the vector cosine similarity between each query term and all non-query terms;

[0029] Constructing a target similarity matrix based on the vector cosine similarity between each pair of texts of each query term and all non-query terms;​​​​​​​​​​​​​​​​​​​​​​​The corpus query matching module is used to input the similarity matrix into the first target corpus, calculate the vector cosine similarity between each query term and all non-query terms and sort them in descending order to obtain the recommendation results of the extended words;

[0038] The data clustering module is used to combine the recommendation results and user information to obtain a ranking result, and to perform clustering updates on the first target corpus based on the ranking result to obtain a second target corpus.

[0039] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data clustering method described in the first aspect.

[0040] Fourthly, the present invention also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a computer program, which, when executed by the processor, implements the data clustering method described in the first aspect.

[0041] Fifthly, the present invention also provides a computer program product, the computer program product comprising a computer program that, when executed by the processor, implements the data clustering method described in the first aspect.

[0042] The data clustering method, system, electronic device, and storage medium provided by this invention initialize an original corpus based on cluster centroids and user features to obtain a first target corpus. The cluster centroids are the centroids obtained by the K-means algorithm based on the labeled words in the dataset of the original corpus. Based on the word segmentation used in the word frequency statistics during the construction of the word frequency inverse document matrix, a similarity matrix adapted to the word frequency inverse document matrix is ​​calculated. The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of extended words. Combining the recommendation results and user information, a ranking result is obtained. Based on the ranking result, the first target corpus is clustered and updated to obtain a second target corpus.

[0043] In the process of data clustering, no manual labeling is required, enabling rapid clustering of newly added data of the same type, which improves the efficiency and accuracy of clustering. At the same time, the cluster centers are determined based on the label words, which solves the problem of a large number of iterations when clustering is carried out by randomly selecting cluster centers, thus improving the clustering effect of data clustering. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 is a flowchart illustrating the data clustering method provided by this invention;

[0046] Figure 2 is a schematic diagram of the data clustering system provided by the present invention;

[0047] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] This invention provides an embodiment of a data clustering method. It should be noted that although the logical order is shown in the flowchart, under certain data conditions, the steps shown or described may be performed in a different order than that shown here.

[0050] Referring to Figure 1, which is a flowchart illustrating the data clustering method provided by the present invention, the data clustering method provided in this embodiment includes:

[0051] Step 101: Initialize the original corpus based on cluster centroids and user features to obtain the first target corpus.

[0052] This invention describes a data clustering method using a data clustering system as the execution subject. However, the execution subject is not limited to a data clustering system. A data clustering system is a system used to divide objects in a dataset into similar groups or clusters, and supports incremental automatic aggregation of labeled data of the same type according to human dimensions. Compared with massive data processing, it has the advantages of high flexibility, small storage, high efficiency, and high accuracy.

[0053] Specifically, the data clustering system establishes an original corpus, preprocesses the text data in the original corpus to obtain a dataset, and extracts keywords from the dataset using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm. These keywords are then segmented and labeled, and the labeled content is used as the cluster centers for the initial K-means clustering. Here, the corpus is a collection of text or language samples, the cluster centers are the K-means center points obtained from the labeled words in the original corpus dataset, and the keywords are words in the dataset that appear more than a preset frequency, which is set according to actual conditions.

[0054] It's important to note that the K-means algorithm is an unsupervised learning algorithm used to divide a dataset into K clusters. By continuously updating the cluster centroids, it gradually optimizes the clustering results. The goal of the K-means algorithm is to minimize the sum of the squared distances from each data point to the centroid of its assigned cluster, i.e., minimizing the sum of squared errors. The TF-IDF algorithm is a feature extraction method used to measure the importance of a term in a corpus. By calculating the frequency of a term in the text and its inverse document frequency (IVF) in the entire corpus, it identifies terms that appear frequently in a specific text but are relatively rare in the overall corpus. For example, terms with a high TF-IDF value are those that appear frequently in the text but are relatively rare in the entire corpus.

[0055] Furthermore, the data clustering system obtains the minimized cluster centers by performing K-means clustering iterations on the cluster centers, and then uses the Minkowski distance to classify the feature words corresponding to user features into the clusters of the nearest cluster centers, thus initializing the original corpus and obtaining the first target corpus. The Minkowski distance is a method for measuring the similarity or distance between two vectors.

[0056] Furthermore, the data clustering system obtains first data by parsing unstructured data or reading structured data from the first target corpus. Key content is extracted from the first data, and corresponding second data is generated through hashing. The top N highest-ranking words in the second data are extracted as keywords. These keywords are then deduplicated and filtered to remove meaningless words (i.e., duplicate keywords). The filtered keywords are used as markers to mark the positions in the second data containing these keywords, resulting in a marked text. Here, the first data is the parsed unstructured data or the read structured data, and the second data is the hash value obtained by converting the parsed unstructured data or the read structured data through hashing. Hashing is an algorithm that compresses a message of arbitrary length into a fixed-length output. Hashing is unique and irreversible, therefore, it can be used for unique data identification.

[0057] It should be noted that structured data refers to data stored according to fixed formats and rules, with each data field having a clear definition and data type. This type of data is usually easy to manage, process, and analyze. Unstructured data, on the other hand, refers to data that is not stored in a fixed format and according to rules. This type of data usually requires mining and analysis to obtain useful information.

[0058] Furthermore, the data clustering system sends labeled text data to the pipeline processor, decomposes the text data that matches the key content according to the local thesaurus, obtains new text data, updates the weights, obtains intelligent word segmentation, adds the intelligent word segmentation to the indexed document and builds an index for keyword retrieval. Therefore, it can be understood that the data clustering system obtains intelligent word segmentation by training labeled text data, and builds an index based on the intelligent word segmentation.

[0059] Step 102: Calculate the similarity matrix that matches the inverse document matrix based on the word segmentation used in the word frequency statistics during the construction of the inverse document matrix.

[0060] Step 103: Input the similarity matrix into the first target corpus, calculate the vector cosine similarity between each query term and all non-query terms and sort them in descending order to obtain the recommendation results of the extended terms;

[0061] Step 104: Combine the recommendation results and user information to obtain a ranking result, and perform clustering update on the first target corpus based on the ranking result to obtain a second target corpus.

[0062] Specifically, the data clustering system performs word segmentation on the first target corpus, counts the word frequency of each segment, obtains the TF-IDF vector of the text to which the segment belongs using the TF-IDF algorithm, constructs a word frequency inverse document matrix based on the word segmentation used for word frequency counting, and calculates a similarity matrix that matches the constructed word frequency inverse document matrix. Here, word segmentation divides a continuous text sequence into meaningful terms or phrases, and the similarity matrix contains the similarity values ​​between word segments.

[0063] Furthermore, the data clustering system inputs the similarity matrix into the first target corpus, calculates the vector cosine similarity between each query term and all non-query terms, constructs a target similarity matrix based on the vector cosine similarity, obtains an expanded word set based on the non-query terms in the target similarity matrix, and sorts the expanded word set in descending order. The recommendation results of the expanded words are then calculated. Here, query terms are the words used by the user when making a query, usually representing the user's query topic or intent; non-query terms are words not directly related to the user's query, usually having no semantic or contextual connection with the query terms; and expanded words are words used in information retrieval to expand the user's query.

[0064] Furthermore, the data clustering system combines the recommendation results with user information and calculates the user interest score for each extended word. It sorts the extended words according to their user interest scores and outputs the sorting results for each extended word, which is to obtain the sorting results related to the user. Based on the sorting results, the first target corpus is clustered and updated to obtain the second target corpus.

[0065] The data clustering method, system, electronic device, and storage medium provided by this invention initialize an original corpus based on cluster centroids and user features to obtain a first target corpus. The cluster centroids are the centroids obtained by the K-means algorithm based on the labeled words in the dataset of the original corpus. Based on the word segmentation used in the word frequency statistics during the construction of the word frequency inverse document matrix, a similarity matrix adapted to the word frequency inverse document matrix is ​​calculated. The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of extended words. Combining the recommendation results and user information, a ranking result is obtained. Based on the ranking result, the first target corpus is clustered and updated to obtain a second target corpus.

[0066] In the process of data clustering, no manual labeling is required, enabling rapid clustering of newly added data of the same type, which improves the efficiency and accuracy of clustering. At the same time, the cluster centers are determined based on the label words, which solves the problem of a large number of iterations when clustering is carried out by randomly selecting cluster centers, thus improving the clustering effect of data clustering.

[0067] Further, step 101 initializes the original corpus based on cluster centroids and user features to obtain the first target corpus, including:

[0068] The text corpus in the original corpus is segmented, stop words are removed, and part-of-speech tags are applied to obtain the dataset of the original corpus;

[0069] Keywords were extracted from the dataset using the TF-IDF algorithm; the keywords were words that appeared more than a preset number of times in the dataset.

[0070] The keywords are segmented and labeled, and the labeled content of the keywords is determined as the cluster centers of the initial K-means clustering.

[0071] The user features are categorized into clusters with the nearest cluster center by using the Minkowski distance, and the original corpus is initialized to obtain the first target corpus.

[0072] Specifically, the data clustering system preprocesses the text corpus in the original corpus. This involves word segmentation, stop word removal, and part-of-speech tagging to obtain the original corpus dataset. Word segmentation breaks down a continuous text corpus into meaningful words; stop word removal removes common, meaningless words; and part-of-speech tagging assigns the part of speech to each word within a sentence. This preprocessing of the text corpus during its construction allows for a better representation of the corpus's linguistic features and structure.

[0073] Furthermore, the data clustering system extracts keywords from the dataset using the TF-IDF algorithm, segments and labels the extracted keywords, and uses the labeled content of the keywords as the cluster centers for the initial K-means clustering. The data clustering system uses the formula for calculating cluster centers and the Minkowski distance to perform K-means clustering iterations on the cluster centers, calculating the center point of the k-th group, which is the minimum cluster center point. The feature words corresponding to user features are then classified into the clusters of the nearest cluster centers, thus initializing the original corpus and obtaining the first target corpus. The formula for calculating the cluster centers is:

[0074]

[0075] Among them, u kis the clustering center point of the k-th group, C(k) is the feature word included in the k-th word cluster, d(x) is the Minkowski distance from the k-th feature word x to the current clustering center point, avg(x) is the average value of the Minkowski distances from all feature words in the k-th group to the current clustering center point, m represents the number of feature words in the k-th group that satisfy the condition d(x) < avg(x), and I[d(x) < avg(x)] is an exponential function that has a value of 1 when d(x) < avg(x) and a value of 0 when d(x) ≥ avg(x).

[0076] It should be noted that in mathematics, through the argmin calculation method, the variable value that can make the objective function obtain the minimum value can be obtained. In the Kmeans algorithm, by minimizing the distance between feature words and satisfying the constraint condition of d(x) < avg(x), the argmin calculation method calculates the variable value that minimizes the objective function, that is, determines the center point of a specific group to achieve the best clustering result.

[0077] Furthermore, it should be noted that the current Kmeans algorithm itself has poor global search ability and strong dependence on the initial clustering center. Moreover, the Kmeans algorithm randomly selects k objects from the samples as the initial clustering center, resulting in a large number of clustering iteration times and poor clustering effects. The embodiment of the present invention clusters feature words based on the Kmeans algorithm, designates k clustering centers for Kmeans as the initial clustering center points specified, solves the problem of a large number of clustering iteration times due to randomly selecting the initial clustering center for clustering. At the same time, by using the Minkowski distance to assign points to the cluster closest to them, the clustering distance k-point distance is improved, and the problem of low accuracy of the clustering result is solved.

[0078] The embodiment of the present invention preprocesses the original corpus to obtain the dataset of the original corpus, extracts the keywords in the dataset through the TF-IDF algorithm, performs word segmentation and marking on the keywords, determines the marked content as the Kmeans initial clustering center point, initializes the original corpus through the calculation formula of the clustering center point and the Minkowski distance, obtains the first target corpus, realizes the rapid clustering of the newly added data of the same type, improves the efficiency and accuracy of clustering, and solves the problem of a large number of iteration times due to randomly selecting the clustering center, improves the clustering effect of data clustering. At the same time, according to the characteristics such as the roles and operation habits of different users, the optimal number of clusters and clustering centers are determined, and users are assigned to different clusters, solving the problem of intelligent text annotation of data for different users.

[0079] Furthermore, step 102 calculates the similarity matrix adapted to the term frequency-inverse document matrix based on the word segmentation used in the term frequency statistics during the construction of the term frequency-inverse document matrix, including:

[0080] The word segmentation used in word frequency statistics is used as a data object and input into a preset natural language processing model to calculate a similarity matrix that matches the word frequency inverse document matrix.

[0081] Specifically, the data clustering system uses a spectral clustering algorithm to cluster the scattered text data in the first target corpus, resulting in a small set of corpus clusters. Therefore, the text data in the corpus clusters are similar.

[0082] Furthermore, the data clustering system preprocesses the text corpus in the corpus cluster set, performs word segmentation on the preprocessed text corpus to obtain an intermediate document, which is to score the word segments. The system then identifies each word segment in the intermediate document using a pre-set thesaurus and merges the corpus cluster sets with the same meaning. Therefore, by merging the corpus cluster sets with the same meaning, the data clustering system reduces the union of the clustered corpus cluster sets. The pre-set thesaurus is set according to the actual situation.

[0083] Furthermore, the data clustering system performs word frequency statistics on the word segments in the merged corpus cluster set, and vectorizes the word frequencies to obtain TF-IDF vectors corresponding to different word segments. Then, using the cosine similarity function, it calculates the cosine similarity between the TF-IDF vectors corresponding to different word segments, i.e., it calculates the cosine similarity between different word segments. The cosine similarity values ​​are then aggregated to obtain a cosine similarity matrix. The cosine similarity function is:

[0084]

[0085] Here, x and y represent the TF-IDF vectors corresponding to the two segments respectively. The greater the cosine similarity, the more similar the two vectors x and y are. Since the TF-IDF weight of a segment is non-negative, that is, the frequency of the segment is non-negative, the cosine similarity of the TF-IDF vectors corresponding to the two segments is in the range of 0 to 1, and the angle between the two TF-IDF vectors is less than or equal to 90°.

[0086] Furthermore, the data clustering system calculates the inverse document frequency (IVF) of each word segment based on word frequency statistics, and aggregates the IVF frequencies to obtain the word frequency-inverse document matrix. The formula for calculating the IVF frequency is as follows:

[0087]

[0088] Where w is the word segment in the corpus cluster set, N is the total number of texts in the corpus cluster set, x is the text containing the word segment w, and 1 indicates that the word segment w appears at least once in all texts when calculating IDF.

[0089] Furthermore, the data clustering system calculates a suitable similarity matrix based on the cosine similarity matrix and the word frequency inverse document matrix, using the formula for calculating the similarity matrix. Therefore, it can be understood that the data clustering system takes the word segmentation used in word frequency statistics as the data object, inputs it into a pre-defined natural language processing model, and calculates a similarity matrix that matches the word frequency inverse document matrix. The formula for the similarity matrix is:

[0090] S′=D×F

[0091] Where D is the inverse document frequency matrix, F is the cosine similarity matrix, S′ is the similarity matrix, D is the set of IDF word frequency results, F is the set of cosine similarity results, and S′ is the final result set.

[0092] This invention uses word segmentation, which is used in word frequency statistics, as a data object and inputs it into a preset natural language processing model. It calculates a similarity matrix that matches the inverse document matrix of word frequency, provides a feature representation of the text corpus, realizes the clustering and similarity analysis of the text corpus, supports the processing and analysis of similar text corpus, thereby improving the efficiency and accuracy of clustering and the clustering effect of data clustering.

[0093] Further, step 103, based on inputting the similarity matrix into the first target corpus, calculates the vector cosine similarity between each query term and all non-query terms and sorts them in descending order to obtain the expanded word recommendation results, including:

[0094] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated.

[0095] Construct a target similarity matrix based on the vector cosine similarity between each query term and each pair of texts of all non-query terms;

[0096] Using non-query terms in the target similarity matrix as extended terms, extended term mining and statistics are performed based on a preset statistical extended term mining algorithm to obtain an extended term set.

[0097] The query terms are embedded into the extended term set and the union of the two sets is taken to obtain the final extended term set. The final extended term set is then sorted in descending order to obtain the recommendation result.

[0098] Specifically, the data clustering system inputs the similarity matrix into the first target corpus and calculates the vector cosine similarity between each query term and all non-query terms in the first target corpus. The vector cosine similarity function is:

[0099]

[0100] Where x and y represent the TF-IDF vectors corresponding to the two word segments, dot(x,y) represents the dot product of x and y, norm(x) represents the norm of x, and norm(y) represents the norm of y.

[0101] Furthermore, the data clustering system constructs a target similarity matrix based on the vector cosine similarity between each query term and all non-query terms in the first target corpus. It should be noted that the target similarity matrix is ​​a symmetric matrix, and the values ​​on the diagonal of the target similarity matrix are all 1.

[0102] Furthermore, the data clustering system extracts non-query terms from the target similarity matrix and identifies them as extended terms. Based on a pre-defined statistical extended term mining algorithm, statistical extended terms are mined from the pseudo-relevance feedback text set. The mined and statistically analyzed extended terms are then aggregated to obtain the extended term set. The pre-defined statistical extended term mining algorithm includes, but is not limited to, statistical extended term mining algorithms based on Copulas theory. These algorithms, based on Copulas theory, analyze the correlation between words in the text through the correlation of multi-dimensional random variables. Pseudo-relevance feedback refers to obtaining more relevant text by utilizing the frequency of query terms in search results during information retrieval.

[0103] Furthermore, the data clustering system embeds query terms into an expanded word set and takes the union of the expanded word sets to obtain the final expanded word set. Based on the expanded words in the final expanded word set, the data clustering system labels the expanded words existing in the first target corpus, and selects m expanded words relevant to the first target corpus as preferred expanded words. The data clustering system then sorts the final expanded word set in descending order of relevance based on the m preferred expanded words and their corresponding vector cosine similarity, obtaining the recommendation result. The formula for calculating the recommendation result is as follows:

[0104] S=cosθ×m

[0105] Where S represents the ranked set of recommendation results, m represents the number of preferred extended words, and cosθ represents the vector cosine similarity.

[0106] This invention improves the relevance and accuracy of query results by inputting a similarity matrix into a first target corpus, calculating the vector cosine similarity between query terms and non-query terms, constructing a target similarity matrix based on the vector cosine similarity, using non-query terms in the target similarity matrix as extended terms, obtaining an extended term set based on a preset statistical extended term mining algorithm, and sorting the extended term set in descending order to obtain recommendation results. This improves the search results' relevance and accuracy, reduces search errors, and increases search accuracy, thereby reducing the user's search time and effort.

[0107] Further, step 104, based on the combination of the recommendation results and user information, obtains a ranking result, and performs clustering updates on the first target corpus based on the ranking result to obtain a second target corpus, including:

[0108] Obtain the user feature tag token weight and the historical total active token weight;

[0109] Based on the time factor, the weight of the user feature tag token, the weight of the historical total active token, and the recommendation results, the user interest score for each extended word is calculated;

[0110] Sort the user interest scores of each extended word from high to low, and output the ranking results for each extended word;

[0111] Output a preset number of target expanded words based on the sorting results, and receive the determination results returned based on the target expanded words;

[0112] The sorting result is determined from the target expanded words based on the determination result.

[0113] Specifically, the data clustering system obtains the weights of user feature tag tokens and historical total activity tokens. Based on the time factor, user feature tag token weights, and historical total activity token weights, it combines the recommendation results with user information and calculates the user interest score for each extended term using the user interest score calculation formula. Here, the user feature tag token weight refers to the weight value corresponding to a user's professional knowledge or interests in a specific field, and the historical total activity token weight refers to the weight value corresponding to a user's total historical activity on the platform. The user interest score calculation formula is as follows:

[0114] M = |Time Factor%(User Feature Tag Token - Total Historical Active Tokens)| × S.

[0115] Where M represents user interest rating data, and S represents the sorted set of recommendation results.

[0116] Furthermore, the data clustering system sorts the user interest scores of each extended word from high to low and outputs the ranking results for each extended word. The ranking results include: when M≥90, the recommendation result is perfect; when M≥80, the recommendation result is excellent; when M≥70, the recommendation result is good; when M≥60, the recommendation result is average; and when M<60, the recommendation result is poor.

[0117] Furthermore, the data clustering system outputs a preset number of target extended words based on the ranking results and sends these words to a network device. The user selects these target extended words through the network device, choosing the one with the highest degree of intent, and outputs a confirmation result, which is then returned to the data clustering system. The data clustering system receives the confirmation result based on the target extended words, determines the ranking result from the target extended words based on the confirmation result, and simultaneously adds the confirmation result and the ranking result to the current user's second target corpus. The preset number is set according to actual conditions. In one embodiment, the data clustering system outputs the top three target extended words in the ranking result.

[0118] In this embodiment of the invention, user interest scores for each extended term are calculated based on time factors, user feature tag token weights, historical total active token weights, and recommendation results. The user interest scores for each extended term are sorted from high to low. Through content extraction and corpus analogy, data clustering and accuracy ranking are completed, and the ranking results are output. A preset number of target extended terms are output based on the ranking results. Then, the ranking result of the determined result is obtained based on the returned determination result. By utilizing CPU resources and clustering data according to user behavior habits, user historical annotation data can be effectively saved without the need for huge computing resources, and the annotation results can be displayed in a targeted manner according to different users.

[0119] Further, after clustering and updating the first target corpus based on the ranking results to obtain the second target corpus, the process includes:

[0120] Respond to the operation request and obtain input data based on the operation request;

[0121] Extract clustering annotations from the second target corpus;

[0122] The input data is labeled with similar data based on the clustering annotation.

[0123] Specifically, the data clustering system receives operation requests sent by users through network devices, obtains user input data based on the operation requests, and responds to user operation requests based on the input data. The input data includes structured data and unstructured data.

[0124] Furthermore, the data clustering system extracts clustering labels from the second target corpus and, based on the extracted clustering labels, performs similar data labeling on the input data. That is, based on the input data, it initializes the corpus, extracts sample data, extracts keywords to calculate the similarity matrix, performs corpus query matching, and scores and labels sample data, thereby achieving new semi-automatic labeling of incremental basic data.

[0125] According to the received operation request and the acquired input data, the embodiments of the present invention extract clustering labels from the second target corpus, and label the input data with similar data based on the clustering labels. By analogy and extraction, the data is quickly clustered into the historical labeling results according to different clustering result labels, reducing the time for manual judgment of incremental data, improving the value of historical data judgment results, and realizing personalized labeling of data.

[0126] Furthermore, the data clustering system and the data clustering method provided by this invention correspond to each other. Figure 2 shows a schematic diagram of the structure of the data clustering system provided by this invention. The data clustering system includes:

[0127] The corpus initialization module 201 is used to initialize the original corpus based on cluster center points and user features to obtain the first target corpus; the cluster center points are the K-means algorithm center points obtained from the labeled words in the dataset of the original corpus.

[0128] The calculation module 202 is used to calculate a similarity matrix that matches the inverse document matrix based on the word segmentation used in the word frequency statistics during the construction of the inverse document matrix.

[0129] The corpus query matching module 203 is used to input the similarity matrix into the first target corpus, calculate the vector cosine similarity between each query term and all non-query terms and sort them in descending order to obtain the recommendation results of the extended words;

[0130] The data clustering module 204 is used to combine the recommendation results and user information to obtain a ranking result, and to perform clustering updates on the first target corpus based on the ranking result to obtain a second target corpus.

[0131] Furthermore, the corpus initialization module 201 is also used for:

[0132] The text corpus in the original corpus is segmented, stop words are removed, and part-of-speech tags are applied to obtain the dataset of the original corpus;

[0133] Keywords were extracted from the dataset using the TF-IDF algorithm; the keywords were words that appeared more than a preset number of times in the dataset.

[0134] The keywords are segmented and labeled, and the labeled content of the keywords is determined as the cluster centers of the initial K-means clustering.

[0135] The user features are categorized into clusters with the nearest cluster center by using the Minkowski distance, and the original corpus is initialized to obtain the first target corpus.

[0136] Furthermore, the computing module 202 is also used for:

[0137] The word segmentation used in word frequency statistics is used as a data object and input into a preset natural language processing model to calculate a similarity matrix that matches the word frequency inverse document matrix.

[0138] Furthermore, the corpus query matching module 203 is also used for:

[0139] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated.

[0140] Construct a target similarity matrix based on the vector cosine similarity between each query term and each pair of texts of all non-query terms;

[0141] Using non-query terms in the target similarity matrix as extended terms, extended term mining and statistics are performed based on a preset statistical extended term mining algorithm to obtain an extended term set.

[0142] The query terms are embedded into the extended term set and the union of the two sets is taken to obtain the final extended term set. The final extended term set is then sorted in descending order to obtain the recommendation result.

[0143] Furthermore, the data clustering module 204 is also used for:

[0144] Obtain the user feature tag token weight and the historical total active token weight;

[0145] Based on the time factor, the weight of the user feature tag token, the weight of the historical total active token, and the recommendation results, the user interest score for each extended word is calculated;

[0146] Sort the user interest scores of each extended word from high to low, and output the ranking results for each extended word;

[0147] Output a preset number of target expanded words based on the sorting results, and receive the determination results returned based on the target expanded words;

[0148] The sorting result is determined from the target expanded words based on the determination result.

[0149] Furthermore, data clustering systems are also used for:

[0150] Respond to the operation request and obtain input data based on the operation request;

[0151] Extract clustering annotations from the second target corpus;

[0152] The input data is labeled with similar data based on the clustering annotation.

[0153] The specific embodiments of the data clustering system provided by this invention are basically the same as the embodiments of the data clustering method, and will not be described in detail here.

[0154] Figure 3 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 3, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a data clustering method, which includes:

[0155] The original corpus is initialized based on cluster centroids and user features to obtain the first target corpus; the cluster centroids are the K-means algorithm centroids obtained from the labeled words in the dataset of the original corpus.

[0156] Based on the word segmentation used in the word frequency statistics during the construction of the inverse word frequency document matrix, a similarity matrix that fits the inverse word frequency document matrix is ​​calculated.

[0157] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of the extended terms.

[0158] Combining the recommendation results and user information, a ranking result is obtained, and the first target corpus is clustered and updated based on the ranking result to obtain the second target corpus.

[0159] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, the present invention also provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, enable the computer to perform the data clustering methods provided by the above methods, the method comprising:

[0161] The original corpus is initialized based on cluster centroids and user features to obtain the first target corpus; the cluster centroids are the K-means algorithm centroids obtained from the labeled words in the dataset of the original corpus.

[0162] Based on the word segmentation used in the word frequency statistics during the construction of the inverse word frequency document matrix, a similarity matrix that fits the inverse word frequency document matrix is ​​calculated.

[0163] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of the extended terms.

[0164] Combining the recommendation results and user information, a ranking result is obtained, and the first target corpus is clustered and updated based on the ranking result to obtain the second target corpus.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data clustering methods provided above, the method comprising:

[0166] The original corpus is initialized based on cluster centroids and user features to obtain the first target corpus; the cluster centroids are the K-means algorithm centroids obtained from the labeled words in the dataset of the original corpus.

[0167] Based on the word segmentation used in the word frequency statistics during the construction of the inverse word frequency document matrix, a similarity matrix that fits the inverse word frequency document matrix is ​​calculated.

[0168] The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of the extended terms.

[0169] Combining the recommendation results and user information, a ranking result is obtained, and the first target corpus is clustered and updated based on the ranking result to obtain the second target corpus.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data clustering method, characterized in that, include: The original corpus is initialized based on cluster centroids and user features to obtain the first target corpus; The cluster centers are the K-means algorithm centers obtained from the labeled words in the dataset of the original corpus. Based on the word segmentation used in the word frequency statistics during the construction of the inverse word frequency document matrix, a similarity matrix that fits the inverse word frequency document matrix is ​​calculated. The similarity matrix is ​​input into the first target corpus, and the vector cosine similarity between each query term and all non-query terms is calculated and sorted in descending order to obtain the recommendation results of the extended terms. Combining the recommendation results and user information, a ranking result is obtained, and the first target corpus is clustered and updated based on the ranking result to obtain a second target corpus. The initialization of the original corpus based on cluster centroids and user features to obtain the first target corpus includes: segmenting the text in the original corpus, removing stop words, and performing part-of-speech tagging to obtain the dataset of the original corpus; extracting keywords from the dataset using the TF-IDF algorithm; the keywords are words that appear more than a preset number of times in the dataset; segmenting and tagging the keywords, and determining the tagged content of the keywords as the cluster centroids for the initial K-means clustering; and classifying the feature words corresponding to the user features into clusters with distances between them using the Minkowski distance. In the clustering of the most recent cluster centers, the original corpus is initialized to obtain the first target corpus; the ranking result is obtained by combining the recommendation result and user information, including: obtaining the user feature tag token weight and the historical total active token weight; calculating the user interest score of each extended word based on the time factor, the user feature tag token weight, the historical total active token weight and the recommendation result; sorting the user interest scores of each extended word from high to low, and outputting the ranking result of each extended word; outputting a preset number of target extended words according to the ranking result, and receiving the determination result returned based on the target extended words; determining the ranking result from the target extended words according to the determination result.

2. The data clustering method according to claim 1, characterized in that, After performing clustering updates on the first target corpus based on the sorting results to obtain the second target corpus, the method further includes: responding to an operation request and obtaining input data based on the operation request; extracting clustering labels from the second target corpus; and labeling the input data with similar data based on the clustering labels.

3. The data clustering method according to claim 1, characterized in that, The formula for calculating the cluster center is: where, u k is the clustering center of the k-th group, C(k) is the feature words included in the k-th word cluster, d(x) is the Minkowski distance from the k-th feature word x to the current clustering center, avg(x) is the mean of the Minkowski distances from all feature words in the k-th group to the current clustering center, m represents the number of feature words in the k-th group that satisfy the condition d(x) < avg(x), and I[d(x) < avg(x)] is a piecewise function that has a value of 1 when d(x) < avg(x) and a value of 0 when d(x) ≥ avg(x).

4. The data clustering method according to claim 1, characterized in that, The step of inputting the similarity matrix into the first target corpus, calculating the vector cosine similarity between each query term and all non-query terms, and sorting them in descending order to obtain the recommendation result of extended words includes: inputting the similarity matrix into the first target corpus, calculating the vector cosine similarity between each query term and all non-query terms; constructing a target similarity matrix based on the vector cosine similarity between each pair of texts between each query term and all non-query terms; using the non-query terms in the target similarity matrix as extended words, performing extended word mining statistics based on a preset statistical extended word mining algorithm to obtain an extended word set; embedding the query terms into the extended word set and taking the union to obtain the final extended word set, and sorting the final extended word set in descending order to obtain the recommendation result.

5. The data clustering method according to any one of claims 1 to 4, characterized in that, The step of calculating a similarity matrix that matches the inverse document matrix based on the word segmentation used in the word frequency statistics during the construction of the inverse document matrix includes: taking the word segmentation used in the word frequency statistics as a data object, inputting it into a preset natural language processing model, and calculating a similarity matrix that matches the inverse document matrix.

6. A data clustering system, characterized in that, include: The corpus initialization module is used to initialize the original corpus based on cluster centroids and user features to obtain the first target corpus; The cluster centers are the K-means algorithm centers obtained from the labeled words in the dataset of the original corpus. The calculation module is used to calculate a similarity matrix that matches the inverse document matrix based on the word segmentation used in the word frequency statistics during the construction of the inverse document matrix. The corpus query matching module is used to input the similarity matrix into the first target corpus, calculate the vector cosine similarity between each query term and all non-query terms and sort them in descending order to obtain the recommendation results of the extended words; The data clustering module is used to combine the recommendation results and user information to obtain a ranking result, and to update the first target corpus based on the ranking result to obtain a second target corpus. The initialization of the original corpus based on cluster centroids and user features to obtain the first target corpus includes: segmenting the text in the original corpus, removing stop words, and performing part-of-speech tagging to obtain the dataset of the original corpus; extracting keywords from the dataset using the TF-IDF algorithm; the keywords are words that appear more than a preset number of times in the dataset; segmenting and tagging the keywords, and determining the tagged content of the keywords as the cluster centroids for the initial K-means clustering; and grouping the feature words corresponding to the user features using the Minkowski distance. The first target corpus is initialized by grouping the data into the cluster of the nearest cluster center. The process of combining the recommendation results and user information to obtain a ranking result includes: acquiring user feature tag token weights and historical total active token weights; calculating user interest scores for each extended term based on the time factor, the user feature tag token weights, the historical total active token weights, and the recommendation results; sorting each extended term by its user interest score from highest to lowest and outputting the ranking result for each extended term; outputting a preset number of target extended terms based on the ranking result and receiving a determination result returned based on the target extended terms; and determining the ranking result from the target extended terms based on the determination result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data clustering method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data clustering method according to any one of claims 1 to 5.

Images