A text clustering analysis system and method

Abstract

The present invention provides a text clustering analysis system and method, which relates to the technical field of computer natural language processing. The target sentence is obtained through a feature representation module, and the text data set in the target sentence is converted into a primary text vector data set and a secondary text vector data set; a first feature representation and a first text distribution probability value are generated according to the primary text vector data set, and a second feature representation and a second text distribution probability are generated according to the secondary text vector data set to generate a feature data group; the clustering analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and combines the preset classification criteria to determine the target sample cluster; the evaluation and optimization module performs quality analysis on the target sample cluster, and outputs the optimal sample cluster according to the quality analysis result. The technical effect of small clustering analysis error and high accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of computer natural language processing, and in particular to a text clustering analysis system and method. Background Art

[0002] With the rapid development of information technology, the amount of text data generated by people every day is growing explosively. From social media posts, news articles to academic papers, these text data contain rich information and knowledge. Efficiently and accurately extracting valuable information from massive text data is a major challenge to the current computer natural language processing capabilities. The text clustering analysis system based on sentence vectors can effectively help users efficiently process and understand a large amount of text information by converting text data into numerical sentence vectors and then performing clustering analysis based on the similarity of the vectors. It is widely used in information retrieval, sentiment analysis, recommendation systems and other usage scenarios to help organize and understand a large amount of unstructured text data. In actual operation, appropriate text representation learning models, similarity calculation methods, clustering algorithms and evaluation indicators are usually selected according to different application scopes to build efficient and accurate clustering systems. Text representation learning models usually use deep models such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformers to automatically learn the feature representation of text. The similarity calculation method usually uses the cosine calculation formula or the Euclidean distance formula to measure the similarity between evaluation texts. Clustering algorithms include but are not limited to K-Means and DBSCAN algorithms, which merge or split categories of text vectors. Evaluation indicators usually use silhouette coefficient and Davies-Bouldin index to measure the average similarity between a sample and other samples in its cluster.

[0003] At present, traditional text clustering analysis systems based on sentence vectors usually remove stop words as interference factors. However, in actual use, completely removing stop words leads to the problem of over-filtering. Stop words may carry important information in a specific context. Blindly removing them will lead to the loss of important information, thus affecting the accurate communication of the meaning of the text. When it is subsequently applied to sentiment analysis scenarios, it will cause large judgment errors.

[0004] Therefore, how to improve the accuracy of neural network in text sentiment analysis has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to improve the accuracy of text sentiment analysis using a neural network, the present application provides a text clustering analysis system and method.

[0006] In the first aspect, the text clustering analysis system provided by the present application adopts the following technical solution:

[0007] A text clustering analysis system, comprising: a feature representation module, a clustering analysis module and an evaluation and optimization module;

[0008] The feature representation module is used to obtain a target sentence and convert a text data set in the target sentence into a primary text vector data set and a secondary text vector data set;

[0009] The feature representation module is used to generate a first feature representation and a first text distribution probability value according to the primary text vector data set, generate a second feature representation and a second text distribution probability according to the secondary text vector data set, and generate a feature data set according to the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability;

[0010] The cluster analysis module is used to determine the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and to determine the target sample cluster in combination with a preset classification standard;

[0011] The evaluation and optimization module is used to perform quality analysis on the target sample cluster and output the optimal sample cluster according to the quality analysis result.

[0012] Optionally, the feature representation module further includes: a text input unit, a bag-of-words analysis unit, a word frequency analysis unit, and a vector feature unit;

[0013] The text input unit is used to collect a target sentence and transmit the target sentence to the bag-of-words analysis unit and the word frequency analysis unit;

[0014] The bag-of-words analysis unit is used to convert the text data set in the target sentence into a primary text vector data set and transmit it to the vector feature unit;

[0015] The word frequency analysis unit is provided with a preset word frequency model, which is used to convert the text data set in the target sentence into a secondary text vector data set according to the preset word frequency model and transmit it to the vector feature unit;

[0016] The vector feature unit is provided with a deep learning model, and the deep learning model generates a feature data set according to the primary text vector data set and the secondary text vector data set.

[0017] Optionally, the deep learning model includes a Transformer deep learning model, and the vector feature unit is further used to substitute the primary text vector dataset and the secondary text vector dataset into the Transformer deep learning model, and obtain an output result of the Transformer deep learning model;

[0018] Determining feature representation information and probability value information according to the output result;

[0019] A feature data set is generated according to the feature representation information and the probability value information.

[0020] Optionally, the text data set includes words, numbers, English letters and punctuation marks.

[0021] Optionally, the bag-of-words analysis unit is further used to convert the text data in the text data set into high-dimensional vectors in sequence to generate a first set of high-dimensional vectors, wherein the text data in a single sentence corresponds to a single set of high-dimensional vectors;

[0022] A primary text vector data set is generated according to the first high-dimensional vector set, in which each of the high-dimensional vectors contains corresponding keyword data, and the keyword data includes text words, numbers and English words.

[0023] Optionally, the word frequency model is further used to convert the text data in the text data set into frequency vectors to generate a first set of frequency vectors, wherein the text data in a single sentence corresponds to a single set of frequency vectors;

[0024] A secondary text vector data set is generated according to the first frequency vector set, in which each frequency vector contains corresponding frequency data, and the frequency data includes text word frequency, number frequency, English letter frequency and tag symbol frequency.

[0025] Optionally, the cluster analysis module includes: a similarity comparison unit and a similarity clustering unit;

[0026] The similarity comparison unit is used to calculate and generate a similarity index based on the feature data group;

[0027] The similarity clustering unit is used to calculate and generate a target sample cluster according to the feature data group.

[0028] Optionally, the evaluation optimization module is provided with an evaluation algorithm model, and the evaluation algorithm model includes a silhouette coefficient algorithm and a Davies-Bouldin index algorithm;

[0029] The evaluation and optimization module is used to substitute the target sample cluster into the evaluation algorithm model to obtain the sample silhouette coefficient and the sample Davies-Bouldin index.

[0030] Optionally, the evaluation and optimization module is further used to judge the quality of the target sample cluster according to the sample silhouette coefficient and the sample Davies-Bouldin index combined with a preset evaluation value, and the judgment result includes that the sample cluster quality is high or the sample cluster quality is low;

[0031] If it is determined that the sample cluster quality of the target sample cluster is high, outputting the target sample cluster to a preset port;

[0032] If it is determined that the sample cluster quality of the target sample cluster is low, a cluster optimization quality is generated and the cluster optimization quality is sent to the feature representation module so that the feature representation module recalculates the feature data set.

[0033] In a second aspect, the present application provides a text clustering analysis method, the text clustering analysis method comprising:

[0034] The feature representation module obtains a target sentence and converts a text data set in the target sentence into a primary text vector data set and a secondary text vector data set;

[0035] The feature representation module generates a first feature representation and a first text distribution probability value according to the primary text vector data set, generates a second feature representation and a second text distribution probability according to the secondary text vector data set, and generates a feature data set according to the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability;

[0036] The cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and determines the target sample cluster in combination with the preset classification standard;

[0037] The evaluation and optimization module performs quality analysis on the target sample cluster and outputs the optimal sample cluster according to the quality analysis result.

[0038] In summary, the present application includes the following beneficial technical effects:

[0039] The present application obtains the target sentence through the feature representation module and converts the text data set in the target sentence into a primary text vector data set and a secondary text vector data set; generates a first feature representation and a first text distribution probability value according to the primary text vector data set, and generates a second feature representation and a second text distribution probability according to the secondary text vector data set to generate a feature data group; the cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and determines the target sample cluster in combination with the preset classification standard; the evaluation and optimization module performs quality analysis on the target sample cluster, and outputs the optimal sample cluster according to the quality analysis result. The technical effect of small cluster analysis error and high accuracy is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural block diagram of the first embodiment of the text clustering analysis system of the present application;

[0041] Figure 2It is a structural block diagram of the second embodiment of the text clustering analysis system of the present application;

[0042] Figure 3 It is a structural block diagram of the third embodiment of the text clustering analysis system of the present application;

[0043] Figure 4 It is a flowchart of the text clustering analysis method of this application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Reference Figure 1 , Figure 1 This is a structural block diagram of the first embodiment of the text clustering analysis system of this application.

[0046] like Figure 1 As shown, the text clustering analysis system proposed in the embodiment of the present application includes: a feature representation module 10, a clustering analysis module 20 and an evaluation and optimization module 30;

[0047] The feature representation module 10 is used to obtain a target sentence and convert a text data set in the target sentence into a primary text vector data set and a secondary text vector data set.

[0048] It should be noted that this embodiment provides a text clustering analysis method and system based on sentence vectors, which has the advantages of flexible filtering of stop words with high retention, small clustering analysis error and high accuracy. It solves the problem that in the use scenario of sentiment analysis, the text clustering analysis system based on sentence vectors excessively filters stop words, and the clustering analysis has large error and low accuracy.

[0049] In the specific implementation, the user terminal is interconnected with the text clustering analysis system. The user inputs sentences through the user terminal, and the feature representation module converts the text data in the sentence into a primary text vector data set and a secondary text vector data set. The text data in a single sentence corresponds to a single set of high-dimensional vectors and a single set of frequency vectors.

[0050] The feature representation module 10 is used to generate a first feature representation and a first text distribution probability value based on a primary text vector data set, generate a second feature representation and a second text distribution probability based on a secondary text vector data set, and generate a feature data group based on the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability.

[0051] The cluster analysis module 20 is used to determine the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and to determine the target sample cluster in combination with the preset classification standard.

[0052] The evaluation and optimization module 30 is used to perform quality analysis on the target sample cluster and output the optimal sample cluster according to the quality analysis result.

[0053] In a specific implementation, the user terminal is interconnected with the text clustering analysis system, the user inputs a sentence through the user terminal, and the feature representation module converts the text data in the sentence into a primary text vector data set and a secondary text vector data set.

[0054] For example, the user enters the following statement:

[0055] I have been waiting, waiting for the promise that once shone in my life, the promise that will never come. Although I know that all this, perhaps, is really just my wishful thinking, just a fantasy in the endless night. But even so, I am still willing to wait foolishly and without reservation.

[0056] Word segmentation: In this optional embodiment, Jieba word segmentation is used to segment text data. Jieba word segmentation has three word segmentation modes, namely, precise mode, full mode and search engine mode. The precise mode attempts to segment the sentence most accurately, which is suitable for document analysis. Therefore, this embodiment uses the precise mode to segment the original text data. In this embodiment, the result after word segmentation is:

[0057] I have been waiting, waiting for the promise that once shone in my life, the promise that will never come. Although I know that all this, perhaps, is really just my wishful thinking, just a fantasy in the endless night. But even so, I am still willing to wait foolishly and without reservation.

[0058] The first-level text vector dataset includes keyword data such as I, always, wait, that, once, shine, in life, forever, will not, come, promise, etc. The keyword data includes subject (noun, pronoun or numeral), predicate (verb or adjective), object, preposition and modal particle. According to the order of sentences separated by periods, the text data in the first sentence corresponds to the first set of high-dimensional vectors, the text data in the second sentence corresponds to the second set of high-dimensional vectors, and so on;

[0059] The secondary text vector dataset includes frequency data of "I", "always", "in", "wait", "in", "that", "once", "shine", "in life", "promise", etc. The frequency data includes word frequency, number frequency, English letter frequency and punctuation frequency. According to the order of sentences separated by periods, the text data in the first sentence corresponds to the first set of frequency vectors, the text data in the second sentence corresponds to the second set of frequency vectors, and so on.

[0060] The feature representation module substitutes the primary text vector data set and the secondary text vector data set into the Transformer deep learning model, analyzes and obtains the feature representation of the text vector and the probability value of the text vector classification, which is the feature data set Tzsj. The feature data set Tzsj corresponds to the distribution probability of each text feature appearing in the paragraph sentence, and retains the stop words, adjectives and modal particles that connect the context and have a connecting role. In this optional embodiment, the word frequency of stop words such as ",", ", " in the second sentence is calculated, and it is substituted into the Transformer deep learning model as the secondary text vector data set for feature learning, which plays a role in adjusting and optimizing the subsequent clustering sample cluster tasks.

[0061] The cluster analysis module calculates the straight-line distance between the high-dimensional vector and the frequency vector based on the feature data group Tzsj, which is helpful for the subsequent evaluation of the similarity between text vectors. It also obtains multiple clustered sample clusters Ybc after screening the feature data group Tzsj according to the classification criteria, which is convenient for the subsequent measurement of the differences between clusters.

[0062] The quality of text clustering analysis is judged by the evaluation and optimization module. If the quality of text clustering analysis is high, the network connection user end outputs the sample cluster Ybc. If the quality of text clustering analysis is low, the network connection feature representation module recalculates and generates the feature data group Tzsj. The calculation is repeated until the sample cluster Ybc with high quality of text clustering analysis is output. The clustering analysis has small error and high accuracy.

[0063] This embodiment obtains the target sentence through the feature representation module and converts the text data set in the target sentence into a primary text vector data set and a secondary text vector data set; generates a first feature representation and a first text distribution probability value according to the primary text vector data set, and generates a second feature representation and a second text distribution probability according to the secondary text vector data set to generate a feature data group; the cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and combines the preset classification criteria to determine the target sample cluster; the evaluation and optimization module performs quality analysis on the target sample cluster, and outputs the optimal sample cluster according to the quality analysis results. The technical effect of small cluster analysis error and high accuracy is achieved.

[0064] Reference Figure 2 , Figure 2 This is a structural block diagram of the second embodiment of the text clustering analysis system of this application.

[0065] like Figure 2 As shown, the feature representation module 10 further includes: a text input unit 101 , a bag-of-words analysis unit 102 , a word frequency analysis unit 103 and a vector feature unit 104 .

[0066] In a specific implementation, the feature representation module includes a text input unit, a bag-of-words analysis unit, a word frequency analysis unit, and a vector feature unit. The text input unit collects a text data set through a network connection to a user terminal, and transmits the text data set to the bag-of-words analysis unit and the word frequency analysis unit through the network. The bag-of-words analysis unit is provided with a bag-of-words model, and the bag-of-words model converts the text data set into a primary text vector data set, and transmits the text data set to the vector feature unit through the network. The word frequency analysis unit is provided with a word frequency model, and the word frequency model converts the text data set into a secondary text vector data set, and transmits the text data set to the vector feature unit through the network. The vector feature unit is provided with a deep learning model, and the deep learning model calculates and generates a feature data set Tzsj according to the primary text vector data set and the secondary text vector data set. The feature representation module is connected to the clustering analysis module through the network.

[0067] A text input unit 101, used for collecting target sentences and transmitting the target sentences to the bag-of-words analysis unit and the word frequency analysis unit;

[0068] The bag-of-words analysis unit 102 is used to convert the text data set in the target sentence into a primary text vector data set and transmit it to the vector feature unit;

[0069] The word frequency analysis unit 103 is provided with a preset word frequency model, which is used to convert the text data set in the target sentence into a secondary text vector data set according to the preset word frequency model and transmit it to the vector feature unit;

[0070] The vector feature unit 104 is provided with a deep learning model, and the deep learning model generates a feature data set according to the primary text vector data set and the secondary text vector data set.

[0071] It should be noted that the deep learning model includes a Transformer deep learning model, and the vector feature unit is also used to substitute the primary text vector dataset and the secondary text vector dataset into the Transformer deep learning model, and obtain the output results of the Transformer deep learning model; determine feature representation information and probability value information based on the output results; and generate a feature data group based on the feature representation information and probability value information.

[0072] It is understandable that the text data set includes words, numbers, English letters and punctuation marks. Including all sentences entered by the user through the user terminal, the expression of the text data set is {Y 1 , Y 2 , Y 3 , ..., Y n},Y 1 To Y n It corresponds in sequence to the text data input by the user through the user terminal.

[0073] In a specific implementation, the bag-of-words model is also used to convert text data in a text data set into high-dimensional vectors in sequence to generate a first high-dimensional vector set, wherein the text data in a single sentence corresponds to a single set of high-dimensional vectors; a first-level text vector data set is generated based on the first high-dimensional vector set, wherein each high-dimensional vector in the first-level text vector data set contains corresponding keyword data, and the keyword data includes text words, numbers, and English words.

[0074] It is understandable that the bag-of-words model concentrates the text data from Y 1 To Y n The text data of a single sentence is converted into multiple sets of high-dimensional vectors in turn. The text data in a single sentence corresponds to a single set of high-dimensional vectors. A single dimension of a single set of high-dimensional vectors corresponds to a single keyword data. The keyword data includes text words, numbers, and English words. The bag-of-words analysis unit combines multiple sets of high-dimensional vectors into a first-level text vector data set. The expression of the first-level text vector data set is , to This corresponds in turn to the keyword data contained in each group of high-dimensional vectors.

[0075] In a specific implementation, the word frequency model is also used to convert text data in the text data set into frequency vectors to generate a first set of frequency vectors, wherein the text data in a single sentence corresponds to a single set of frequency vectors;

[0076] A secondary text vector data set is generated according to the first frequency vector set, in which each frequency vector contains corresponding frequency data, and the frequency data includes text word frequency, number frequency, English letter frequency and label symbol frequency.

[0077] It should be noted that the word frequency model is based on the text data set from Y 1 To Y n The text data of a single sentence is converted into multiple frequency vectors in turn. The text data in a single sentence corresponds to a single frequency vector. A single dimension of a single frequency vector corresponds to a single frequency data. The frequency data includes word frequency, number frequency, English letter frequency and punctuation frequency. The word frequency analysis unit combines multiple frequency vectors into a secondary text vector data set. The expression of the secondary text vector data set is {TX1 TX 2 TX 3 ,...,TX n}, TX 1 To TX n This corresponds in turn to the frequency data contained in each group of frequency vectors.

[0078] In a specific implementation, the vector feature unit is provided with a Transformer deep learning model, which calculates and generates a feature data set Tzsj according to the primary text vector data set and the secondary text vector data set, and the calculation formula is as follows:

[0079] ;

[0080] In the formula, Tzsj represents the characteristic data group, The keyword data contained in the high-dimensional vector and the frequency data contained in the frequency vector are substituted into the Transformer deep learning model to analyze and obtain the feature representation of the text vector, where a single group of high-dimensional vectors and a single group of frequency vectors correspond to the text data in the same sentence, i represents a collection of multiple groups of high-dimensional vectors and frequency vectors, and Z i represents any element in the input set i, e Zi represents the value of the exponential function, K represents the total number of elements in the input set i, j represents the index of the sum of all elements in the input set i, represents the sum of all elements, which is 1. It means that the Transformer deep learning model converts each element in set i into a corresponding probability value through the SoftMax function.

[0081] Preferably, the similarity comparison unit calculates and generates the similarity index Xszs according to the feature data set Tzsj, and the calculation process is as follows:

[0082] The probability values ​​corresponding to all elements in the feature data set Tzsj are recorded as S, where S={S 1 , S 2 , S 3 ,...,S n},S 1 To S n All probability values ​​are between 0 and 1;

[0083] According to the probability values ​​S corresponding to all elements, the feature representations of high-dimensional vectors and frequency vectors in the feature data set Tzsj are screened, and the coordinate data of the high-dimensional vector keyword data with the highest probability value is taken, which is recorded as P 1 (x 1 ,y 1), take the coordinate data of the frequency vector frequency data with the highest probability value, denoted as P 2 (x 2 ,y 2 );

[0084] According to the keyword coordinates P 1 (x 1 ,y 1 ) and frequency coordinate P 2 (x 2 ,y 2 ), calculate the Euclidean distance between two coordinate points:

[0085] ;

[0086] In the formula, Xszs represents the similarity index, x 2 -x 1 Represents the keyword coordinate P 1 (x 1 ,y 1 ) and frequency coordinate P 2 (x 2 ,y 2 ) in the x-axis dimension, y 2 -y 1 Indicates the keyword coordinate P 1 (x 1 ,y 1 ) and frequency coordinate P 2 (x 2 ,y 2 ) on the y-axis, It represents the straight-line distance between two coordinate points in multidimensional space, which is the similarity index of two vector data.

[0087] Preferably, the similarity clustering unit calculates and generates the sample cluster Ybc according to the feature data group Tzsj, and the calculation process is as follows:

[0088] ;

[0089] In the formula, Ybc represents the sample cluster, BZ represents the fixed parameter that defines the classification standard in the sample cluster calculation formula, It means that fixed parameters are used as classification criteria to screen the feature representations of all high-dimensional vectors and frequency vectors in the feature data group Tzsj to obtain multiple clustered sample clusters.

[0090] Reference Figure 3 , Figure 3 This is a structural block diagram of the third embodiment of the text clustering analysis system of this application.

[0091] like Figure 3As shown, in one embodiment, the cluster analysis module 20 includes: a similarity comparison unit 201 and a similarity clustering unit 202 .

[0092] The similarity comparison unit 201 is used to calculate and generate a similarity index according to the feature data group.

[0093] The similarity clustering unit 202 is used to calculate and generate a target sample cluster according to the feature data group.

[0094] In a specific implementation, the clustering analysis module includes a similarity comparison unit and a similarity clustering unit. The similarity comparison unit calculates and generates a similarity index Xszs based on the feature data group Tzsj, and transmits it to the similarity clustering unit through the network. The similarity clustering unit calculates and generates a sample cluster Ybc based on the feature data group Tzsj. The clustering analysis module is connected to the evaluation optimization module through the network.

[0095] In one embodiment, the evaluation optimization module is provided with an evaluation algorithm model, which includes a silhouette coefficient algorithm and a Davies-Bouldin index algorithm; the evaluation optimization module is used to substitute the target sample cluster into the evaluation algorithm model to obtain the sample silhouette coefficient and the sample Davies-Bouldin index.

[0096] It should be noted that the evaluation and optimization module determines the quality of text clustering analysis based on the similarity index Xszs and the sample cluster Ybc, and connects the user end or the vector feature unit through the network.

[0097] The evaluation and optimization module is also used to judge the quality of the target sample cluster according to the sample silhouette coefficient and the sample Davies-Bouldin index combined with the evaluation preset value, and the judgment result includes the sample cluster quality is high or the sample cluster quality is low;

[0098] If the sample cluster quality of the target sample cluster is determined to be high, the target sample cluster is output to a preset port;

[0099] If the sample cluster quality of the target sample cluster is determined to be low, a cluster optimization quality is generated and sent to the feature representation module so that the feature representation module recalculates the feature data set.

[0100] In a specific implementation, when the sample contour coefficient and the sample Davies-Bouldin index are both greater than the evaluation preset values, the evaluation optimization module determines that the quality of the text clustering analysis is high, and outputs the sample cluster Ybc to the user end through the network. When any one of the sample contour coefficient and the sample Davies-Bouldin index is less than the evaluation preset value or the similarity index Xszs is less than the similarity preset value, the evaluation optimization module determines that the quality of the text clustering analysis is low, and generates a clustering optimization instruction and transmits it to the vector feature unit. The vector feature unit recalculates and generates a feature data group Tzsj until the sample contour coefficient and the sample Davies-Bouldin index are both greater than the preset values ​​and the similarity index Xszs is greater than the similarity preset value.

[0101] It should be noted that, in this embodiment, a text input unit, a bag-of-words analysis unit, a word frequency analysis unit, and a vector feature unit are set through a feature representation module, and the user terminal is interconnected with the text input unit. The user inputs a sentence through the user terminal. The bag-of-words analysis unit and the word frequency analysis unit convert the text data in the sentence into a primary text vector data set and a secondary text vector data set. The text data in a single sentence corresponds to a single group of high-dimensional vectors and a single group of frequency vectors. A single dimension of a single group of high-dimensional vectors corresponds to a single keyword data. The keyword data is specifically the main keyword phrase that expresses the meaning of the text, and the contribution rate to the text clustering analysis is 80%. A single dimension of a single group of frequency vectors corresponds to a single frequency data. The frequency data focuses on retaining the deep meaning of the use of punctuation marks in the text data, and the contribution rate to the text clustering analysis is 20%. The vector feature unit generates a feature data set Tzsj by calculating the Transformer deep learning model, analyzes the distribution probability of each text feature appearing in the paragraph sentence, retains the stop words, adjectives, and modal particles that connect the context and have a connecting role, and flexibly filters the stop words with high retention. In this embodiment, a similarity comparison unit and a similarity clustering unit are set through a cluster analysis module. The similarity comparison unit calculates and generates a similarity index Xszs based on the feature data group Tzsj, which is helpful for the subsequent evaluation of the similarity between text vectors. The similarity clustering unit calculates and generates a sample cluster Ybc based on the feature data group Tzsj, divides the text data into multiple categories, and ensures that the feature representations within the same category are as similar as possible. The quality of text clustering analysis is higher and the error is smaller when applied to the field of sentiment analysis. The evaluation and optimization module determines the quality of text clustering analysis based on the similarity index Xszs and the sample cluster Ybc. If the quality of text clustering analysis is high, the network connection user end outputs the sample cluster Ybc. If the quality of text clustering analysis is low, the network connection feature representation module recalculates and generates the feature data group Tzsj, and the calculation is cyclical until the sample cluster Ybc with high quality of text clustering analysis is output. The clustering analysis has small error and high accuracy.

[0102] The present application embodiment provides a text clustering analysis method, referring to Figure 4 , Figure 4 This is a flow chart of the first embodiment of the text clustering analysis method of this application.

[0103] In this embodiment, the text clustering analysis method includes the following steps:

[0104] Step S10: The feature representation module obtains the target sentence and converts the text data set in the target sentence into a primary text vector data set and a secondary text vector data set.

[0105] Step S20: The feature representation module generates a first feature representation and a first text distribution probability value according to the primary text vector data set, generates a second feature representation and a second text distribution probability according to the secondary text vector data set, and generates a feature data group according to the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability.

[0106] Step S30: The cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and determines the target sample cluster in combination with the preset classification standard.

[0107] Step S40: the evaluation and optimization module performs quality analysis on the target sample cluster and outputs the optimal sample cluster according to the quality analysis result.

[0108] This embodiment obtains the target sentence through the feature representation module and converts the text data set in the target sentence into a primary text vector data set and a secondary text vector data set; generates a first feature representation and a first text distribution probability value according to the primary text vector data set, and generates a second feature representation and a second text distribution probability according to the secondary text vector data set to generate a feature data group; the cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and combines the preset classification criteria to determine the target sample cluster; the evaluation and optimization module performs quality analysis on the target sample cluster, and outputs the optimal sample cluster according to the quality analysis results. The technical effect of small cluster analysis error and high accuracy is achieved.

[0109] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0110] In addition, for technical details not fully described in this embodiment, please refer to the text clustering analysis method provided in any embodiment of the present application, which will not be repeated here.

[0111] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0112] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application. The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structures or equivalent process changes made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are similarly included in the patent protection scope of the present application.

Claims

1. A text clustering analysis system, characterized in that: Includes: feature representation module, cluster analysis module and evaluation and optimization module; The feature representation module is used to obtain a target sentence and convert a text data set in the target sentence into a primary text vector data set and a secondary text vector data set; The feature representation module is used to generate a first feature representation and a first text distribution probability value according to the primary text vector data set, generate a second feature representation and a second text distribution probability according to the secondary text vector data set, and generate a feature data set according to the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability; The cluster analysis module is used to determine the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and to determine the target sample cluster in combination with a preset classification standard; The evaluation and optimization module is used to perform quality analysis on the target sample cluster and output the optimal sample cluster according to the quality analysis result; The feature representation module further includes: a text input unit, a bag-of-words analysis unit, a word frequency analysis unit, and a vector feature unit; The text input unit is used to collect a target sentence and transmit the target sentence to the bag-of-words analysis unit and the word frequency analysis unit; The bag-of-words analysis unit is used to convert the text data set in the target sentence into a primary text vector data set and transmit it to the vector feature unit; The word frequency analysis unit is provided with a preset word frequency model, which is used to convert the text data set in the target sentence into a secondary text vector data set according to the preset word frequency model and transmit it to the vector feature unit; The vector feature unit is provided with a deep learning model, and the deep learning model generates a feature data set according to the primary text vector data set and the secondary text vector data set; The deep learning model includes a Transformer deep learning model, and the vector feature unit is further used to substitute the primary text vector data set and the secondary text vector data set into the Transformer deep learning model, and obtain the output result of the Transformer deep learning model; Determining feature representation information and probability value information according to the output result; generating a feature data set according to the feature representation information and the probability value information; The bag-of-words model converts the text data in the text data set into multiple groups of high-dimensional vectors in sequence. The text data in a single sentence corresponds to a single group of high-dimensional vectors. A single dimension of a single group of high-dimensional vectors corresponds to a single keyword data. The keyword data includes text words, numbers, and English words. The word frequency model converts the text data in the text dataset into multiple sets of frequency vectors in sequence. The text data in a single sentence corresponds to a single set of frequency vectors. A single dimension of a single set of frequency vectors corresponds to a single frequency data. The frequency data includes word frequency, number frequency, English letter frequency, and punctuation frequency.

2. The text clustering analysis system according to claim 1, characterized in that: The text data set includes words, numbers, English letters and punctuation marks.

3. The text clustering analysis system according to claim 1, characterized in that: The bag-of-words analysis unit is further used to convert the text data in the text data set into high-dimensional vectors in sequence to generate a first set of high-dimensional vectors, wherein the text data in a single sentence corresponds to a single set of high-dimensional vectors; A primary text vector data set is generated according to the first high-dimensional vector set, in which each of the high-dimensional vectors contains corresponding keyword data, and the keyword data includes text words, numbers and English words.

4. The text clustering analysis system according to claim 1, characterized in that: The word frequency model is further used to convert the text data in the text data set into a frequency vector to generate a first set of frequency vectors, wherein the text data in a single sentence corresponds to a single set of frequency vectors; A secondary text vector data set is generated according to the first frequency vector set, in which each frequency vector contains corresponding frequency data, and the frequency data includes text word frequency, number frequency, English letter frequency and tag symbol frequency.

5. The text clustering analysis system according to claim 1, characterized in that: The cluster analysis module includes: a similarity comparison unit and a similarity clustering unit; The similarity comparison unit is used to calculate and generate a similarity index based on the feature data group; The similarity clustering unit is used to calculate and generate a target sample cluster according to the feature data group.

6. The text clustering analysis system according to claim 1, characterized in that: The evaluation optimization module is provided with an evaluation algorithm model, and the evaluation algorithm model includes a silhouette coefficient algorithm and a Davies-Bouldin index algorithm; The evaluation and optimization module is used to substitute the target sample cluster into the evaluation algorithm model to obtain the sample silhouette coefficient and the sample Davies-Bouldin index.

7. The text clustering analysis system according to claim 6, characterized in that: The evaluation and optimization module is further used to judge the quality of the target sample cluster according to the sample silhouette coefficient and the sample Davies-Bouldin index combined with the evaluation preset value, and the judgment result includes that the sample cluster quality is high or the sample cluster quality is low; If it is determined that the sample cluster quality of the target sample cluster is high, outputting the target sample cluster to a preset port; If it is determined that the sample cluster quality of the target sample cluster is low, a cluster optimization quality is generated and the cluster optimization quality is sent to the feature representation module so that the feature representation module recalculates the feature data set.

8. A text clustering analysis method, characterized in that: The text clustering analysis method comprises: The feature representation module obtains a target sentence and converts a text data set in the target sentence into a primary text vector data set and a secondary text vector data set; The feature representation module generates a first feature representation and a first text distribution probability value according to the primary text vector data set, generates a second feature representation and a second text distribution probability according to the secondary text vector data set, and generates a feature data set according to the first feature representation, the first text distribution probability value, the second feature representation and the second text distribution probability; The cluster analysis module determines the straight-line distance between the high-dimensional vector and the frequency vector according to the feature data group, and determines the target sample cluster in combination with the preset classification standard; The evaluation and optimization module performs quality analysis on the target sample cluster and outputs the optimal sample cluster according to the quality analysis result; The feature representation module further includes: a text input unit, a bag-of-words analysis unit, a word frequency analysis unit, and a vector feature unit; The text input unit collects a target sentence and transmits the target sentence to the bag-of-words analysis unit and the word frequency analysis unit; The bag-of-words analysis unit converts the text data set in the target sentence into a primary text vector data set and transmits the data set to the vector feature unit; The word frequency analysis unit is provided with a preset word frequency model, converts the text data set in the target sentence into a secondary text vector data set according to the preset word frequency model and transmits it to the vector feature unit; The vector feature unit is provided with a deep learning model, and the deep learning model generates a feature data set according to the primary text vector data set and the secondary text vector data set; The deep learning model includes a Transformer deep learning model, and the vector feature unit substitutes the primary text vector data set and the secondary text vector data set into the Transformer deep learning model, and obtains an output result of the Transformer deep learning model; Determining feature representation information and probability value information according to the output result; generating a feature data set according to the feature representation information and the probability value information; The feature representation module is further used to convert the text data in the text data set into multiple groups of high-dimensional vectors in sequence, the text data in a single sentence corresponds to a single group of high-dimensional vectors, a single dimension of the single group of high-dimensional vectors corresponds to a single keyword data, and the keyword data includes text words, numbers and English words; Among them, the feature representation module is also used to convert the text data in the text data set into multiple groups of frequency vectors in sequence, the text data in a single sentence corresponds to a single group of frequency vectors, and a single dimension of the single group of frequency vectors corresponds to a single frequency data, and the frequency data includes text word frequency, number frequency, English letter frequency and punctuation frequency.

Images