Multidimensional clustering and topic evolution method for network big data text analysis

Abstract

The multi-dimensional clustering topic evolution method for network big data text parsing of the present application selects different topic dimensions and performs text parsing and text classification through doubt degree and log-likelihood function evaluation indicators. First, the optimal topic dimension of the topic evolution model is estimated. A method based on doubt degree and log-likelihood dynamic function is used to judge the topic dimension in the corpus, and the changes between doubt degree and log-likelihood dynamic function under different topic dimensions are calculated to obtain different optimal topic dimensions; second, a topic evolution model based on text parsing is used to evaluate the modeling effect under different topic dimensions from the perspective of text parsing using visualization methods; third, a topic evolution model based on text classification is based on the distribution relationship between documents and topics, and a text representation is constructed through the topic distribution, and this is used as a feature vector to determine the optimal topic dimension. The accuracy and efficiency of network big data text parsing and classification are high.

Description

Technical Field

[0001] The present application relates to a big data text parsing and classification method, and in particular to a multi-dimensional clustering and topic evolution method for network big data text parsing, belonging to the technical field of data text classification. Background Art

[0002] Emerging smart cities and intelligent cities are rapidly developing, using information technology to regulate and manage cities and provide residents with more comprehensive and relevant services. A mobile phone terminal can easily access a wide range of information, such as map services, news media information, advertising and promotions, and more. With the development and popularization of the Internet, the explosive growth of information has forced information managers to face the management difficulties brought about by big data. In the face of the advent of big data, effective organization and management have become the current focus and difficulty.

[0003] Despite the emergence of a wide variety of complex and diverse information carriers, the most common form of information is still text. Not only is text the most direct way for people to receive information, but it also offers the lowest energy consumption. Therefore, how to effectively manage text information and intuitively present it is becoming a hot topic in natural language processing.

[0004] When processing text, the first step is to represent it through vectorization, converting text information into something that computers can process. A more established approach uses vector space models to represent text structure, transforming the study of text into the study of word vectors. Classifiers are then constructed using classification algorithms such as Bayesian classification and support vector machines. These algorithms learn from a collection of text samples with known categories to judge unknown text. Of course, these methods still need improvement. First, the parts of speech that make up a document vary, and the number of words is related to document length. Long documents contain many words, resulting in a large dimensionality in the final model representing the words. Second, classification is based on the semantics of the text, while vector space models represent word distributions. This discrepancy can lead to inaccurate classification results. First, dimensionality reduction techniques can be used to compress the dimensions of the vector space model representing text to a lower dimensional space. Second, when extracting text words, words that better represent the text content are retained to improve text classification accuracy. Topic modeling can be used to learn the distribution of "topics" within a document, and each "topic" represents a distribution of words. In a topic space, a topic is composed of a set of words, so the number of topics is much smaller than the number of words. Representing text in a topic space not only solves the high-dimensionality problem of representing text using word vectors, but also improves the problem that scattered words cannot express the semantic meaning of the text. However, how to more intuitively understand the meaning of "topics" in topic evolution models and how to gain a macro understanding of the content of a document collection have become the primary challenges that topic models such as data evolution need to address.

[0005] The problems that need to be solved by the existing network text parsing and classification models and the key technical difficulties of this application include:

[0006] (1) Existing technologies are unable to effectively manage text information and intuitively present the information expressed by the text. First, the parts of speech that make up a document are different, and the number of words is related to the length of the document. A long document contains many words, resulting in a huge dimension of words represented in the final model. Second, the classification problem is based on the semantics of the text, while the vector space model represents the distribution of words. The difference between the two may cause inaccurate classification results through the classifier. Existing technologies lack targeted dimensionality reduction technology and cannot compress the dimension of the vector space model representing the text into a low-dimensional space. On the other hand, when extracting text words, it is impossible to retain the words that express the text content to improve the accuracy of text classification. Through topic modeling, the distribution of "topics" in documents can be learned, and each "topic" is the distribution of words. However, existing technologies cannot intuitively understand the meaning of "topics" in the topic evolution model, and cannot understand the content of the document set from a macro perspective, resulting in poor accuracy and low efficiency in network big data text parsing and classification.

[0007] (2) The topic model describes the documents in the corpus as a distribution of topics, and topics are distributions of terms. When faced with text parsing problems, there is often a certain degree of ambiguity about which terms can best express the text. When faced with text classes, the spatial dimensions of the text description are generally very high. The more topic dimensions there are, the higher the difficulty and computational complexity of classification will be. If there are fewer topic dimensions, a lot of important information will be discarded. Existing technologies cannot make a good trade-off. There is a lack of optimal topic dimension estimation for the topic evolution model, and it is impossible to combine the language model evaluation indicators with the structural characteristics of the data evolution model itself. There is a lack of a method to judge the topic dimension in the corpus based on doubt degree and log-likelihood dynamic function, and it is impossible to calculate the changes between doubt degree and log-likelihood dynamic function under different topic dimensions, and it is impossible to obtain different optimal topic dimensions.

[0008] (3) The existing technology lacks a topic evolution model based on text parsing, and cannot use visualization methods to evaluate the modeling effect under different topic dimensions from the perspective of text parsing. It lacks an analysis method that highlights high-frequency words in the text parsing of "document-term", lacks a text parsing method based on data evolution and a term analysis method in the topic space, and cannot simultaneously determine the optimal topic dimension through the distribution of "topic-topic" and "topic-term" after modeling; the existing technology lacks a topic evolution model based on text classification, lacks a text representation based on the distribution relationship between documents and topics through topic distribution, and cannot determine the optimal topic dimension by comparing different topic models through text classification. It adopts a single selection method based on doubt degree, and does not optimize the topic evolution model through the semantic expression of high-probability terms and specific comparison results from the perspective of text parsing and text classification, and cannot evaluate the modeling effect of selecting topic dimensions based on different indicators. Summary of the Invention

[0009] This application selects different topic dimensions and performs text parsing and text classification through the evaluation indicators of doubt and log-likelihood function. First, the optimal topic dimension of the topic evolution model is estimated. Combining the evaluation indicators of the language model with the structural characteristics of the data evolution model itself, a method based on doubt and log-likelihood dynamic function is adopted to judge the topic dimension in the corpus, calculates the changes between doubt and log-likelihood dynamic function under different topic dimensions, and uses M-layer cross-validation to find the topic dimension with the lowest doubt and the topic dimension with the largest log-likelihood dynamic function to obtain different optimal topic dimensions; second, a topic evolution model based on text parsing, which uses visualization methods to evaluate the modeling effect under different topic dimensions from the perspective of text parsing; third, a topic evolution model based on text classification, which is based on the distribution relationship between documents and topics, and constructs text representation through topic distribution, and uses this as a feature vector, and compares different topic models through text classification to determine the optimal topic dimension. The accuracy and efficiency of network big data text parsing and classification are high.

[0010] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0011] A multi-dimensional clustering topic evolution method for network big data text parsing constructs an evaluation method for estimating topic dimensions in data evolution models from the perspectives of text parsing and text classification. By calculating the doubt degree and log-likelihood dynamic function under different topic dimensions, a data-driven approach is used to estimate the topic dimension. The results of corpus modeling using the estimated topic dimension are then combined with text parsing and text classification to judge the optimal topic dimension. Through visualization under the topic evolution model, namely, from the perspectives of word frequency and document, word frequency and classification, and topic distribution based on data evolution, the text content is intuitively displayed in the form of word clouds and bubble charts, resulting in a dynamic comparison of the modeling effects based on log-likelihood dynamic function and modeling based on doubt degree.

[0012] M1 - Optimal topic dimension estimation for the topic evolution model: Combining language model evaluation metrics with the structural characteristics of the data evolution model itself, a method based on doubt and log-likelihood dynamic functions is used to determine the topic dimension in the corpus. The changes between doubt and log-likelihood dynamic functions under different topic dimensions are calculated. M-layer cross-validation is used to determine the topic dimension with the lowest doubt and the topic dimension with the largest log-likelihood dynamic function, thereby obtaining different optimal topic dimensions.

[0013] M2 - Topic Evolution Model Based on Text Parsing: This model uses visualization methods to evaluate the effectiveness of modeling under different topic dimensions from the perspective of text parsing. By highlighting high-frequency words in the "document-term" text parsing, it constructs a text parsing method based on data evolution and a term analysis method in the topic space. The optimal topic dimension is determined by the distribution of "topic-topic" and "topic-term" after modeling.

[0014] M3-Topic evolution model based on text classification: Based on the distribution relationship between documents and topics, text representation is constructed through topic distribution and used as feature vector. Different topics are modeled through text classification comparison to determine the optimal topic dimension. The topic evolution model is optimized through the semantic expression of high-probability terms and specific comparison results, and the modeling effect of the topic dimension is evaluated based on different indicators.

[0015] Preferably, the optimal topic dimension estimation of the data evolution model is: by setting different topics M, observing the current number of topics M, using multi-layer cross-validation doubt and log-likelihood dynamic functions, and determining the M value by comparing the visualization graph, which is the optimal topic dimension estimation value of the current corpus.

[0016] Preferably, the data evolution model: assuming that the test text set is represented by M, the terms in the bag-of-words model are represented by w, and the likelihood log value describes the distribution of the terms in the topic space, first, given the prior parameters of the "document-term" matrix, the prior distribution parameter values are determined by using Gibbs sampling, the corpus is divided into a training set and a test set, the model parameters are trained using the training set, and then the log-likelihood dynamic function of the test set is calculated using VEM estimation;

[0017] Estimating the topic dimension is a process of continuous model comparison. It analyzes the changes in the mean of the model doubt and the mean of the log-likelihood dynamic function when the topic dimension changes. The cross-validation method is used to test the data. When modeling the corpus, the text data is randomly divided into M groups of equal size. One group is selected as the test set. The remaining M-1 groups of data are modeled to train the topic evolution model. The trained topic evolution model is then used to model the test set. A set of doubt and log-likelihood dynamic functions are calculated. Repeating this process multiple times under the same conditions will result in multiple sets of doubt and log-likelihood dynamic functions. The mean of the multiple sets of doubt and log-likelihood dynamic functions is output as the result to complete the multi-layer cross-validation.

[0018] When evaluating the modeling effect of the topic evolution model, it is determined that the lower the doubt, the better, and the larger the log-likelihood dynamic function, the better. After a topic modeling, the doubt and log-likelihood dynamic function are calculated simultaneously. A multi-layer cross-validation is performed under the current model parameters to obtain the average doubt and log-likelihood dynamic function. Then, the doubt and log-likelihood dynamic function trends output by the topic dimension analysis are changed. The process of solving the average doubt and log-likelihood dynamic function is as follows:

[0019] Step 1: Given the subject dimension;

[0020] Step 2: Select the Gibbs topic modeling algorithm;

[0021] Step 3: Solve the doubt degree and log-likelihood dynamic function of the current model;

[0022] Step 4: Repeat the M-layer cross-validation 6 times and record the average doubt and log-likelihood dynamic function;

[0023] The topic dimension of the topic evolution model is simulated and determined by the M-layer cross-validation method, and the topic value is determined by combining two evaluation indicators, the doubt degree and the log-likelihood function value.

[0024] Preferably, a topic evolution model based on text parsing is used: by analyzing the text after topic modeling and evaluating the effect after modeling to obtain the optimal modeling topic dimension, a topic evolution model is first constructed based on the estimated topic dimension corpus modeling to express two sets of "document-topic" and "topic-term" relationships. One is to express the core of the corpus by extracting terms under the topic as keywords, and to give an evaluation through comparison based on the analysis of the text word frequency word cloud; the other is to use the topic similarity evaluation method between topic structures as the evaluation of topic modeling, and to evaluate the modeling effects of different topic dimensions from an intuitive and objective perspective from the data evolution vis visualization analysis.

[0025] Preferably, word cloud analysis based on text word frequency: in word cloud analysis, the size of word weight is measured based on text word frequency, and high-weight words are output as the subject words of the word cloud. The process of generating the word cloud includes two steps: first, the weight of each word is calculated for the preprocessed text or corpus; second, the size of the word in the composition space is estimated according to the size of the word weight and the word cloud is output. When outputting the word cloud, the priority of words with different weights in the composition space is considered, and the stability and rationality of the word layout are met by using a compact layout method.

[0026] Preferably, text parsing based on word cloud: the corpus is modeled through a data evolution model, and the distribution of "topic-term" in the corpus is learned. According to the estimated number of two topics, the corpus is subjected to topic modeling with a topic dimension of 20 and a topic modeling with a topic dimension of 60, respectively, to form two groups of comparisons. The distribution of terms under each topic is analyzed in the modeled corpus. The terms under each topic are distributed according to the probability of p(w|z). Words with high probability are prioritized in each topic space. It is approximately determined that words with high word frequency in different topic spaces are prioritized. In the modeling of the two groups, words with higher probability in 120 topic spaces are extracted respectively and the word frequency is counted, and then displayed in the form of word clouds.

[0027] Preferably, the visualization analysis based on the topic evolution model:

[0028] Data evolution vis network visualization is used as a text parsing tool after topic modeling at different topic dimensions. Through multi-scale analysis, the relative distances between objects are grasped, and the data expression in low-dimensional space is obtained. The distribution of corpus documents and the distribution of terms in the topic space after data evolution modeling are analyzed, and the probability of terms is expressed by the ranking of terms.

[0029] After preprocessing the text in the corpus, the text content of text_content is formed. Then, topic modeling is performed on the text based on this. The modeling results are visualized through the topic evolution model, and the output results are formulated into a web page by writing a script;

[0030] The ranking of terms is changed by adjusting the parameter a to help explain the topic. If a is close to 1, the terms at the top are more frequent in the topic space; if a is closer to 0, the terms at the top are more unique in the topic space. By adjusting the size of a, the weight of the terms under the topic is changed, affecting the ranking of the terms. The value of a is determined by the following formula:

[0031] relevance(w|t)=a*p(w|t)+(1+*p(w|t) / p(w) Equation 1

[0032] Based on the relevance of a certain word topic, it is adjusted by a. By comparing the frequency of a given word in the corpus with the topic-specific frequency, it is quickly determined whether it is a highly relevant selected topic; θ represents the vector of topic distribution, p(θ) is the distribution of θ, N represents the number of words in the corpus, w_n is the nth word w, z_n represents the selected topic, p(z|θ) represents the probability of topic z under the topic distribution θ, p(w|z) represents the probability of word w under topic z, expressed as an M×V matrix, M represents the topic, V represents the word in the corpus, α is the parameter describing the distribution p(θ), which affects the distribution of the topic θ vector, and β is the parameter describing the distribution p(w|z), which affects the distribution of the word under topic z;

[0033] The relationship between topics is analyzed by multidimensional scaling, which projects the high-dimensional topic distribution into a two-dimensional space and describes the differences between topics by the size and distance of the bubble chart.

[0034] From the perspective of the distribution of terms under a topic, the fewer topic structures a term has associated with it in the topic space, the more evenly distributed the "topic-term" is. The modeling effect is evaluated by the relationship between different modeling topic dimensions and the topic similarity obtained after modeling. That is, the model is optimal when the average similarity between topics is the smallest.

[0035] The probability distribution of topics in the term space is used to represent the topic vector, the cosine distance is used to represent the difference of topics, and cosine(z i ,z j ) is smaller, the greater the difference in the subject space:

[0036]

[0037] The quality of the topic structure is characterized by the average similarity of the topic vectors:

[0038]

[0039] The modeling effects of the two are compared directly by comparing the average topic similarities under different topic dimension modeling. The average similarities of all topics in the topic evolution model under different topic dimensions are compared. It is determined that when the topic dimension is 20, the difference of topic vectors in the topic space is greater, and the modeling performance with a topic dimension of 20 is relatively better.

[0040] Preferably, a topic evolution model based on multi-dimensional clustering of text is used: the "document-topic" matrix obtained after data evolution modeling of the test set in the corpus is used as the training set of the nearest neighbor classifier, the test set is obtained by the same method and predictions are made through the nearest neighbor classifier, first, the text representation of the data evolution modeling is constructed by estimating different topic dimensions, and the pros and cons of different topic dimensions are estimated through feedback on the classification effect; then the optimal text representation of the data evolution modeling is compared with the text representation constructed by the vector space model.

[0041] Preferably, a text classification model based on data evolution dimensionality reduction is constructed to classify text by using a data evolution model to model the text representation of the corpus. The specific steps are as follows:

[0042] Step 1: Use the data evolution model to model the training document set and obtain the distribution of documents on topics as the vector representation of the documents;

[0043] Step 2: Use the nearest neighbor algorithm to train the vector in step 1;

[0044] Step 3: Use the results in step 1 to model the test document set and obtain the topic distribution in the test document set;

[0045] Step 4: Represent the result as a text vector with a fixed number of text features;

[0046] Step 5: Predict the category of the text vector using the classifier obtained in step 2.

[0047] Preferably, a text classification model based on the nearest neighbor is used: by representing the text in the corpus as a feature vector, the category of the unknown text is determined to be the same as the majority category of T adjacent samples in the feature item space. The number of nearest neighbor samples T and the distance function are known. The T value is the square root of the number of training sets, and the distance function is the Euclidean distance. The algorithm flow is as follows:

[0048] Step 1: First, divide the corpus into a test document set and a training document set, and perform preprocessing;

[0049] Step 2: Extract specific parts of speech for all document sets in the corpus and convert them into text representation based on word vectors;

[0050] Step 3: Select consistent feature vectors based on text word frequency;

[0051] Step 4: For the vector to be classified in the training document set, traverse the distance between the vector and the known classification vector;

[0052] Step 5: Select the nearest vectors in the T vector space and calculate the category weight of the majority vector;

[0053] Step 6: Label the unknown vector as the class with the largest weight for the majority class.

[0054] Compared with the existing technology, the innovation and advantages of this application are:

[0055] (1) This application selects different topic dimensions and performs text parsing and text classification through the evaluation indicators of doubt degree and log-likelihood function. First, it constructs the optimal topic dimension estimation of the topic evolution model. Combining the evaluation indicators of the language model with the structural characteristics of the data evolution model itself, a method based on doubt degree and log-likelihood dynamic function is adopted to judge the topic dimension in the corpus, calculates the changes between doubt degree and log-likelihood dynamic function under different topic dimensions, and uses M-layer cross-validation to find the topic dimension with the lowest doubt degree and the topic dimension with the largest log-likelihood dynamic function, thereby obtaining different optimal topic dimensions; second, it constructs a topic evolution model based on text parsing. Firstly, a visualization method is used to evaluate the modeling effect under different topic dimensions. By highlighting the analysis method of high-frequency words in the comparison of "document-term" text analysis, a text analysis method based on data evolution and a term analysis method in the topic space are constructed. At the same time, the optimal topic dimension is determined by the distribution of "topic-topic" and "topic-term" after modeling; thirdly, a topic evolution model based on text classification is constructed. Based on the distribution relationship between documents and topics, text representation is constructed through topic distribution, and this is used as a feature vector. Different topic models are modeled through text classification comparison to determine the optimal topic dimension. The accuracy and efficiency of network big data text analysis and classification are high.

[0056] (2) One of the innovations of this application is that in the use of topic evolution model for text representation, it avoids the single selection method of doubt degree and proposes to use two judgment indicators, doubt degree and log likelihood dynamic function, for selection. The cross-validation method is used to record the doubt degree and log likelihood dynamic function of the model under different topic dimensions in a data-driven manner. The average change results of doubt degree and log likelihood dynamic function are used as the basis for selecting topic dimensions. From the perspective of text analysis and text classification, the topic evolution model is optimized through the semantic expression of high-probability terms and specific comparison results, and the modeling effect of selecting topic dimensions based on different indicators is evaluated, which is conducive to obtaining the optimal topic dimension under different situations.

[0057] (3) This application proposes to evaluate the modeling effect of the data evolution model from the perspective of text parsing and text classification. In terms of text parsing, the modeling effect of the topic evolution model under different topic dimensions is explored, and the modeling effect is measured from the perspectives of word cloud analysis, potential topic distribution, and topic similarity, so as to evaluate the model of different topic dimension estimation in the topic evolution model. In terms of text classification, it is proposed to evaluate the modeling effect of the data evolution model from the perspective of text classification results. Existing text classification research focuses on the performance comparison between different classification methods. This application proposes to evaluate the modeling effect of the topic evolution model of different topic dimensions from the perspective of task results. For the estimation of multiple topic dimensions, the corpus is modeled and represented in different topic dimensions. The modeling effect under different topic dimensions is evaluated through the nearest neighbor classification results. Text classification based on topic dimensionality reduction has good feasibility and effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the overall block diagram of the optimal topic dimension estimation for the data evolution model.

[0059] Figure 2 This is the overall framework of the topic evolution model based on text parsing.

[0060] Figure 3 This is the overall framework of the topic evolution model based on multi-dimensional text clustering.

[0061] Figure 4 It is a flowchart of text classification based on data evolutionary dimensionality reduction.

[0062] Figure 5 It is a flowchart of text classification based on the nearest neighbor. DETAILED DESCRIPTION

[0063] The following, in conjunction with the accompanying drawings, further describes the technical solution of the multi-dimensional clustering topic evolution method for network big data text parsing provided by this application, so that those skilled in the art can better understand this application and implement it.

[0064] With the development of network information technology, while people enjoy the convenience brought by information data, they also have to face the information confusion caused by the information explosion. Text information is used in various scenarios as a carrier of information dissemination. Therefore, the text parsing and classification of text information has become a hot topic in the field of natural language processing.

[0065] Unlike the previous vector space model representation, the topic model describes the documents in the corpus as a distribution about topics, and topics are distributions about terms, that is, a document is described by a fixed number of topics, and each topic is described by specific words. When faced with text parsing problems, which terms can well express the text is often vague; when faced with text classes, the spatial dimension of describing the text is generally very high. In text parsing and classification, the topic dimension of the topic model is crucial. The more topic dimensions there are, the higher the difficulty and computational complexity of classification. If there are fewer topic dimensions, a lot of important information will be discarded. This application selects different topic dimensions through the evaluation indicators of doubt degree and log-likelihood function and performs text parsing and text classification, and concludes that the modeling effect based on the log-likelihood dynamic function is better than the modeling effect based on doubt degree.

[0066] (1) Estimation of the optimal topic dimension of the topic evolution model. This application combines the language model evaluation indicators with the structural characteristics of the data evolution model itself, adopts a method based on doubt degree and log-likelihood dynamic function to judge the topic dimension in the corpus, calculates the changes between doubt degree and log-likelihood dynamic function under different topic dimensions, and uses M-layer cross-validation to find the topic dimension with the lowest doubt degree and the topic dimension with the largest log-likelihood dynamic function, thereby obtaining different optimal topic dimensions;

[0067] (2) Topic evolution model based on text parsing. This application uses visualization methods to evaluate the effects of modeling under different topic dimensions from the perspective of text parsing. By highlighting the analysis method of high-frequency words in the comparison of "document-term" text parsing, a text parsing method based on data evolution and a term analysis method in the topic space are constructed. At the same time, the optimal topic dimension is determined by the distribution of "topic-topic" and "topic-term" after modeling;

[0068] (3) Topic evolution model based on text classification. Based on the distribution relationship between documents and topics, the text representation is constructed through the topic distribution and used as the feature vector. Different topics are modeled through text classification comparison to determine the optimal topic dimension. At the same time, a comparative experiment is conducted with the traditional vector space model. The feasibility and effectiveness of text classification based on topic dimensionality reduction are demonstrated through experiments.

[0069] 1. Optimal Topic Dimension Estimation for Data Evolution Models

[0070] The first difficulty in establishing a data evolution model is to estimate how many topics M there are in the corpus, and the topic dimension of the data evolution model cannot be known in advance. This application constructs a strategy to determine the topic dimension in the data evolution model based on the change in doubt and the log-likelihood dynamic function. The doubt gradually decreases and tends to be flat as the number of topic dimensions increases, while the log-likelihood dynamic function gradually increases and tends to be flat as the number of topic dimensions increases. As long as the topic dimension is large enough, the doubt must be low enough or the log-likelihood dynamic function is large enough, but such a model has the problem of overfitting. By setting different topics M, observing the current number of topics M, using multi-layer cross-validation doubt and log-likelihood dynamic function, and determining the M value by comparing the visualization graph, it is the optimal topic dimension estimate of the current corpus. The overall block diagram of the optimal topic dimension estimation of the data evolution model is as follows: Figure 1 .

[0071] (1) Data Evolution Model

[0072] Assume that the test text set is represented by M, the terms in the bag-of-words model are represented by w, and the log-likelihood value describes the distribution of the term in the topic space. First, given the prior parameters of the "document-term" matrix, the prior distribution parameter values are determined by using Gibbs sampling. By dividing the corpus into a training set and a test set, the model parameters are trained using the training set, and then the log-likelihood dynamic function of the test set is calculated using VEM estimation.

[0073] Estimating the topic dimension is a process of continuous model comparison, analyzing the changes in the mean of the model's doubt and the mean of the log-likelihood dynamic function when the topic dimension changes, and using the cross-validation method to test the data. When modeling the corpus, the text data is randomly divided into M groups of equal size, one group is selected as the test set, and the remaining M-1 groups of data are modeled to train the topic evolution model. The trained topic evolution model is then used to model the test set, and a set of its doubt and log-likelihood dynamic functions are calculated. Repeating this process multiple times under the same conditions will result in multiple sets of doubt and log-likelihood dynamic functions, and the mean of multiple sets of doubt and log-likelihood dynamic functions is output as the result to complete multi-layer cross-validation to avoid overfitting or underfitting of the model.

[0074] When evaluating the modeling effect of the topic evolution model, it is determined that the lower the doubt, the better, and the larger the log-likelihood dynamic function, the better. After a topic modeling, the doubt and log-likelihood dynamic function are calculated simultaneously. Under the current model parameters, a multi-layer cross-validation is performed to obtain the average doubt and log-likelihood dynamic function. Then, the doubt and log-likelihood dynamic function trends output by the topic dimension analysis are changed. The process of solving the average doubt and log-likelihood dynamic function is as follows:

[0075] Step 1: Given the subject dimension;

[0076] Step 2: Select the Gibbs topic modeling algorithm;

[0077] Step 3: Solve the doubt degree and log-likelihood dynamic function of the current model;

[0078] Step 4: Repeat the M-layer cross-validation 6 times and record the average doubt and log-likelihood dynamic function.

[0079] The topic dimension of the topic evolution model is simulated and determined through the M-layer cross-validation method. At the same time, the two evaluation indicators of doubt degree and log-likelihood function value are combined to determine the topic value, so as to make up for the simple reliance on doubt degree to evaluate the quality of the model.

[0080] 2. Topic Evolution Model Based on Text Parsing

[0081] In order to solve the problem of how to visualize and analyze texts in a corpus, this application analyzes the texts after topic modeling and evaluates the effects of modeling to obtain the optimal modeling topic dimension. First, a topic evolution model is constructed based on the estimated topic dimension corpus modeling to express the relationship between two groups of "document-topic" and "topic-term". One is to express the core of the corpus by extracting terms under the topic as keywords, and to give an evaluation by comparing with the analysis of the text word frequency word cloud; the other is to use the topic similarity evaluation method between topic structures as an evaluation of topic modeling, and to evaluate the modeling effects of different topic dimensions from an intuitive and objective perspective from the data evolution vis visualization analysis. Figure 2 shown.

[0082] (1) Word cloud analysis based on text word frequency

[0083] In word cloud analysis, the weight of terms is measured based on the frequency of text words, and high-weight terms are output as the subject words of the word cloud. The process of generating a word cloud includes two steps: first, the weight of each word is calculated for the preprocessed text or corpus; second, the size of the word in the composition space is estimated according to the size of the word weight and the word cloud is output. When outputting the word cloud, the priority of terms with different weights in the composition space is considered, and the stability and rationality of the term layout are met by using a compact layout method.

[0084] (2) Text parsing based on word cloud

[0085] The corpus is modeled through the data evolution model, and the "topic-term" distribution in the corpus is learned. According to the estimated number of two topics, the corpus is subjected to topic modeling with a topic dimension of 20 and a topic modeling with a topic dimension of 60, respectively, to form two groups of comparisons. The distribution of terms under each topic is analyzed in the modeled corpus. The terms under each topic are distributed according to the probability of p(w|z). Words with high probability are prioritized in each topic space. It is approximately determined that words with high word frequency in different topic spaces are prioritized in front. In the two groups of modeling, 120 words with high probability in the topic space are extracted respectively, and the word frequency is counted, and then displayed in the form of a word cloud.

[0086] (3) Visualization Analysis Based on Topic Evolution Model

[0087] Data evolution vis network visualization is used as a text parsing tool after topic modeling of different topic dimensions. Through multi-scale analysis, the relative distance between objects is grasped, and the expression of data in low-dimensional space is obtained. The distribution of corpus documents and the distribution of terms in the topic space after data evolution modeling are analyzed, and the probability information of terms is expressed by the ranking of terms.

[0088] After preprocessing the text in the corpus, the text content of text_content is formed. Then, topic modeling is performed on the text based on this. The modeling results are visualized through the topic evolution model. The output results are formulated in the form of web pages by writing scripts. The algorithm process of the modeling process with a topic dimension of 20 is as follows:

[0089] (1)text_content# represents the text content after word segmentation;

[0090] (2) Convert text_content into list type and assign it to text_contents;

[0091] (3) Remove the empty items in text_contents and assign them to doc.list;

[0092] (4) Convert doc.list to table type and assign it to term.table;

[0093] (5) Sort term.table in descending order;

[0094] (6) Get the header of term.table;

[0095] (7) Specify the del deletion parameter to be a word with a frequency less than 5;

[0096] (8) Get the header of term.table;

[0097] (9) Remove the table entries in term.table that satisfy del;

[0098] (10) Assign the names in term.table to vocab;

[0099] (11) Specify the method get.terms with parameter x;

[0100] (12) Get the position of index x in vocab;

[0101] (13) index cannot be empty;

[0102] (14) Combine the position of index and the number of times x appears into a vector;

[0103] (15) Assign the result of executing the get.terms method on doc.list to documents;

[0104] (16) Save the results to fit;

[0105] (17) Calculate the probability distribution theta of the document in the dataset;

[0106] (18) Label the result as theta_20;

[0107] (19) Calculate the probability distribution of terms in the topic space;

[0108] (20) Label the result as phi_20;

[0109] (21) Convert the topic terms into a list topic_words;

[0110] (22) Display topic_words;

[0111] (23) Set term weights term.weights;

[0112] (24) Get the length of the document doc.length;

[0113] (25) Generate json variables in web page information;

[0114] (26) Get the document length doc.length and vocabulary vocab;

[0115] (27) Set term weights term.weights

[0116] (28) Set json output serVis;

[0117] The ranking of terms is changed by adjusting the parameter a to help explain the topic. If a is close to 1, the terms at the top are more frequent in the topic space; if a is closer to 0, the terms at the top are more unique in the topic space. By adjusting the size of a, the weight of the terms under the topic is changed, affecting the ranking of the terms. The value of a is determined by the following formula:

[0118] Relevance(w|t)=a*p(w|t)+(1+a)*p(w|t) / p(w) Formula 1

[0119] Based on the relevance of a certain word topic, it is adjusted by a. By comparing the frequency of a given word in the corpus with the topic-specific frequency, it is quickly determined whether it is a highly relevant selected topic; θ represents the vector of topic distribution, p(θ) is the distribution of θ, N represents the number of words in the corpus, w_n is the nth word w, z_n represents the selected topic, p(z|θ) represents the probability of topic z under the topic distribution θ, p(w|z) represents the probability of word w under topic z, expressed as an M×V matrix, M represents the topic, V represents the word in the corpus, α is the parameter describing the distribution p(θ), which affects the distribution of the topic θ vector, and β is the parameter describing the distribution p(w|z), which affects the distribution of the word under topic z;

[0120] The relationship between topics is analyzed through multidimensional scaling. The high-dimensional topic distribution is projected into a two-dimensional space, and the size and distance of the bubble chart are used to describe the differences between topics. The modeling effect is better when M=20.

[0121] From the perspective of the distribution of terms under a topic, the fewer topic structures a term has associated with it in the topic space, the more evenly distributed the "topic-term" is. The modeling effect is evaluated by the relationship between different modeling topic dimensions and the topic similarity obtained after modeling. That is, when the average similarity between topics is the smallest, the model is optimal.

[0122] A large number of experiments have shown that there is an optimal structure in the topic structure implicit in the extracted data set by performing data evolution topic modeling on the corpus. Therefore, the probability distribution of topics in the term space is used to represent the topic vector, the cosine distance is used to represent the difference of the topic, and cosine(z i ,z j ) is smaller, the greater the difference in the subject space:

[0123]

[0124] The quality of the topic structure is characterized by the average similarity of the topic vectors:

[0125]

[0126] The modeling effects of the two are compared directly by comparing the average topic similarities under different topic dimensions. The topic similarities under different topic dimensions are: M = 20, Avg_cosine = 0.0283, M = 60, Avg_cosine = 0.03008073;

[0127] By comparing the average similarities of all topics in the topic evolution model under different topic dimensions, it is determined that when the topic dimension is 20, the difference of topic vectors in the topic space is greater, and it is determined that the modeling performance with a topic dimension of 20 is relatively better.

[0128] 3. Topic Evolution Model Based on Text Multidimensional Clustering

[0129] The "document-topic" matrix obtained after data evolution modeling of the test set in the corpus is used as the training set of the nearest neighbor classifier. The test set is obtained in the same way and predictions are made through the nearest neighbor classifier. First, the text representation of data evolution modeling is constructed by estimating different topic dimensions, and the quality of different topic dimensions is estimated through classification effect feedback; then the text representation of the optimal data evolution modeling is compared with the text representation constructed by the vector space model to verify the feasibility and effectiveness of the data evolution model in classification, such as Figure 3 shown.

[0130] (1) Text classification model based on data evolution dimensionality reduction

[0131] A relatively important step in the text classification problem is to represent the text as a vector. Generally, the obvious problem when using a vector representation based on terms is that there is often a huge spatial dimension. Feature extraction is used to reduce the spatial dimension and thus reduce the dimensionality. The reason for dimensionality reduction is due to two considerations: first, it is for the convenience of calculation. Excessive dimensionality means that time-consuming calculations are involved; second, it is to improve accuracy. In the process of dimensionality reduction, useless terms are removed and more useful features are retained. Regardless of which feature selection algorithm is used, the feature vector that finally represents the text is a part of the original term set, and the representation of the text still only stays at the word level, and there is no in-depth study of the text semantics. The present application constructs a method for text classification by using a data evolution model to represent the text after modeling the corpus.

[0132] like Figure 4 The specific steps are as follows:

[0133] Step 1: Use the data evolution model to model the training document set and obtain the distribution of documents on topics as the vector representation of the documents;

[0134] Step 2: Use the nearest neighbor algorithm to train the vector in step 1;

[0135] Step 3: Use the results in step 1 to model the test document set and obtain the topic distribution in the test document set;

[0136] Step 4: Represent the result as a text vector with a fixed number of text features;

[0137] Step 5: Predict the category of the text vector using the classifier obtained in step 2.

[0138] (2) Text classification model based on the nearest neighbor

[0139] By representing the text in the corpus as a feature vector, the classification method determines that the category of the unknown text is the same as the majority category of T adjacent samples in the feature item space. The number of the most adjacent samples T and the distance function are known. The T value is the square root of the number of training sets, and the distance function is the Euclidean distance.

[0140] like Figure 5 As shown, the algorithm flow is as follows:

[0141] Step 1: First, divide the corpus into a test document set and a training document set, and perform preprocessing;

[0142] Step 2: Extract specific parts of speech for all document sets in the corpus and convert them into text representation based on word vectors;

[0143] Step 3: Select consistent feature vectors based on text word frequency;

[0144] Step 4: For the vector to be classified in the training document set, traverse the distance between the vector and the known classification vector;

[0145] Step 5: Select the nearest vectors in the T vector space and calculate the category weight of the majority vector;

[0146] Step 6: Label the unknown vector as the class with the largest weight for the majority class.

Claims

1. A multi-dimensional clustering topic evolution method for network big data text analysis, characterized by: We construct an evaluation method for estimating topic dimensions in data evolution models from the perspective of text parsing and text classification. By calculating the dynamic function of doubt and log-likelihood under different topic dimensions, we use a data-driven approach to estimate topic dimensions. Then, based on the results of corpus modeling using the estimated topic dimension, we combined text parsing and text classification to determine the optimal topic dimension. Through visualization under the topic evolution model, we visually displayed the text content in the form of word clouds and bubble charts, comparing word frequency with documents, word frequency with classification, and topic distribution based on data evolution. This allowed us to draw a dynamic comparison between the modeling effects of the log-likelihood dynamic function and the modeling effects of the doubt level. A1 - Estimating the Optimal Topic Dimension of the Topic Evolution Model: Combining language model evaluation metrics with the structural characteristics of the data evolution model, we use a method based on doubt and log-likelihood dynamic functions to determine the topic dimension in the corpus. We calculate the changes in doubt and log-likelihood dynamic functions under different topic dimensions, and use M-layer cross-validation to determine the topic dimension with the lowest doubt and the topic dimension with the highest log-likelihood dynamic function, thereby obtaining different optimal topic dimensions. A2 - Topic Evolution Model Based on Text Parsing: This includes word cloud analysis based on text word frequency, text parsing based on word clouds, and visualization analysis based on the topic evolution model. From the perspective of text parsing, visualization methods are used to evaluate the effectiveness of modeling under different topic dimensions. By highlighting high-frequency words in "document-term" text parsing, a text parsing method based on data evolution and a term analysis method in the topic space are constructed. The optimal topic dimension is determined by analyzing the "topic-topic" and "topic-term" distributions after modeling. A3 - Topic evolution model based on text classification: This includes a text classification model based on data evolutionary dimensionality reduction and a text classification model based on nearest neighbors. Based on the distribution relationship between documents and topics, the model constructs a text representation using the topic distribution and uses this as a feature vector. Different topics are modeled through text classification comparison to determine the optimal topic dimension. The topic evolution model is optimized through the semantic expression of high-probability terms and specific comparison results. The modeling effect of the topic dimension selection is evaluated based on different indicators. Data Evolution Model: Assume that the test document set is represented by D, the terms in the bag-of-words model are represented by w, and the log-likelihood value describes the distribution of the terms in the topic space. First, given the prior parameters of the "document-term" matrix, the prior distribution parameter values are determined by Gibbs sampling. The corpus is divided into training and test sets, the model parameters are trained using the training set, and then the log-likelihood dynamic function of the test set is calculated using VEM estimation. Estimating the topic dimension is a process of continuous model comparison. It analyzes the changes in the mean of the model doubt and the mean of the log-likelihood dynamic function when the topic dimension changes. The cross-validation method is used to test the data. When modeling the corpus, the text data is randomly divided into E groups of equal size. One group is selected as the test set. The remaining E-1 groups of data are used to model and train the topic evolution model. The trained topic evolution model is then used to model the test set. A set of doubt and log-likelihood dynamic functions are calculated. Repeating this process multiple times under the same conditions will obtain multiple sets of doubt and log-likelihood dynamic functions. The mean of the multiple sets of doubt and log-likelihood dynamic functions is output as the result to complete the multi-layer cross-validation. When evaluating the modeling effect of the topic evolution model, it is determined that the lower the doubt, the better, and the larger the log-likelihood dynamic function, the better. After a topic modeling, the doubt and log-likelihood dynamic function are calculated simultaneously. A multi-layer cross-validation is performed under the current model parameters to obtain the average doubt and log-likelihood dynamic function. Then, the doubt and log-likelihood dynamic function trends output by the topic dimension analysis are changed. The process of solving the average doubt and log-likelihood dynamic function is as follows: Step 1: Given the subject dimension; Step 2: Select the Gibbs topic modeling algorithm; Step 3: Solve the doubt degree and log-likelihood dynamic function of the current model; Step 4: Repeat the M-layer cross-validation 6 times and record the average doubt and log-likelihood dynamic function; The topic dimension of the topic evolution model is simulated and determined by the M-layer cross-validation method, and the topic value is determined by combining two evaluation indicators, the doubt degree and the log-likelihood function value.

2. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Optimal topic dimension estimation of the data evolution model: By setting different topics B and observing the current number of topics B, multi-layer cross-validation doubt and log-likelihood dynamic functions are used to determine the B value by comparing the visualization graph, which is the optimal topic dimension estimate of the current corpus.

3. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Topic evolution model based on text parsing: By analyzing the text after topic modeling and evaluating the modeling effect, the optimal modeling topic dimension is obtained. First, a topic evolution model is constructed based on the estimated topic dimension corpus modeling to express two sets of "document-topic" and "topic-term" relationships. One set expresses the core of the corpus by extracting terms under the topic as keywords, and gives an evaluation through comparison based on text word frequency word cloud analysis; the other set evaluates the topic modeling through the topic similarity evaluation method between topic structures, and evaluates the modeling effect of different topic dimensions from an intuitive and objective perspective through data evolution vis visualization analysis.

4. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Word cloud analysis based on text word frequency: In word cloud analysis, the weight of terms is measured based on the frequency of the text word. High-weighted terms are output as the key words of the word cloud. The process of generating a word cloud includes two steps: first, the weight of each word is calculated for the preprocessed text or corpus; Secondly, the word cloud is output based on the size of the word weight and the proportion of the word in the composition space is estimated. When outputting the word cloud, the priority of words with different weights in the composition space is considered, and then a compact layout method is used to ensure the stability and rationality of the word layout.

5. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Word cloud-based text parsing: The corpus is modeled using a data evolution model to learn the "topic-term" distribution within the corpus. Based on two estimated topic numbers, topic modeling with a topic dimension of 20 and a topic dimension of 60 is performed on the corpus, forming two comparison groups. The distribution of terms under each topic is analyzed in the modeled corpus. The terms under each topic are distributed according to the probability of p(w|t). Words with high probabilities are prioritized in each topic space. Words with high frequency in different topic spaces are prioritized in the front. In the two modeling groups, 120 words with high probabilities in each topic space are extracted and their frequencies are counted, and then displayed as word clouds.

6. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Visual analysis based on the topic evolution model: Data evolution vis network visualization is used as a text parsing tool after topic modeling at different topic dimensions. Through multi-scale analysis, the relative distances between objects are grasped, and the data expression in low-dimensional space is obtained. The distribution of corpus documents and the distribution of terms in the topic space after data evolution modeling are analyzed, and the probability of terms is expressed by the ranking of terms. After preprocessing the text in the corpus, the text content of text_content is formed. Then, topic modeling is performed on the text based on this. The modeling results are visualized through the topic evolution model, and the output results are formulated into a web page by writing a script; The ranking of terms is changed by adjusting the parameter a to help explain the topic. If a is close to 1, the terms at the top are the ones with the highest frequency in the topic space. If a is closer to 0, the terms at the top are the ones that are unique to the topic space. By adjusting the value of a, the weight of the terms under the topic is changed, which affects the ranking of the terms. The value of a is determined by the following formula: Relevance(w|t)=a*p(w|t)+(1+a)*p(w|t) / p(w) Formula 1 Based on the relevance of a word topic, mediated by a, by comparing the frequency of a given word in the corpus with the topic-specific frequency, it is quickly determined whether it is highly relevant to the selected topic; The relationship between topics is analyzed by multidimensional scaling, which projects the high-dimensional topic distribution into a two-dimensional space and describes the differences between topics by the size and distance of the bubble chart. From the perspective of the distribution of terms under a topic, the fewer topic structures a term has associated with it in the topic space, the more evenly distributed the "topic-term" distribution is. The modeling effect is evaluated by the relationship between different modeling topic dimensions and the topic similarity obtained after modeling. That is, the model is optimal when the average similarity between topics is the smallest. The probability distribution of topics in the term space is used to represent the topic vector, the cosine distance is used to represent the difference of topics, and cosine(z i ,z j ) is smaller, the greater the difference in the subject space: The quality of the topic structure is characterized by the average similarity of the topic vectors: The modeling effects of the two are compared directly by comparing the average topic similarities under different topic dimension modeling, and the average similarities of all topics in the topic evolution model under different topic dimensions are compared.

7. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Topic evolution model based on text multidimensional clustering: The "document-topic" matrix obtained after data evolution modeling of the test set in the corpus is used as the training set of the nearest neighbor classifier. The test set is obtained in the same way and predictions are made by the nearest neighbor classifier. First, the text representation of the data evolution model is constructed by estimating different topic dimensions, and the quality of different topic dimensions is estimated through classification feedback. Then, the optimal text representation of the data evolution model is compared with the text representation constructed by the vector space model.

8. The multi-dimensional clustering topic evolution method for network big data text analysis according to claim 1 is characterized in that: Text classification model based on data evolutionary dimensionality reduction: This method uses the data evolution model to model the text representation of the corpus and then classify the text. The specific steps are as follows: Step 1: Use the data evolution model to model the training document set and obtain the distribution of documents on topics as the vector representation of the documents; Step 2: Use the nearest neighbor algorithm to train the vector in step 1; Step 3: Use the results in step 1 to model the test document set and obtain the topic distribution in the test document set; Step 4: Represent the result as a text vector with a fixed number of text features; Step 5: Predict the category of the text vector using the classifier obtained in step 2.

9. The method for multi-dimensional clustering and topic evolution of network big data text analysis according to claim 1 is characterized in that: The nearest neighbor-based text classification model represents the text in the corpus as a feature vector. In the classification process, the category of the unknown text is assumed to be the same as the majority category of T adjacent samples in the feature item space. The number of nearest neighbor samples T and the distance function are known. The T value is the square root of the number of training sets, and the distance function is the Euclidean distance. The algorithm flow is as follows: Step 1: First, divide the corpus into a test document set and a training document set, and perform preprocessing; Step 2: Extract specific parts of speech for all document sets in the corpus and convert them into text representation based on word vectors; Step 3: Select consistent feature vectors based on text word frequency; Step 4: For the vector to be classified in the training document set, traverse the distance between the vector and the known classification vector; Step 5: Select the nearest vectors in the T vector space and calculate the category weight of the majority vector; Step 6: Label the unknown vector as the class with the largest weight for the majority class.

Images