Network public opinion text classification method based on topic recognition

Abstract

The application discloses a network public opinion text classification method based on theme identification, and specifically comprises the following steps: step 1, selecting an original data set and processing the data set; step 2, performing theme identification on the data processed in step 1 to obtain text data related to a network public opinion theme; step 3, performing adversarial training on the text data obtained in step 2 by using a hybrid data enhancement technology; step 4, removing irrelevant words from the data processed in step 3; and step 5, importing the data processed in step 4 into a TextCNN model to perform text classification. The application solves the problems that the comment data crawled in the field of law-related public opinion in the network space is mostly short texts and the core content is sparse, and the text semantic expression of law-related network public opinion is implicit and professional terms are difficult to identify.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer natural processing, and relates to a network public opinion text classification method based on theme identification. BACKGROUND

[0002] With the popularity of the Internet and the rapid development of computer-related technologies, the presentation mode of social public opinion has gradually changed from word of mouth or passive information acquisition from newspapers, radios and televisions to virtual network space. Everyone can become an active spreader of public opinion through a smart phone or a computer keyboard, thereby forming network public opinion under the background of the Internet. Among many network public opinion classifications, the network public opinion related to law is extremely different, involves various language terms, is obscure and difficult to understand, and has a serious and far-reaching negative impact on the nature of social public opinion related to law, which seriously threatens the safety of citizens' person and property and the health of their minds.

[0003] With the continuous development of deep learning in the field of computers, the NLP related technology improves the understanding and processing ability of various text languages, and the existing text classification technology has excellent performance in the semantic information mining of text. The existing text classification models include Fasttext, DPCNN, TextRNN, TextRCNN, BiLSTM_Attention and TextCNN, among which TextCNN has no change in network structure compared with the traditional CNN network of image. It only has one layer of convolution and one layer of max-pooling, and finally connects the output to the softmax to n-classify. However, some of these models have the disadvantages of low efficiency and unsuitability for short text processing, and lack of consideration for the classification of legal-related texts with many professional terms and obscure words. SUMMARY

[0004] The purpose of the application is to provide a network public opinion text classification method based on theme identification, which solves the problems of short text and sparse core content of the crawled comment data in the field of network public opinion related to law, and the problems of obscure semantic expression and difficult-to-identify professional terms of network public opinion text related to law.

[0005] The technical solution adopted by the application is a network public opinion text classification method based on theme identification, which specifically includes the following steps:

[0006] Step 1, selecting an original data set and processing the data set;

[0007] Step 2, performing theme identification on the data processed in step 1 to obtain text data related to the theme of network public opinion;

[0008] Step 3, the text data obtained in step 2 is subjected to adversarial training by using a mixed data enhancement technique;

[0009] Step 4, irrelevant words are removed from the data processed in step 3;

[0010] Step 5, the data processed in step 4 is imported into a TextCNN model for text classification.

[0011] The application also has the characteristics that:

[0012] The specific process of step 1 is:

[0013] Step 1.1, user comments of the public opinion topic to be processed are crawled from the public opinion platform as the original data set, the text splicing operation of the topic post is performed on the original data in json format by using pandas in Python, and the format conversion processing is performed to save the data in csv format;

[0014] Step 1.2, the topic post obtained in step 1.1 is subjected to Chinese text data segmentation by using Jieba segmentation tool, and then the punctuation symbols and stop words after text segmentation are manually deleted;

[0015] Step 1-3, the text data processed in step 1.2 is subjected to classification label category division, and is divided into n label categories;

[0016] Step 1-4, according to the label divided in step 1-3, the text processed in step 1.2 is subjected to label annotation;

[0017] Step 1-5, a term dictionary of the classified network public opinion topic text is manually constructed, wherein the words in the dictionary are taken from the text preprocessed by segmentation in step 1-2 after manual discrimination.

[0018] The specific process of step 2 is:

[0019] Step 2.1, the text processed in step 1.2 is spliced according to the label annotated in step 1.3, and a document-level data set in the form of topic post is constructed; wherein it is assumed that a corpus library contains D texts, each corpus library text contains B word pairs, each corpus library text contains N words, and each corpus library text involves K topics;

[0020] Step 2.2, the document-level data set after topic splicing in step 2.1 is input into a BTM model as an input document, the BTM model first generates a word pair corpus, and the word pair mode is converted from short text by using the corpus information to describe the topic distribution and word distribution from the level of the corpus.

[0021] The specific process of step 2.2 is:

[0022] Step 2-2-1, obtaining the word distribution ф in a specific topic through the calculation of the mathematical probability model of Dirichlet distribution k ~ Dir(β), where ф is the topic-word probability distribution, i.e. k = P(w | r), Dir is the mathematical probability Dirichlet distribution used in the BTM model, β is the Dirichlet prior parameter, the label topic is denoted by k, the word vector formed by the corpus is denoted by w, and the topic distribution is denoted by r, and P(w | r) represents the joint probability distribution of w and r;

[0023] Step 2-2-2, randomly obtaining a topic distribution θ from the entire text data set obtained in step 2-1 d ~ Dir(α), where θ is the document-topic probability distribution, the corpus text is denoted by d, Dir is the mathematical probability Dirichlet distribution used in the BTM model, and α is the Dirichlet prior parameter;

[0024] Step 2-2-3, for a word pair b in the word pair set B, b ∈ B, randomly extracting a topic k from the topic distribution θ to obtain a topic assignment k ~ Multi(θ k ); randomly extracting a word pair biterm from the topic k, i.e. i and b j , b i , b j ~ Multi(ф k ), where Multi(*) represents a multinomial distribution with * as the parameter, θ k represents the probability distribution of K topics in the entire corpus, i.e. k = P(k);

[0025] The topic distribution of the text is represented by the following formula (1):

[0026] P(d) = ∑ k P(k)P(b i | k)P(b j | k) = ∑ k θ k ф i|k ф j|k (1);

[0027] Where, ∑ k * represents the summation operation for each topic k related to the (*) operation; θ k , P(k) represents the probability distribution of the kth topic in the entire corpus; P(b # | k) represents the topic distribution corresponding to the text taken word b # , since the word pair b = (b i , bj ), so here # takes i or j; ф #|k represents the probability distribution of word b under topic k; #

[0028] According to formula (1), the joint distribution probability of word pair b=(b i , b j ) is calculated, the corpus contains K topics and |B| word pairs, and word pair b is composed of b i and b j , therefore, the probability P(B) of the whole corpus of the two-word set B is expressed by formula (2) as follows:

[0029] P(B)=Π (i,j) ∑ k P(k)P(b i |k)P(b j |k) (2);

[0030] Step 2-2-4, according to the expected probability of the word pair in the document, the inference of the document topic is carried out, and the conditional probability distribution of the topic in the document is calculated:

[0031] P(k|d)=∑ b P(k|b)P(b|d) (3);

[0032] Where P(k|d) represents the conditional probability distribution of topic k in document d, ∑ b (*) represents the summation operation of the (*) operation for each word pair b in the text corpus, P(k|b) represents the conditional probability distribution of word pair b corresponding to topic k, and P(b|d) represents the conditional probability distribution of word pair b contained in text d;

[0033]

[0034] Where P(k|b) represents the conditional probability distribution of topic k in document b, P(b # |k) represents the conditional probability distribution of b # in the word pair contained under topic k, since the word pair b=(b i , b j ), so here # takes i or j, and ∑ k (*) represents the summation operation of the (*) operation for topic k; in order to obtain the conditional probability P(b|d) of the word pair in the document, the word pair can be used as an intermediate medium to calculate:

[0035]

[0036] Where n d ​(b) represents the total frequency of co-occurrence word pair b in document d, and in the public opinion text data set, P(b|d) can be regarded as the uniform distribution of all co-occurrence word pairs b in document d;

[0037] Step 2-2-5, Gibbs sampling is performed on the network public opinion text data based on the MCMC algorithm, and when the Gibbs sampling converges, the convergence condition is that the transition probability matrix can converge to a stable probability distribution, that is, the sampling theme of all words is obtained, and the corresponding relationship between all the sampled words and themes is obtained to obtain the document-theme distribution θ d and the theme-word distribution ф k , so as to filter out irrelevant data from the crawled text data and only leave the text data related to the network public opinion theme.

[0038] The specific process of step 3 is as follows:

[0039] Step 3-1, first, the data set after the irrelevant text is preliminarily filtered by the BTM model in step 2 is input to the translation software for language conversion, and the converted text is trained with the original data set by the generative adversarial network, wherein the overall optimization objective function of the generative adversarial network is represented as:

[0040] arg min L max D V(L,D) (6);

[0041] Wherein, arg refers to the principal value of the argument of a complex number, formula (6) represents the value of the variable when L in V(L,D) is minimized and D is maximized;

[0042] The function V(L,D) in formula (6) is a measure of the discriminability of the discriminator, which is represented as:

[0043]

[0044] Wherein, in the generative adversarial network, the generator is abstracted as G(·), and the sample distribution generated by the generator can be denoted as P g (x);The real sample distribution is denoted as P d (x), and the discriminator is abstracted as a function D(·), which can be used to implicitly represent P g (x) and P d (x);

[0045] Step 3-2, fix the generator, and get the maximum value of the discriminator, that is, take the derivative of the polynomial and take the extreme value, then the analytical expression of the discriminator is:

[0046]

[0047]

[0048] Step 3-3, when the discriminator reaches the optimal, the actual optimization of the generative adversarial network is the JS divergence, wherein the JS divergence represents the gap between the generated sample and the real sample, and when the generator distribution produces a distribution P L (x) consistent with the real distribution P data (x), the generator reaches the optimal and realizes data enhancement.

[0049] The specific process of step 4 is as follows:

[0050] Step 4-1, by predicting the probability P(R=r|C=c) of the context-specific word r of the center word c in the text data after the adversarial training, wherein the size of the text vocabulary is X, the size of the hidden layer is Y, the neurons of adjacent layers are fully connected, the input layer is encoded by one-hot word vector x={x1,…,x X}, and it is required that only one in each word vector is 1 and the rest are 0;

[0051] Step 4-2, by randomly sampling the word pairs in the text data to form positive and negative sample pairs, for each word encountered in the training original text, it is set that they all have a certain probability to be deleted from the text, and this deletion probability is related to the frequency of the word;

[0052] Step 4-3, then learn the weight by the back propagation algorithm and gradient descent, realize the probability maximization when predicting w(c,j) at the cth context position, and the loss function L can be used:

[0053]

[0054] The objective function is:

[0055]

[0056] Wherein,

[0057] The beneficial effect of the present application is that the present application researches the text classification task in the field of network public opinion, with the popularity of mobile Internet, the formation of network public opinion is more rapid, more extensive, and has a huge impact on society. But lack of effective automated data review technology, so that it is difficult to realize the rapid and efficient identification of network public opinion information. Text classification as an important means of network information review, can provide strong help for relevant departments to strengthen network supervision and purify network environment, and the performance of text classification algorithm directly affects the difficulty and quality of the above work. The present application improves the common benchmark model in the classification task in combination with the characteristics of the network public opinion comment data set, so as to better adapt to the efficient text classification demand under the scene of implicit semantic expression and more terms. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is the model structure diagram used in the network public opinion text classification method based on topic recognition of the present application;

[0059] Figure 2 is the BTM topic model structure decomposition diagram used in the network public opinion text classification method based on topic recognition of the present application;

[0060] Figure 3 is the decomposition diagram of the BackTranlate&&AdversarialTraining data enhancement method used in the network public opinion text classification method based on topic recognition of the present application;

[0061] Figure 4 is the decomposition diagram of the Skip-Gram model in which the negative sampling mechanism is introduced in the word embedding model in the network public opinion text classification method based on topic recognition of the present application. DETAILED DESCRIPTION

[0062] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0063] The network public opinion text classification method based on topic recognition of the present application solves the problems of short text sparsity in the field of network public opinion related to law and the problem that the semantic expression of network public opinion text related to law is implicit and professional terms are difficult to identify. The optimization method based on TextCNN model is used, the specific embodiment is to innovatively preprocess in the text classification stage, then the negative sampling mechanism is introduced in the embedding layer, and finally the Max-CovariancePooling method of covariance is integrated in the pooling stage, so as to improve the text classification performance and efficiency of network public opinion related to law from multiple angles.

[0064] The network public opinion text classification method based on topic recognition of the present application, as Figure 1As shown, specifically includes the following steps:

[0065] Step 1, from micro-blog, blog, etc. Public opinion platform to crawl the user real comments of the public opinion theme to be processed as the basic data set of the text processing model, and then the real comment text data of the user involved in the network public opinion field of the text classification processing is preprocessed, labeled and professional term dictionary is constructed.

[0066] Step 1-1, since the original text data set is the user comment of the platform, the comment word length is from several words to one or two hundred words, first of all, the pandas in Python is used to splice the text of the theme post of the original data in json format, and the format conversion processing is carried out to save the csv format for the subsequent text representation.

[0067] Step 1-2, using Jieba word segmentation tool to perform Chinese text data word segmentation on the theme post obtained in step 1, and then manually deleting the punctuation and stop words after the word segmentation of the spliced text, thus completing the text preprocessing step.

[0068] Step 1-3, next, according to the actual demand, the user real comments of the public opinion theme after the splicing of step 1-1 and the preprocessing of step 1-2 are classified and divided into n label categories (where n is an integer and n≥2), the n label categories and the rule division here are the personal individualization setting of the processor on the basis of having certain understanding of the processing text, which is the setting of the text theme label. Let the number of label themes named in this step be k.

[0069] Step 1-4, next, the text after the preprocessing step in step 1-2 is manually labeled with the multiple classification labels set, thus completing the label labeling step.

[0070] Step 1-5, since the network public opinion text to be processed may involve slang and professional terms with deeper meanings, in order to classify the subsequent text, it is necessary to manually construct a term dictionary of the classified network public opinion topic text, wherein the words in the dictionary are taken from the text preprocessed by word segmentation in step 1-2 after artificial discrimination, and are arranged from the division of slang and professional language in the text. Slang refers to words that express a particular meaning with agreed words in the field of study, and professional terms refer to the uniform industry designation for a particular thing in the field of study, which is difficult for outsiders to understand. The term dictionary can also collect existing related rich professional corpus on the Internet to supplement the terms of the classified text topic. That is, the term dictionary is derived from manual extraction of text terms and Internet professional corpus supplement, until the constructed term dictionary covers all slang and professional terms in the studied text, and the construction step of the professional term dictionary is completed.

[0071] The network public opinion topic professional term dictionary completed by the above steps can deeply mine the deep information of specific words from the original text data, and the professional term dictionary auxiliary model can realize the prediction of classification, which lays a foundation for subsequent text processing.

[0072] Step 2, since there is a serious data text imbalance problem between the corresponding text classification data sets formed by the pre-processing and artificial labeling of step 1, therefore, as shown in Figure 2 , this step introduces BTM (Biterm Topic Model-Word Pair Topic Model) to identify the theme of the text data, so that the detection efficiency of public opinion text under data imbalance is significantly improved.

[0073] Step 2-1, the text data set obtained by pre-processing in step 1 is further spliced according to the respective subject post, i.e. the labeled label, to obtain spliced documents of text data labeled with the same class, thereby constructing document-level data in units of subject posts, that is, a subject document is obtained for each subject label, in order to generate a subject post document by the BTM topic model in the following step, and to filter out the generated subject words containing public opinion topics and all subject posts under this category. It is assumed that the corpus contains D texts, each corpus text contains B word pairs, each corpus text contains N words, and each corpus text involves K topics (uppercase represents the total set, lowercase represents the individual, the same below).

[0074] Step 2-2, the document-level dataset after the subject splicing in step 2-1 is input into the BTM model, in which a word pair corpus is automatically generated, the public opinion text information is converted into a word pair mode by using rich corpus information, and the theme distribution and the word distribution are described from the level of the corpus. The specific generation process of the corpus in BTM can be described as the following steps, and in the actual design process of the experiment of the BTM model, the Scikit-learn library in Python needs to be used and referred to:

[0075] Step 2-2-1, the word distribution under a specific theme is obtained by calculation of the Dirichlet distribution mathematical probability model k ~ Dir(β), where ф is the theme-word probability distribution, that is, ф k = P(w | r), Dir is the Dirichlet distribution used in the BTM model, β is the Dirichlet prior parameter, the label theme is represented by k, the word vector formed by the corpus is represented by w, and the theme distribution is represented by r, and P(w | r) represents the joint probability distribution of w and r.

[0076] Step 2-2-2, a theme distribution θ is randomly obtained in the entire text data set obtained in step 2-1 d ~ Dir(α), where θ is the document-theme probability distribution, the corpus text is represented by d, and Dir is the Dirichlet distribution used in the BTM model, and α is the Dirichlet prior parameter.

[0077] Step 2-2-3, for a word pair b in the word pair set B, b ∈ B, a theme k is randomly extracted from the theme distribution θ, and a theme assignment k ~ Multi(θ k ) is obtained; a word pair biterm is randomly extracted from the theme k, that is, b i and b j , b i , b j ~ Multi(ф k ), where Multi(*) represents a multinomial distribution with * as the parameter, θ k represents the probability distribution of K themes in the entire corpus, that is, θ k = P(k).

[0078] According to the above three generation processes of steps 2-2-1 to 2-2-3, the following formula can be obtained:

[0079] P(d) = ∑ k P(k)P(b i | k)P(b j | k) = ∑ k θ k фi|k ф j|k (1);

[0080] where P(d) represents the topic distribution of the text; ∑ k * represents the summation operation of the (*) operation for each topic k; θ k P(k) represents the probability distribution of the kth topic in the entire corpus; P(b # |k) represents the conditional probability distribution of the word pair b # corresponding to the topic distribution, and i or j is taken here because the word pair b=(b i , b j ); ф #|k represents the probability distribution of the word b # under the topic k.

[0081] According to the above formula (1), the joint distribution probability of the word pair b=(b i , b j ) is calculated. The corpus contains K topics and |B| word pairs, and the word pair b is composed of b i and b j . Thus, the probability of the entire corpus of the word pair set B can be calculated using the formula:

[0082] P(B) = Π (i,j) ∑ k P(k)P(b i |k)P(b j |k) (2);

[0083] Step 2-2-4, according to the expected probability of the word pair in the document, the inference of the document topic is carried out, and the conditional probability distribution of the topic in the document is calculated:

[0084] P(k|d) = ∑ b P(k|b)P(b|d) (3);

[0085] where P(k|d) represents the conditional probability distribution of the topic k in the document d, ∑ b (*) represents the summation operation of the (*) operation for each word pair b in the text corpus, P(k|b) represents the conditional probability distribution of the word pair b corresponding to the topic k, and P(b|d) represents the conditional probability distribution of the word pair b contained in the text d.

[0086]

[0087] where P(k|b) represents the conditional probability distribution of the topic k in the document b, P(b # |k) represents the conditional probability distribution of the word pair b #conditional probability distribution of the word pair b = (b i , b j ), where i or j, ∑ k (*) represents the summation operation of the (*) operation for the topic k. In order to obtain the conditional probability P(b|d) of the word pair in the document, the word pair can be used as an intermediate medium to calculate:

[0088]

[0089] where n d (b) represents the total frequency of the co-occurrence word pair b in the document d, and in the public opinion text data set, P(b|d) can be regarded as the uniform distribution of all co-occurrence word pairs b in the document d.

[0090] Step 2-2-5, since the BTM model uses the maximum likelihood method to obtain the marginal probability distribution, the result is quite complex, and it is impossible to accurately calculate the parameters {θ, ф}. Therefore, the Gibbs sampling based on the MCMC algorithm can be used to sample the network public opinion text data to solve the BTM model. After the Gibbs sampling converges, the topic of the i-th word is obtained, and similarly, when the topics of all words are obtained by sampling, the word distribution of each topic can be obtained by counting the topic counts of all words. Then, by counting the topic counts of the words corresponding to each document, the topic distribution of each document can be obtained.

[0091] Gibbs sampling is to simulate the joint distribution by sampling the conditional distribution, and then derive the conditional distribution directly from the simulated joint distribution, and then cycle. The main idea of Gibbs sampling is to alternately replace the values of one variable with the values of another variable. In the BTM model, three types of latent variables need to be sampled, namely k, θ and ф. When using the folded Gibbs sampling technique, θ and ф can be integrated from the conjugate prior parameters α and β.

[0092] In Gibbs sampling, the BTM model randomly selects the initial state of the Markov chain, and then calculates the conditional distribution P(k|k i , b j ) of each bitermb = (b -b , b b ), where B is the set of word pairs biterm. The model obtains the conditional probability by using the chain rule of the joint probability of the entire corpus:

[0093]

[0094] where k b represents the topic k corresponding to the b-th word pair, and (k -b, B) represents the topic assignment of all word pairs biterm except b, P(k b |k -b , B) represents the conditional distribution probability of a word pair in the word pair set B, P(k -b , B) represents the probability distribution of the topic k of the word pair set B except the word pair b, α represents the parameter of the Dirichlet distribution of the document-topic probability, P(B|k) represents the conditional probability distribution of the word pair B under the topic k, P(k) represents the probability distribution of the topic k, P(B -b |k -b ) represents the conditional probability distribution of the word pair set B except the word pair b under the condition of the topic k of the word pair b, P(k -b ) represents the probability distribution of the topic k except the word pair b.

[0095] In the formula, P(B|k) can be obtained by integrating Φ, wherein Φ is composed of K topic-word distributions, and the calculation method of P(B|k) is as follows:

[0096]

[0097] In formula (7), Γ(·) is a standard Gamma function, and P(k) can be obtained by integrating α, wherein

[0098]

[0099] After P(B -b |k -b ) and P(k -b ) are calculated by analogy, they are substituted into the formula, and according to the expression Γ(x+1)=xΓ(x), the probability distribution formula required by the Gibbs sampling in the BTM model can be obtained:

[0100]

[0101] Wherein, n k represents the number of word pairs bitermb assigned to the topic k, n w|r represents the number of feature words w assigned to the topic r, and M represents M different words in the entire corpus. Once it is confirmed that the word pair bitermb is assigned to a certain topic k, the two words b i and b j under bitermb will be assigned to the topic k at the same time. The topic-word distribution ф and the topic-document distribution θ in the corpus can be easily estimated according to the following two formulas.

[0102]

[0103]

[0104] where K is the size of the number of topics, B represents the number of biterms in the entire corpus, ф w|r is the probability of word w under topic r, θ k is the probability of topic k.

[0105] When the Gibbs sampling converges, the condition for convergence is that it can converge to a stable probability distribution through the transition probability matrix, that is, the sampling topic of all words can be obtained. Using the correspondence between all the sampled words and topics, we can obtain the document-topic distribution θ d and the topic-word distribution ф k . Thus, irrelevant data is filtered out from the massive crawled text data, leaving only text data related to network public opinion topics, thereby reducing the burden of subsequent work.

[0106] In addition, regarding the Markov chain algorithm: the algorithm first constructs a Markov chain that meets the corresponding requirements based on known conditions. When it meets the condition of gradually converging to a stationary state distribution, it becomes the posterior distribution of the parameter to be estimated. Then, based on the existing Markov chain, samples that meet the posterior standard distribution are generated. Finally, estimation is performed based on the above existing samples. The specific operation steps are as follows:

[0107] Assume that the probability function is P(h), where h represents a state, and T(h'|h) represents the probability of state h transitioning to state h' on Markov chain T. The condition for Markov chain T to meet the stationary distribution at a certain time is:

[0108] P(h t )T(h t-1 |h t )=P(h t-1 )T(h t |h t-1 ) (12);

[0109] where P(h t ) is the state distribution probability at time t, P(h t-1 ) represents the state distribution probability at time t-1, T(h t |h t-1 ) represents the probability of state h t-1 transitioning to state h t , and T(h t-1 |h t ) represents the probability of state h t transitioning to state h t-1 .

[0110] At this time P(h) is a stationary distribution, and the state of the Markov chain at this time changes from convergence to stationary.

[0111] Step 3, as shown in the figure, the application introduces a mixed data enhancement technology, named BackTranlate&&Adversarial Training method, to mix data enhancement methods, including back translation and the use of adversarial training model, so as to further improve the model generalization ability on the basis of the subject model identified in step 2. Figure 3

[0112] Step 3-1, first, according to the data set after the preliminary filtering of irrelevant text processed by the BTM model in step 2, input to the translation software for language conversion, first translated into other language, this process can try to translate multiple foreign languages, and then translated back to Chinese. Back translation is not only directly related to language translation, but also involves the culture behind the language, so it can dig deeper text information. Finally, the back translated text and the original data set are trained by the generative adversarial network, and the overall optimization objective function of the generative adversarial network is represented as:

[0113] arg min L max D V(L,D) (13);

[0114] Wherein, arg refers to the principal value of the argument of complex number, the above formula represents the value of the variable when L in V(L,D) formula is minimum and D is maximum.

[0115] The function V(L,D) in the above formula is a measure of the discriminant ability of the discriminator, which can be represented as:

[0116]

[0117] Wherein, in the generative adversarial network, the generator is abstracted as G(·), and the sample distribution generated by the generator can be denoted as P g (x). Correspondingly, the real sample distribution is denoted as P d (x). The discriminator is abstractly represented as a function D(·), which outputs a scalar to represent the judgment of the authenticity of the current input data. Through sufficient data sampling, the function D(·) can be used to implicitly represent P g (x) and P d (x).

[0118] Step 3-2, fix the generator, and get the maximum value of the discriminator, that is, take the derivative of the polynomial and take the extreme value, then the analytical expression of the discriminator is:

[0119] ​

[0120] Substituting them in, we get:

[0121]

[0122] Step 3-3: Next, when the discriminator reaches its optimal state, that is, when the condition of the arg(*) function is met, the generative adversarial network actually optimizes the Jensen-Shannon divergence (JS divergence for short), where the JS divergence represents the difference between the generated samples and the real samples. When the generator distribution produces the distribution P... L (x) and the true distribution P data The generator reaches its optimal state when (x) is consistent. The generator function is abstracted as G(·), and the distribution of the samples it generates can be denoted as P. L (x); the true sample distribution is denoted as P. data (x), the discriminator function is abstracted as D(·), which outputs a scalar to represent the judgment of the authenticity of the current input data, thereby solving the problem of the sparse related words under the topic k, realizing the supplementation of some imbalanced label data, and constructing a complete and comprehensive text classification dataset.

[0123] The BackTranslate & Adversarial Training method largely prevents model overfitting, compensates for imbalances in some label data, achieves balance in the sampling process, and constructs a complete and comprehensive text classification dataset. This data augmentation technique enhances the data through backtranslation and adversarial training. Compared to other augmentation methods, it employs a hybrid approach, making it more suitable for text processing under current online public opinion themes, leveraging the advantages of different methods and integrating them.

[0124] Step 4: As Figure 4 As shown, based on the preliminary balancing of the lexical text imbalance problem under different topics in step 3 of the adversarial training text dataset, the TextCNN model, which introduces a negative sampling mechanism in the word embedding part, is used to improve the training speed of the Skip-Gram model, thereby improving the running efficiency of the online public opinion text classification model.

[0125] Step 4-1: The Skip-Gram model is used to predict the context word corresponding to a given center word. It predicts the probability P(R = r | C = c) of the context-specific word r for a center word c in the adversarially trained text data. This assumes the text vocabulary size is X, the hidden layer size is Y, adjacent neurons are fully connected, and the input layer uses one-hot encoding of the word vector x = {x1, ..., x...}. X The expression} indicates that each word vector must have only one 1 and the rest must be 0.

[0126] The weight values from the input layer to the hidden layer can be represented by an X x Y matrix W, where each row of the W matrix represents a Y-dimensional word vector associated with the input layer, denoted as X w . Specifically, given a context (a word), if the kth element of the word vector x k = 1 and the rest are 0, then:

[0127]

[0128] h is copied and transposed from the kth row of the matrix W, is the vector representation of the input word, and the linking (activation) function of the hidden layer unit is a simple linear transformation (i.e., directly passing the weighted input sum to the next layer).

[0129] At the output layer, instead of a multinomial distribution, C multinomial distributions are output. Each output is calculated using the same weight matrix from the hidden layer to the output layer:

[0130]

[0131] where w I represents the input word, w c,j represents the actual falling of the cth word in the output layer to the jth word, i.e., neuron, w R,c represents the cth word in the output layer that should fall to the Rth neuron, y c,j represents the normalized probability of the cth word in the output layer actually falling on the jth neuron, u c,j represents the unnormalized value of the cth word in the output layer actually falling on the jth neuron. Because the output layer shares weights, we have:

[0132]

[0133] where, represents the output vector of the jth word, whose value is the jth column of the weight matrix from the hidden layer to the output layer.

[0134] Step 4-2, by randomly sampling word pairs in the text data, form positive and negative sample pairs. For each word encountered in the training original text, set them to have a certain probability of being deleted from the text, and this deletion probability is related to the frequency of the word.

[0135] W i is a word, and M(W i ) is the frequency of the word W i in all corpora, then the probability P(W i ) that the word is left is represented as:

[0136]

[0137] where each word is assigned a weight, i.e.f(W i ), which represents the frequency of the word's appearance.

[0138] Step 4-3, next, the weights are learned by the back propagation algorithm and gradient descent, and the probability maximization is achieved when predicting w(c,j) at the cth context position, and the loss function L can be used:

[0139]

[0140] The objective function is:

[0141]

[0142] where,

[0143] After the operation and calculation of the above steps, further irrelevant words can be removed from the text data subjected to adversarial training, and words with high relevance to the theme can be left, so as to further improve the classification efficiency of the model based on the existing text data set.

[0144] Step 5, the text data under the balanced label after steps 1 to 4 is substituted into the existing TextCNN model for text classification, and the Max-Covariance Pooling method is used to combine the maximum pooling method commonly used in the original text classification method and the covariance pooling method to realize the pooling process of TextCNN, so as to improve the performance of the original model in the text classification task.

[0145] Step 5-1, in the CNN convolutional neural network, the feature extraction part in the pooling layer is optimized, and the Max-Covariance Pooling method is introduced, and the maximum pooling and covariance pooling fusion method is used for dimension reduction processing of the features.

[0146] The specific description is: in the pooling process, the maximum value of the input corresponding to the pooling window is,

[0147]

[0148] where Y is the output feature map, X is the input feature map, n represents the feature map number, a is the channel number, h is the row number, w is the column number, k is the length and width size of the pooling window, hs is the row number corresponding to h with a step size of s, and ia is the corresponding column number. When back propagation, the loss of the next layer is transmitted to the position of the maximum value of the pooling window,

[0149]

[0150] where F(·) is a function that finds the k largest X and returns a two-dimensional vector p(n, a, h, w). h , k w

[0151] Step 5-2, in the structure of the TextCNN model, the word vector input convolutional layer generates a corresponding local feature vector under the convolution operation of multiple convolution kernels. These vectors are aggregated into a feature vector after pooling in the pooling layer, and finally input into the Softmax layer after multiple fully connected layers for multi-class classification. After the convolution layer of the TextCNN model, a set of feature vectors X i (i = 1, 2, …, k) of size d are obtained, where k is the number of the set of feature vectors (k is the number of convolution kernels). The idea of covariance pooling is to regard the feature vector as a random variable, and each element of the feature vector is a sample value of the random variable.

[0152] The covariance matrix encodes the relationship between two or more random variables. This covariance matrix For k random variables, the element at position i, j is the covariance between the i-th and j-th feature vectors. The covariance matrix of the feature vectors is defined as:

[0153]

[0154] where each pair of feature vectors can be regarded as a pair of discrete random variables (X i , X j ) with expected values E(X i ) and E(X j ), respectively. The covariance of X i and X j can be represented as:

[0155]

[0156] It is obviously more cumbersome to calculate the elements in the covariance matrix C one by one. In fact, if we concatenate each feature vector X i into a feature vector matrix W of size (d, k), then the covariance matrix can be calculated as:

[0157]

[0158] where 1 is a full 1 matrix with the same size as W. The covariance matrix C is symmetric and positive definite, so it can be orthogonally decomposed as:

[0159] ​C = UΛU T (28);

[0160] where Λ is a diagonal matrix whose entries are the eigenvalues of C. Then a square root of C (i.e. the co-standard deviation matrix) is:

[0161]

[0162] The motivation of co-pooling is to find a new quantity that can represent the distribution of the whole feature vector data. In the previous derivation, the size of the feature vector matrix C is (d, k), and then the size of the co-standard deviation matrix is (k, k). Therefore, the co-standard deviation matrix can be expanded as:

[0163]

[0164] where y k represents the set of co-standard deviations of the k-th feature vector output to the feature vectors output by other convolutional kernels (the k co-standard deviations are concatenated into this vector), which describes the correlation between the k-th feature vector output and other feature vectors. Then, the quantity that can better represent the data distribution of the k-th feature vector is the average of the components of its co-standard deviation vector:

[0165]

[0166] That is, for a set of input feature vectors with a size of (d, k), the output of co-pooling is a two-dimensional tensor with a size of (1 x k), where the value of each component in the convolution output dimension is k .

[0167] Compared with the maximum value used in max-pooling, it can be seen that z k takes into account the relationship between the feature vectors output by each convolutional kernel.

[0168] Step 5-3, finally set two hyperparameters k1 and k2 and satisfy k1 + k2 = 1, and finally set the output of each block of the pooling layer as

[0169] Z = k1Z k1 + k2Z k2 (32);

[0170] Finally, the result is input to the activation function. For the binary classification label problem proposed in step 1, the sigmoid function is used for activation:

[0171]

[0172] Correspondingly, for the multi-classification label problem proposed in step 1, the softmax function is used for activation:

[0173]

[0174] wherein z i is the output value of the i-th node, and C is the number of output nodes, i.e. the number of categories of classification.

[0175] In the model, the pooling layer simultaneously uses the maximum pooling and covariance pooling mode for processing, and the results are input to the fully connected layer to linearly combine the extracted features, then a Dropout layer is added to prevent model overfitting, and then a fully connected layer is connected to reduce the dimension of the feature vector.

Claims

1. A network public opinion text classification method based on topic recognition, characterized in that: Specifically comprising the following steps: Step 1, select the original data set, process the data set; the specific process of step 1 is: Step 1.1, the user comments of the public opinion theme to be processed are obtained from the public opinion platform as the original data set, the text splicing operation of the theme post is carried out on the original data in json format through pandas in Python, and the format conversion is processed to save the data in csv format; Step 1.2, using Jieba word segmentation tool to carry out Chinese text data word segmentation on the theme post obtained in step 1.1, and then manually deleting the punctuation symbols and stop words after the spliced text word segmentation; Step 1.3, the text data processed in step 1.2 is classified and labeled into n categories; Step 1.4, according to the label divided in step 1.3, the text processed in step 1.2 is labeled; Step 1.5, manually constructing the term dictionary of the classified network public opinion theme text, wherein the words in the dictionary are taken from the text preprocessed by word segmentation in step 1.2 after artificial judgment; Step 2, the data processed in step 1 is subjected to theme recognition to obtain text data related to network public opinion theme; the specific process of step 2 is: Step 2.1, according to the label labeled in step 1.3, the text processed in step 1.2 is spliced to construct a document level data set in the form of theme post; wherein it is assumed that the corpus contains D texts, each corpus text contains B word pairs, each corpus text contains N words, and each corpus text involves K themes; Step 2.2, the document level data set after theme splicing in step 2.1 is input into the BTM model as the input document, and the BTM model first generates a word pair corpus, which converts short text into word pair mode using corpus information to describe theme distribution and word distribution from the level of corpus; The specific process of step 2.2 is: Step 2.2.

1. Obtain the word distribution under a specific topic by calculating the mathematical probability model of Dirichlet distribution ~Dir(β), where ф is the topic-word probability distribution, i.e. =P(w | r), Dir is the mathematical probability Dirichlet distribution used in the BTM model, β is the Dirichlet prior parameter, the label topic is represented by k, the word vector formed by the corpus is w, and the topic distribution is r, and P(w | r) represents the joint probability distribution of w and r. Step 2.2.2, randomly obtain a topic distribution in the entire text data set obtained in step 2.1 ~ Dir(a), where is the document-topic probability distribution, the corpus text is denoted by d, Dir is the mathematical probability Dirichlet distribution used in the BTM model, and a is the Dirichlet prior parameter; Step 2.2.3, for each biterm b in the set B, b ~ B, a topic k is randomly drawn from the topic distribution P(k) and a biterm b is randomly drawn from the topic k, i.e. b ~ Multi(P(k | b)) (1) where P(k) is the probability distribution of K topics in the whole corpus, i.e.​​​​​​​​​ P(d) = 1 - e~d = 1 - e~d (1) wherein, denotes a summation operation over each topic k related (*) operation; , P(k) denotes the probability distribution of the kth topic over the entire corpus; denotes the words taken from the text corresponding topic distribution, since the word pair b=( , ) takes i or j here; denotes the probability distribution of the word under topic k; According to formula (1), the joint distribution probability of the word pair b=( , ) is calculated, the corpus contains K topics and |B| word pairs, and the word pair b is composed of and , and thus the probability P(B) of the entire corpus of the double-word set B is expressed by formula (2) as follows: P(B)= (2) Step 2.2.4, according to the expected probability of the word pair in the document, the theme of the document is inferred, and the conditional probability distribution of the theme in the document is calculated: P(k|d) = P(d|k)P(k) / P(d) (3) where P(k|d) represents the conditional probability distribution of topic k in document d, represents the summation operation over the (*) operation for each word pair b in the text corpus, represents the conditional probability distribution of topic k for word pair b, represents the conditional probability distribution of word pair b contained in text d; P(k|b) = P(b|k)P(k) / P(b) (4) where P(k|b) denotes the conditional probability distribution of topic k in document b, denotes the conditional probability distribution of the pair of words under topic k, since the pair of words b=( , ) is taken as i or j, denotes the summation operation over the (*) operation for topic k; to obtain the conditional probability P(b|d) of the pair of words in a document, the pair of words is taken as an intermediate medium to compute: P(b|d) = P(d|b)P(b) (5) wherein, represents the total frequency of co-occurrence word pair b appearing in document d, and in the public opinion text data set, P(b|d) is considered as the uniform distribution of all co-occurrence word pairs b in document d; Step 2.2.5, using Gibbs sampling based on MCMC algorithm to network public opinion text data, when Gibbs sampling converges, the convergence condition is that the transition probability matrix converges to a stable probability distribution, that is, the sampling theme of all words can be obtained, using the corresponding relationship between all sampled words and themes, the document-theme distribution of each document is obtained and each theme-word distribution , so as to filter out irrelevant data from the crawled text data and only leave the text data related to the network public opinion theme; Step 3, using mixed data enhancement technology to carry out adversarial training on the text data obtained in step 2; Step 4, removing irrelevant words from the data processed in step 3; Step 5, importing the data processed in step 4 into the TextCNN model for text classification. 2.The network public opinion text classification method based on theme recognition according to claim 1, characterized in that: The specific process of step 3 is: Step 3.1, first, the data set after preliminary filtering of irrelevant text processed by the BTM model in step 2 is input into the translation software for language conversion, and the converted text is subjected to training of the generative adversarial network with the original data set, wherein the overall optimization objective function of the generative adversarial network is represented as: （6） where arg denotes the principal value of the argument of a complex number, and equation (6) represents the value of the variable when L is minimum and D is maximum. the value of the variable when L is minimum and D is maximum. The function V(L, D) in formula (6) is a measure of the discrimination ability of the discriminator, which is represented as: Wherein, in the generative adversarial network, the generator is abstracted as G(·), the sample distribution generated by the generator is denoted as ; the real sample distribution is denoted as , and the discriminator is abstractly represented as a function D(·), which is used to implicitly represent and ; Step 3.2, fix the generator, and get the maximum value of the discriminator, that is, carry out derivative operation on the polynomial and take the extreme value, then the analytical expression of the discriminator is obtained as: （8） Step 3.3, when the discriminator reaches the optimum, the actual optimization of the generative adversarial network is the JS divergence, wherein the JS divergence represents the gap between the generated sample and the real sample, and when the distribution generated by the generator distribution is consistent with the real distribution , the generator reaches the optimum, and data enhancement is achieved. ​ 3.The network public opinion text classification method based on theme recognition according to claim 2, characterized in that: The specific process of step 4 is: Step 4.1, by analyzing the central words in the text data after adversarial training. Predict its context-specific words probability The text vocabulary size is assumed to be... The size of the hidden layer is The neurons in adjacent layers are fully connected, and the input layer encodes word vectors using one-hot encoding. This means that each word vector must have only one 1 and the rest must be 0; Step 4.2, by randomly sampling word pairs in the text data, form positive and negative sample pairs, for each word encountered in the training original text, set them to have a probability of being deleted from the text, and this deletion probability is related to the frequency of the word; Step 4.3, then learn the weights by back propagation algorithm and gradient descent, realize probability maximization when predicting w(c, j) at the cth context position, use the loss function L: （10） The objective function is: （11） wherein .

Images