A method, system, and storage medium for intelligent classification of policy texts based on dual-channel neural networks and attention mechanisms.

Abstract

This invention relates to an intelligent classification method for policy texts based on a dual-channel neural network and an attention mechanism, comprising the following steps: S1. Preprocessing the collected policy texts; S2. Embedding the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vectors; S3. Inputting the word vectors into a CNN neural network and a BILSTM neural network with an attention mechanism, respectively, to obtain important local features and global features; S4. Merging the obtained important local features and global features to obtain policy text information features, inputting the obtained policy text information features into a fully connected layer neural network, and using the softmax function to calculate the classification probability of the policy text to obtain the classification result.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, and more specifically, to a method, system, and storage medium for intelligent classification of policy texts based on a dual-channel neural network and attention mechanism. Background Technology

[0002] With rapid economic and social development, managing external environmental information is indispensable for corporate decision-making. Policies are a crucial component of this information, serving as national guidelines and reflecting a country's or region's social and economic environment. Therefore, research on policy management has naturally become a hot topic in corporate management research. However, due to the complex structure and high information content of policy texts, corporate managers need to spend significant time and effort monitoring policy information, leading to incomplete policy monitoring and poor identification of key policy information. Driven by the trends of digital transformation and technological development, big data technology is increasingly important in corporate management. In recent years, intelligent classification and analysis methods and models for policy texts have gradually emerged. Convolutional Neural Networks (CNNs) are widely used, extracting local features from policy text information to achieve policy text classification. Some also employ Recurrent Neural Networks (RNNs), using the recurrent mechanism within the neural network to obtain contextual features of policy text information, thus achieving policy text classification. However, existing intelligent classification and analysis methods for policy texts rely too heavily on a single classification model, ignoring the imbalance and rich semantic information of policy text content. This leads to accuracy issues in intelligent classification and analysis of policy texts, posing difficulties for practical applications.

[0003] Therefore, how to achieve intelligent classification of policy texts based on their characteristics and assist enterprise managers in decision-making is a direction that those skilled in the art need to study. Summary of the Invention

[0004] To address the issues of incomplete policy monitoring and insufficient ability to identify key policy information caused by the irregular and lengthy policy monitoring time of enterprise managers in the past, this invention proposes an intelligent policy text classification method based on a dual-channel neural network and attention mechanism. This method effectively solves the efficiency problem of tedious and repetitive policy monitoring and analysis by enterprise managers.

[0005] To achieve the above-mentioned objectives, the technical solution adopted is as follows:

[0006] A policy text intelligent classification method based on dual-channel neural networks and attention mechanisms includes the following steps:

[0007] S1. Perform text preprocessing on the collected policy texts;

[0008] S2. Embed the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vectors;

[0009] S3. Input the word vectors into a CNN neural network and a BILSTM neural network with attention mechanisms respectively to obtain important local features and global features;

[0010] S4. The obtained important local features and global features are merged to obtain policy text information features. The obtained policy text information features are input into a fully connected neural network, and the softmax function is used to calculate the classification probability of the policy text to obtain the classification result.

[0011] Preferably, step S1, which involves preprocessing the collected policy text, includes sequentially performing data cleaning, sentence segmentation, and word segmentation operations.

[0012] Preferably, in step S2, the Skip-gram model includes an input layer, a hidden layer, and an output layer. In the preprocessed text information, let the current word at time t be w(t). The one-hot vector of the current word w(t) is input into the input layer, and the one-hot vector of the current word w(t) is weighted through the hidden layer. Finally, the context words of the current word w(t) are obtained in the output layer. Words are mapped to word vectors with d dimensions, denoted as w(t-2), w(t-1), w(t+1), and w(t+2). The mathematical expression of the objective function of the Skip-gram model is:

[0013]

[0014] Where Context(·) is the context vocabulary for retrieving the current word w(t), L Skip-gram The Skip-gram model predicts the output probabilities of context words based on the current word w(t);

[0015] Text information with d-dimensional word vectors is input into the word embedding layer to obtain word vectors A = {x1, x2, ..., x...} n}; n is the length of the policy text.

[0016] Preferably, in step S3, inputting word vectors into a BILSTM neural network to obtain global features specifically includes:

[0017] BILSTM neural networks include forward LSTM and inverse LSTM;

[0018] Let the word vector A = {x1, x2, ..., x...}n} is used as the input to the forward LSTM and the inverse LSTM, and is processed by the forward LSTM and the inverse LSTM.

[0019] For word vector x i Sequence features were read from left to right and right to left respectively to obtain the positive information feature h1 in the hidden layer. i and reverse information feature h2 i For positive information feature h1 i and reverse information feature h2 i By integrating and splicing, we obtain [h1] i h2 i ], represented as h i ;

[0020] Word vector A = {x1, x2, ..., x} n The resulting global features are H = [h1, h2, ..., h n ].

[0021] Preferably, in step S3, inputting word vectors into a CNN neural network with an attention mechanism to obtain important local features includes:

[0022] Let the current state characteristic be Query i Other features of the sequence in the current state are Key i ;

[0023] The correlation importance of the current state characteristics with other characteristics is calculated based on the energy function E;

[0024] The softmax function is used to calculate the correlation coefficient g between the current state feature and other features of the sequence. i ;

[0025] The energy function E of the current state characteristics i Correlation coefficient g with current features i Multiplying them together gives the global importance C of the feature. i ;

[0026] Select the feature Query with the highest global importance, and use a convolutional neural network to extract the feature Query with the highest global importance, denoted as Q. i Q i After processing by convolution, we obtain D. i D i As an important local feature.

[0027] Preferably, in step S4, the obtained important local features and global features are merged to obtain policy text information features, specifically represented as follows:

[0028]

[0029] Preferably, in step S4, the softmax function is used to calculate the classification probability of the policy text, including:

[0030] p = soft max(w s L+b s )

[0031] Among them, w s Let b be the weight matrix in the softmax function. s is the bias term in the softmax function, and p is the classification probability of the policy text.

[0032] Meanwhile, this invention also provides a policy text intelligent classification system based on a dual-channel neural network and attention mechanism, the specific scheme of which is as follows:

[0033] It includes a text preprocessing module, a word vector generation module, a feature acquisition module, and a classification module;

[0034] The text preprocessing module is used to preprocess the collected policy texts;

[0035] The word vector generation module is used to embed the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vector generation.

[0036] The feature acquisition module is used to input word vectors into CNN neural networks and BILSTM neural networks with attention mechanisms to obtain important local and global features;

[0037] The classification module is used to merge the obtained important local features and global features to obtain policy text information features. The obtained policy text information features are then input into a fully connected neural network, and the softmax function is used to calculate the classification probability of the policy text to obtain the classification result.

[0038] In addition, the present invention also provides a storage medium, the specific solution of which is as follows:

[0039] A storage medium includes a memory and a processor. The memory stores a program, which, when executed by the processor, performs the steps of the above-described policy text intelligent classification method based on a dual-channel neural network and attention mechanism.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This invention provides an intelligent policy text classification method based on a dual-channel neural network and an attention mechanism. By using a dual-channel neural network to acquire both global and local important features of the text, it quickly and accurately captures the features of the text. This overcomes the shortcomings of traditional intelligent policy text classification methods, which suffer from poor classification performance, and also solves the problems of long processing times and unsatisfactory results associated with manual policy text classification. Furthermore, this invention employs a self-attention mechanism to acquire local important features, thereby improving the accuracy of this acquisition. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the intelligent classification method for policy text based on a dual-channel neural network and attention mechanism of the present invention.

[0044] Figure 2 This is a network structure diagram of the Skip-gram model of the present invention.

[0045] Figure 3 This is a diagram illustrating the process of extracting global features using the BILSTM neural network of the present invention.

[0046] Figure 4 This is the LSTM neural network structure model of the present invention.

[0047] Figure 5 This diagram illustrates the working process of the Attention mechanism in extracting important local features in this invention.

[0048] Figure 6 This diagram illustrates the process of extracting important local features using the CNN neural network of this invention.

[0049] Figure 7 This is a flowchart illustrating the output process of the intelligent classification model for policy texts in this invention. Detailed Implementation

[0050] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0051] Example 1

[0052] The specific workflow of the policy text intelligent classification method based on dual-channel neural network and attention mechanism provided by this invention is as follows: Figure 1 As shown, the collected policy texts are first subjected to data cleaning, sentence segmentation, and word segmentation preprocessing to transform them into a unified text data file format suitable for word vector training. This text data file format serves as the input to the policy text classification model. The policy text classification model consists of three parts: word embedding, feature extraction, and output.

[0053] To better preserve the semantic information of the text, the word embedding part adopts the Skip-gram model in Word2vec word vectors, which predicts context words of a certain length based on the current word. For example... Figure 2 As shown, the Skip-gram model consists of three parts: an input layer, hidden layers, and an output layer. The one-hot vector of the current word w(t) is input into the input layer. The hidden layer then weights the one-hot vector of the current word w(t). Finally, the output layer obtains the context words of the current word w(t), mapping the words to d-dimensional word vectors, denoted as w(t-2), w(t-1), w(t+1), and w(t+2). The mathematical expression for its objective function is:

[0054]

[0055] Where Context(·) is the context vocabulary for retrieving the current word w(t), L Skip-gram The Skip-gram model predicts the output probabilities of context words based on the current word w(t);

[0056] Text information with d-dimensional word vectors is input into the word embedding layer to obtain word vectors A = {x1, x2, ..., x...} n}; n is the length of the policy text.

[0057] Taking advantage of the rich semantic information in policy texts, the feature extraction part consists of a global feature extraction part and an important local feature extraction part. The global feature extraction part includes a BILSTM neural network, while the important local feature extraction part includes an attention mechanism and a CNN convolutional neural network.

[0058] The BILSTM neural network is composed of a forward LSTM and an inverse LSTM. The LSTM neural network model is as follows: Figure 4 As shown, the word vector matrix A = {x1, x2, ..., x...} is... n The word vector x in} i x serves as the input to the forward and inverse LSTM neural networks at time t. t An LSTM neural network consists of a forget gate, an input gate, and an output gate, and controls the information flow through a gating mechanism. Here, Ct represents the LSTM unit state at time t; ht represents the hidden layer output state of the LSTM at time t.

[0059] Forget gate: Reads the hidden output state h from the previous time step. t-1 and the word vector x at time t t A value between 0 and 1 is obtained by passing through a sigmoid layer, denoted as the forgetting weight f. t , representing the output unit state C at the previous moment. t-1 With the current output unit state C t The degree of correlation.

[0060]

[0061] In the formula, W f Let b be the forget gate weight matrix. f For the forget gate bias term, h t-1 x represents the hidden layer output state at the previous time step. t f is the word vector at the current time. t σ represents the correlation between the output cell state at the previous time step and the current output cell state, and σ is the activation function.

[0062] Input gate: the word vector x at time t t Compared to the hidden layer output state h at the previous time step t-1 The control information weight coefficient i is obtained after using the sigmoid function. t Simultaneously, using the word vector x at time t t Compared to the hidden layer output state h at the previous time step t-1 As input, the tanh function is used to calculate the alternatives for updating the current unit state using the current word vector.

[0063] i t =σ(Wi [h t-1 ,x t +b i ])

[0064]

[0065] In the formula, W i W C Let b be the input gate weight matrix. i b c For the input gate bias term, h t-1 x represents the hidden layer output state at the previous time step. t Let i be the word vector at the current time. t The weighting coefficients represent the control information between the current word vector and the hidden layer output state at the previous time step. The options for updating the current unit state for the current word vector are σ, where σ is the activation function.

[0066] Finally, the forgetting weight f is used. t The previous output unit state C t-1 , Options for updating the current word vector and the current unit state Control information weighting coefficient i t The current output unit state C is obtained through calculation. t .

[0067]

[0068] In the formula, f t For forgetting weights, C t-1 C is the output unit state of the previous time step. t i represents the current output unit state. t To control the information weighting coefficient, C t This represents the current state of the output unit.

[0069] Output gate: the word vector x at time t. t Compared to the hidden layer output state h at the previous time step i-1 σ is obtained after sigmoid function calculation. t The current output unit state C t The control coefficient σ obtained by the calculation t The final hidden layer output at the current time step is calculated using the tanh function.

[0070] σ t =(W o [h t-1 ,x t ]+b o )

[0071] In the formula, h t-1 x represents the hidden layer output state at the previous time step. t W is the word vector at the current time. o Let b be the output gate weight matrix. o For the output gate bias term, σ t This is the control factor.

[0072] h t =σ t ×tanh(c t )

[0073] In the formula, C t σ represents the current state of the output unit. t For control coefficients, This is the final hidden layer output.

[0074] The working processes of both forward and inverse LSTM neural networks are as described above. Figure 3 As shown, the text matrix A = {x1, x2, ..., x...} n As input to a bidirectional LSTM, the text sequence is processed by both forward and inverse LSTMs to read sequence features in both left-to-right and right-to-left directions, thus obtaining the forward information h1 in the hidden layer. i and reverse information h2 i By integrating and splicing the positive and negative information features, [h1] is obtained. i h2 i ], represented as h i Therefore, word vector x i The extracted global feature is h i The global features of the semantic information of the entire text matrix are H = [h1, h2, ..., h...]. n ].

[0075] The extraction of important local features mainly consists of self-attention mechanisms and convolutional neural networks (CNNs). For example... Figure 5 As shown, the features of the current sequence word are input, denoted as Query. i Other features of the sentence information sequence are Key j，j＝1,2,...,i-1,i+1.... The energy function E is used to calculate the correlation between the current sequence word features and other features of the sentence sequence. In this case, the energy function E is calculated using a multiplicative attention method, and the obtained correlation is denoted as Ei. i,j The calculation formula is as follows:

[0076]

[0077] In the formula, Query iKey is a feature of the current sequence of words. j E represents other features of the sentence information sequence containing the current sequence word. i,j This represents the degree of association between the current sequence word and other features of the sentence sequence in which the current sequence word is located.

[0078] The obtained energy function E is used to calculate the normalized score of the matching between the current sequence word features and other features of the sentence sequence through the softmax function.

[0079] g = soft max(E) i，j )

[0080] In the formula, E i,j denoted by , represents the degree of association between the current sequence word and other features of the sentence sequence containing the current sequence word, where g is the association coefficient between the current sequence word and other features of the sentence sequence containing the current sequence word.

[0081] Query the current state characteristics i Other features of the sentence sequence Key j，j＝1,2,...,i-1,i+1.... The summation of the current state features with their corresponding feature vectors yields the final importance of the current state features to the textual information of the sentence.

[0082]

[0083] In the formula, Key j Let C be the other features of the sentence sequence containing the current sequence word, g be the correlation coefficient between the current sequence word and the other features of the sentence sequence containing the current sequence word, and C be the other features of the sentence sequence containing the current sequence word. i The final importance of the current state features to the text information of the sentence is obtained.

[0084] The final importance C of all current state features to the text information of the sentence. i By comparing, we can obtain the maximum value C. max The value, i.e., the feature Q with the highest global importance, is selected. uerymax ,like Figure 6 As shown, a convolutional neural network is used to extract the feature Q with the highest global importance. uerymax This feature is extracted and used as a local important feature of the sentence text information, denoted as Q. i The nonlinear activation function ReLU is used to apply the Q-squared function. i The convolution operation is expressed as follows, where w is the weight parameter in the convolution kernel and b is the bias term:

[0085] D i =f(w·Q) i +b)

[0086] In the formula, w is the weight parameter of the convolution kernel, b is the bias term of the convolution kernel, and Q...i Here, f represents the local important features of the sentence text information, and D represents the convolution operation. i Q is a local important feature of the sentence text information. i Feature information after convolution operation.

[0087] like Figure 7 As shown, the global features H of the text information and the local importance features Q of the text sentences are finally combined. i By merging, we obtain the text information features.

[0088] The output part consists of a fully connected layer and a softmax function. The obtained text information features are input into the fully connected neural network, and the softmax function is used to calculate the text classification probability p, achieving intelligent text classification. Its expression is shown below, where w... s Let b be the weight matrix. s For bias terms:

[0089] p = soft max(w s L+b s )

[0090] In the formula, w s Let b be the weight matrix in the softmax function. s Here, L is the bias term in the softmax function, and L represents the text information feature.

[0091] Example 2

[0092] This embodiment provides a policy text intelligent classification system based on a dual-channel neural network and attention mechanism, the specific scheme of which is as follows:

[0093] It includes a text preprocessing module, a word vector generation module, a feature acquisition module, and a classification module;

[0094] The text preprocessing module is used to preprocess the collected policy texts;

[0095] The word vector generation module is used to embed the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vector generation.

[0096] The feature acquisition module is used to input word vectors into CNN neural networks and BILSTM neural networks with attention mechanisms to obtain important local and global features;

[0097] The classification module is used to merge the obtained important local features and global features to obtain policy text information features. The obtained policy text information features are then input into a fully connected neural network, and the softmax function is used to calculate the classification probability of the policy text to obtain the classification result.

[0098] Example 3

[0099] This embodiment provides a storage medium, including a memory and a processor. The memory stores a program, and when the program is executed by the processor, it performs the method steps of the policy text intelligent classification method based on dual-channel neural network and attention mechanism described in Embodiment 1.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A policy text intelligent classification method based on dual-channel neural network and attention mechanism, characterized by: Includes the following steps: S1. Perform text preprocessing on the collected policy texts; S2. Embed the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vectors; S3. Input the word vectors into a CNN neural network and a BILSTM neural network with attention mechanisms respectively to obtain important local features and global features; S4. The obtained important local features and global features are merged to obtain policy text information features. The obtained policy text information features are input into a fully connected neural network, and the softmax function is used to calculate the classification probability of the policy text to obtain the classification result. Step S1 involves preprocessing the collected policy text, including sequentially performing data cleaning, sentence segmentation, and word segmentation operations. In step S2, the Skip-gram model includes an input layer, a hidden layer, and an output layer. In the preprocessed text information, let the current word at time t be... ; In the input layer, the current word The one-hot vector is input and processed by the hidden layer for the current word. The one-hot vectors are weighted and finally used to obtain the current word in the output layer. The context vocabulary; mapping words to d-dimensional word vectors, denoted as... The mathematical expression for the objective function of the Skip-gram model is: ； Where Context(·) is used to retrieve the current word Contextual vocabulary, For the Skip-gram model, based on the current word Predict the output probability of context words; Text information with d-dimensional word vectors is input into the word embedding layer to obtain word vectors. ; n is the length of the policy text; In step S3, the word vectors are input into the BILSTM neural network to obtain global features, specifically including: BILSTM neural networks include forward LSTM and inverse LSTM; Word vectors As input to the forward LSTM and the inverse LSTM, it is processed by the forward LSTM and the inverse LSTM. For word vectors Sequence features were read from left to right and right to left respectively to obtain the positive information features in the hidden layer. and reverse information features For positive information features and reverse information features By integrating and splicing, we can obtain , represented as ; Word vectors The obtained global features are ; In step S3, the word vectors are input into a CNN neural network with an attention mechanism to obtain important local features, including: Let the current state characteristics be Other features of the sequence in the current state are ; The correlation importance of the current state characteristics with other characteristics is calculated based on the energy function E; The softmax function is used to calculate the correlation coefficient between the current state features and other features of the sequence. ; The energy function of the current state characteristics Correlation coefficient with current features Multiplying yields the global importance of the features. ; The feature query with the highest global importance is selected, and a convolutional neural network is used to extract this feature query, denoted as . , After processing by convolution, we obtain... ,Will As an important local feature.

2. The policy text intelligent classification method based on dual-channel neural network and attention mechanism according to claim 1, characterized in that: In step S4, the obtained important local features and global features are merged to obtain policy text information features, specifically represented as follows: 。 3. The policy text intelligent classification method based on dual-channel neural network and attention mechanism according to claim 2, characterized in that: In step S4, the softmax function is used to calculate the classification probability of the policy text, including: ； in, This is the weight matrix in the softmax function. is the bias term in the softmax function, and p is the classification probability of the policy text.

4. A policy text intelligent classification system based on a dual-channel neural network and attention mechanism, applying the policy text intelligent classification method based on a dual-channel neural network and attention mechanism as described in any one of claims 1-3, characterized in that: It includes a text preprocessing module, a word vector generation module, a feature acquisition module, and a classification module; The text preprocessing module is used to preprocess the collected policy texts; The word vector generation module is used to embed the preprocessed text information into word vectors using the Skip-gram model in Word2vec word vector generation. The feature acquisition module is used to input word vectors into CNN neural networks and BILSTM neural networks with attention mechanisms to obtain important local and global features; The classification module is used to merge the obtained important local features and global features to obtain policy text information features. The obtained policy text information features are then input into a fully connected neural network, and the softmax function is used to calculate the classification probability of the policy text to obtain the classification result.

5. A storage medium comprising a memory and a processor, wherein the memory stores a program, characterized in that: When the program is executed by the processor, it performs the method steps of the policy text intelligent classification method based on dual-channel neural network and attention mechanism as described in any one of claims 1-3.

Images