A standard content text classification method that integrates global semantic features and splicing features

Abstract

This invention discloses a standard content text classification method that integrates global semantic features and concatenated features. The steps are: (1) extracting the serialized vector of the standard content text; (2) determining the context representation of each word; (3) extracting word context weights and global semantic information; (4) extracting text feature information; and (5) fusing the output features to obtain the final category prediction probability. This invention uses a layer- and word-based local information extraction method to obtain more local semantic features, integrates multiple networks to effectively extract global semantic features, and finally fuses the two, providing a high-accuracy text classification method for standard content text classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of text classification and standard digitization, in particular, it is mainly a kind of standard content text classification method fusing global semantic features and splicing features. BACKGROUND

[0002] Standard classification is an indispensable link in standard digitization work, and automatic classification of standard content helps standard users to compare quickly. At present, the text classification method mainly includes neural network model, random forest model and the like. The model based on convolutional neural network utilizes sliding convolution window to obtain local semantic features at different abstraction levels, but lacks the modeling ability of context information. The model based on recurrent neural network captures global semantic information through recursive calculation, but lacks the perception ability of local information. The attention mechanism highlights key information by using weight adjustment, but ignores time sequence information. The pre-training model is pre-trained by using large-scale corpus, but the application scene is obviously limited. The method based on random forest model can balance errors to a certain extent, but the operation time is relatively long, and the application expansion is poor.

[0003] Standard content has obvious cross-reference characteristics, and the same standard internally refers to each other, different standards refer to each other, and different standards have obvious differences in definition and requirements. Therefore, a standard content text classification method capable of accurately extracting context semantic features of standard content, with high classification accuracy and good application expansion, is needed to provide strong support for the field of standard digitization. SUMMARY

[0004] In order to solve the problems existing in the prior art, the present application provides a kind of standard content text classification method fusing global semantic features and splicing features, and its specific process is as shown in Figure 1

[0005] The technical scheme implementation steps are as follows:

[0006] (1) extract the serialized vector E of standard content text:

[0007] Obtain the vector representation of the sentence in the text, and obtain

[0008] X=[x1,x2,…,x n ]

[0009] In the formula, X is the vector representation of the sentence, x i represents the i th character in the sentence text, the serialized text vector is extracted by inputting the text X into the ALBERT layer for serialization operation, and

[0010] E=[E1,E2,…,E n ]​

[0011] wherein E represents a character array of the sentence text after serialization, E i represents the serialized character of the i-th word in the text;

[0012] (2) determining the context representation of each word

[0013] The character representation processed by ALBERT is input into the BiLSTM layer, and the BiLSTM obtains the forward hidden layer state r t and the backward hidden layer state l t at the t-th moment by forward and backward LSTM, and splices r t and l t to obtain the hidden layer state vector h t , and finally combines h t with the word vector information E t to obtain the final context representation of each word The calculation process is as follows

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] The above formula is the three gates (input gate, forget gate, output gate) of the forward LSTM, is the three gates of the backward LSTM, and the six gates can control the information flow direction, solve the gradient disappearance and explosion problem of the traditional RNN; in the forward LSTM, the hidden layer state r t-1For r t The update is affected; for backward LSTM, the hidden layer state l t+1 For l t The update has an impact. W is the weight matrix; b is the bias term; σ is the sigmoid activation function; c is the state variable, which, together with the output gate, controls the final hidden layer state; * is the Hadamard product; tanh is the hyperbolic tangent function; This is a vector concatenation operation. The character array with context information after BiLSTM processing is...

[0027]

[0028] (3) Extracting word context weights and global semantic information F global :

[0029] The character array output by BiLSTM

[0030]

[0031] An attention mechanism is used to assign a weight to the context of each word to reflect the importance of different words to the global semantic features of the sentence. Finally, the global semantic information F is obtained by weighted summation. global

[0032]

[0033]

[0034]

[0035] W u Let b be the weight matrix. u For the bias term, u a For a randomly initialized, learnable context vector, a t Let F be the normalized weight of the word at time t. After weighting through an attention mechanism, the final result is a vector F containing global information. global ;

[0036] (4) Extract text feature information F local :

[0037] An improved CNN model based on layer and word local information extraction is adopted. In addition to pooling the feature information of the layer at different one-dimensional convolutional kernels, it also pools the feature information of the word at different layers, taking h of different widths. l One-dimensional convolution kernel (h) l (where E is the kernel width) is applied to the text representation [E1, E2, ..., E i ,…,E nThe feature matrix M is extracted.

[0038]

[0039]

[0040] M = [R1, R2, …, R l ]

[0041]

[0042]

[0043] L i = [r i1 , r i2 , …, r il ]

[0044]

[0045]

[0046]

[0047] W il is the l-th one-dimensional convolution kernel weight matrix corresponding to the i-th word in the structure, which acts on the word vector window to generate new features r il , where b is the bias term, and relu is the activation function. The one-dimensional convolution kernel acts on each possible sub-vector window to generate the l-th one-dimensional convolution kernel feature vector R l , and multiple convolution kernel feature vectors are combined to obtain the feature matrix M. The row vectors of the M matrix represent the same word in different layers of vector representation, and the features of the same layer can be represented by the column vectors of M. The max-pooling operation used in this patent acts on the row vectors and column vectors of the feature matrix M, respectively, and the method adopts the method of max-pooling to obtain the most significant layer features and word features Finally, all significant layer features and word features are spliced to obtain the final feature representation F local . R i represents the layer features of the i-th layer, and L j represents the features of the j-th word.

[0048] (5) The final class prediction probability p is obtained by fusing the output features:

[0049] p1 = softmax1(W g F global + b g )

[0050] p2 = softmax2(Wlocal F local +b local )

[0051]

[0052] Text semantic feature information F obtained by attention mechanism global Text feature information representation F obtained by improved CNN based on layer and word local information extraction local Act on softmax1 classifier and softmax2 classifier respectively, obtain classification probability p1 and p2 respectively, take the arithmetic mean of the two probabilities for fusion, and finally obtain class prediction probability p, wherein W g W local b g b local are bias terms.

[0053] The present application has the advantages over the prior art:

[0054] (1) The layer and word based local information extraction method used in the present application can extract more local information than the previous model, and the network information extraction which is insufficient in local information extraction can be greatly improved.

[0055] (2) The method provides a method for text classification using global and local feature information, which fully extracts local and global information and fully integrates model advantages. The word sequence vector is used as an embedding layer to obtain better text representation, and multiple networks are used to effectively extract global semantic features. The layer and word based local information extraction method is used to obtain more local semantic features, and finally the two are fused. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to better understand the present application, further description will be made in combination with the accompanying drawings.

[0057] Figure 1 is the step flow chart of the standard content text classification method for fusing global semantic features and splicing features;

[0058] Figure 2 is the algorithm flow chart of the standard content text classification method for fusing global semantic features and splicing features;

[0059] Figure 3 is the network model schematic diagram of the standard content text classification method for fusing global semantic features and splicing features;

[0060] Figure 4 is the accuracy comparison diagram of the standard content text classification method for fusing global semantic features and splicing features; DETAILED DESCRIPTION

[0061] The application will be further described in detail through the embodiments.

[0062] In the embodiment, two standard data sets, a gas accident standard and a dangerous chemical accident standard, are selected for testing, which include standard sets in different aspects such as disposal process, response plan and pipe material, and each type of standard set contains 150 standards, and a total of 300 standards.

[0063] The standard content text classification method provided by the application fuses global semantic features and splicing features, and the algorithm process is as shown in Figure 2 , and the specific steps are as follows:

[0064] (1) Extract the serialized vector E of the standard content text:

[0065] Taking the gas accident handling standard data set as an example, the average number of words in the sentence in the gas accident handling standard data set is 19, and the text representation of the corresponding sentence is

[0066] X=[x1,x2,…,x i ,…,x 19 ]

[0067] In the formula, X is the vector representation of the sentence, x i represents the i-th character in the sentence text, and by inputting the text X into the ALBERT layer for serialization operation, the serialized text vector

[0068] E=[E1,E2,…,E i ,…,E 19 ]

[0069] Wherein E represents the serialized character array of the sentence text, E i represents the serialized character of the i-th word in the text, and the embedding dimension is 768.

[0070] (2) Determine the context representation of each word

[0071] The character representation processed by the ALBERT is input into the BiLSTM layer, and the network model is as shown in Figure 3 , r 19 is the representation vector of the forward channel output sentence, l1 is the representation vector of the backward channel, BiLSTM obtains the forward hidden layer state r t and the backward hidden layer state l t at the t-th moment through the forward and backward LSTM, wherein 1≤t≤19, and E tr t and l t are concatenated as the hidden layer state vector h t , 1≤t≤19. Finally, h t is combined with the word vector information E t to obtain the final contextual representation of each word The calculation process is as follows,

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] The above formula is the three gates (input gate, forget gate, and output gate) of the forward LSTM, is the three gates of the backward LSTM, which can control the flow of information and solve the gradient vanishing and explosion problems of traditional RNN; in the forward LSTM r t-1 is the word embedding vector of the previous word, while in the backward LSTM l t+1 is the word vector embedding of the previous word; W is a weight matrix initialized with a random distribution satisfying ; b is the bias term, initialized to 0; σ is the sigmoid activation function; c is the state variable, which controls the final hidden layer state together with the output gate; * is the Hadamard product; tanh is the hyperbolic tangent function; is the vector concatenation operation.

[0085] The output of BiLSTM is a sentence array that introduces context information into each character. Since the sentence is composed of 19 words, the feature vector corresponding to 19 characters is obtained

[0086]

[0087] (3) Extracting word context weight and global semantic information F global :

[0088] The feature representation vector of a sentence composed of 19 characters

[0089]

[0090] Each word representation is matched with a weight through the attention mechanism to reflect the importance of different words to the global semantic features of the sentence. Finally, the global semantic information F is obtained by weighted summation global .

[0091]

[0092]

[0093]

[0094] W u is the weight matrix, initialized to satisfy the distribution b u is the bias term, initialized to 0, u a is a randomly initialized N(0,1), a learnable context vector, a t is the normalized weight of the word at time t.

[0095] (4) Extracting text feature information F local :

[0096] An improved CNN model based on layer and word local information extraction is adopted. In addition to layer-based feature information under different one-dimensional convolution kernels, the features based on words under different layers are also pooled. Different width sizes of h l one-dimensional convolution kernels (h l is the convolution kernel width, satisfying h1>h2>…>h i >…>h 19 , are set in training. In this experiment, h l =1,2,3) are applied to the text representation [E1,E2,…,E i ,…,E 19 ] to extract feature matrix M.

[0097]

[0098]

[0099] M = [R1, R2, R3]

[0100]

[0101]

[0102] L i = [r i1 , r i2 , r i3 ]

[0103]

[0104]

[0105]

[0106] W il is the l-th one-dimensional convolution kernel weight matrix corresponding to the i-th element in the structure, the initialization distribution satisfies N(0, 1), which acts on the word vector window to generate new features r i , where b is the bias term b = 0.01, and relu is the activation function. One-dimensional convolution kernel acts on each possible sub-vector window to generate a feature vector R, and three layers of convolution feature vectors are obtained. A plurality of feature vectors are combined to obtain a feature matrix M. The maximum pooling operation is respectively performed on each layer feature R i and each word feature L i in the feature matrix M to obtain the most significant layer feature and element feature Finally, all the most significant features are spliced to obtain the final feature representation F local .

[0107] (5) The final class prediction probability p is obtained by fusing the output:

[0108] p1 = softmax1(W g F global +b g )

[0109] p2 = softmax2(W local F local +b local )

[0110]

[0111] The text semantic feature information F globalF is represented by feature information based on layers and elements. local The values ​​are applied to the softmax1 and softmax2 classifiers respectively, yielding classification probabilities p1 and p2. The arithmetic mean of these two probabilities is then fused to obtain the final class prediction probability p, where W... g With W local As trainable weights, all are initialized with N(0,1), b g With b local This is a bias term, initialized to 0.

[0112] To verify the accuracy of this invention in classifying standard content text, a text classification experiment was conducted. The experimental results are as follows: Figure 4 As shown. By Figure 4 It can be seen that the performance of this method is improved compared to other models.

Claims

1. A standard content text classification method fusing global semantic features and stitching features, characterized in that, The method comprises the following steps: Step one: Extracting serialized vectors of standard content text : Obtaining a vector representation of a sentence in the text, obtaining: ； In the formula, It is the vector representation of the sentence. This indicates the first [item] in the sentence text. Each character, through text The input is fed into the ALBERT layer for serialization, and the serialized text vector is extracted to obtain: ； wherein represents the serialized character array of the sentence text, represents the serialized character of the th word in the text; Step two: Determine the contextual representation of each word : The character representation processed by ALBERT is input into the BiLSTM layer. The BiLSTM obtains the 6th character through forward and backward LSTM operations. Forward hidden layer state at any time and backward hidden layer state and will and Concatenated together to form the hidden layer state vector Finally With word vector information By combining these, we obtain the final contextual representation of each word. ; The calculation process is as follows: ； ； ； ； ； ； ； ； ； ； ； ； The above formula , , are respectively the input gate, the forget gate and the output gate of the forward LSTM, , , are the three gates of the backward LSTM, and the six gates can control the information flow to solve the gradient disappearance and explosion problem of the traditional RNN; in the forward LSTM, the hidden layer state has an impact on the update of , and for the backward LSTM, the hidden layer state has an impact on the update of ; is a weight matrix; is a bias term; is a sigmoid activation function; is a state variable, which together with the output gate controls the final hidden layer state; is a Hadamard product; is a hyperbolic tangent function; is a vector splicing operation. The character array with context information after BiLSTM processing is: ； Step three: Extracting word context weights and global semantic information : The character array output by the BiLSTM: The attention mechanism matches a weight for each word according to the context, so as to reflect the importance of different words to the global semantic features of the sentence, and finally obtains the global semantic information through weighted summation ; ; ; ; is a weight matrix, is a bias term, is a randomly initialized, learnable context vector, is the normalized weight of the word at the moment; after weighting by the attention mechanism, the final vector with global information is obtained ; Step four: Extracting text feature information : An improved CNN model based on layer and word local information extraction is adopted, in addition to layer-based feature information under different one-dimensional convolution kernels for pooling, the features under different layers based on words are also pooled, and different width sizes one-dimensional convolution kernels are applied to text representation to extract feature matrix , is the convolution kernel width; ; ; ; ; ; ; ; ; ; is the th one-dimensional convolution kernel weight matrix corresponding to the th word pair, which acts on the word vector window to generate new features , where is the bias term, is the activation function; the one-dimensional convolution kernel acts on each possible sub-vector window to generate the th one-dimensional convolution kernel feature vector , and multiple convolution kernel feature vectors are combined to obtain the feature matrix ; The row vectors of the matrix represent the vector representation of the same word at different layers, and the features of the same layer can be represented by the column vectors of ; The max pooling operation used in this patent acts on the row vectors and column vectors of the feature matrix , respectively, and the method adopts the method of max pooling; the most significant layer features and word features are obtained; finally, all significant layer features and word features are spliced to obtain the final feature representation ; represents the layer feature of the th layer, represents the feature of the th word; Step five: fusion of output features to get final class prediction probabilities : ; ; ; Text semantic feature information obtained through attention mechanism Text feature information representation obtained through improved CNN based on layer and word local information extraction respectively acting on the classifier and the classifier, respectively obtaining classification probabilities with taking the arithmetic mean of the two probabilities to fuse, finally obtaining class prediction probability wherein with is a trainable weight, with is a bias term.

Images