A semi-structured text classification method based on graph attention

Abstract

A semi-structured text classification method based on graph attention includes the following technical steps: Step 1, text preprocessing, data cleaning, forming a graph matrix M; Step 2, forming a coding layer, using the Albert pre-training model to obtain the embedding matrix; Step 3, performing attention operation on the embedding matrix, and performing weighted calculation based on the graph matrix; Step 4, compressing the feature vector matrix and performing text classification. For text classification problems in the financial field, especially in the case of semi-structured text, ordinary classifiers cannot be combined with structured information for recognition, which is effective in improving the accuracy of the final classifier. At the same time, for the model structure, the ability of information representation extraction is greatly improved, thereby increasing the accuracy of the final softmax classification, which has great market promotion value.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification, and in particular to a semi-structured text classification method based on graph attention. Background Art

[0002] In natural language processing, text classification is a relatively mature and common text processing task. Common tasks include text label classification, sentiment classification, etc. Under the condition of sufficient training samples, these tasks can achieve relatively good results. In some private fields (such as the financial field), there is a special type of text data information, which is mainly composed of a large number of numbers, and different data types are distinguished according to a certain arrangement order, such as Figure 1 , Figure 2 As shown, Figure 1 It is the secondary current bond transaction information. Figure 2 It is first-level bidding information. Judging them not only depends on text information, but also needs to consider their structural information. In various text processing tasks, this type of data needs to be classified. Professionals judge the category of a text based on text information and their arrangement structural information. Therefore, how to construct structural information that considers both text information and text arrangement is a difficult problem for existing classification algorithms.

[0003] Regarding the task of text classification, there are a series of algorithm models, such as Fasttext, TextCNN and TextRNN, as well as a series of BERT-based classification models derived with the development of pre-training models, TextCNN+attention and TextRNN+attention that integrate the attention mechanism, etc. These models can better solve some text classification problems in general domains. For the text corpus in the financial field, a large part of it is the semi-structured text data described above. There are currently two main processing technologies. One is based on the "keyword + rule engine" technology, which distinguishes different businesses based on keyword discrimination or other rhetoric methods. The second is to use the common text classification algorithm models in the NLP field mentioned above, such as the classification model of Bert+. If the computing performance needs to be considered, models such as FastText and TextCNN are used for classification tasks with structural information or that rely on structural information for judgment. Both methods have certain limitations. First, the solution based on keyword + rule engine requires industry experts to provide experience and integrate them. Generally, it can be handled better in a more idealized corpus, but it will be stretched when dealing with actual online business data. Either there is a conflict of rules or it is not well considered. Moreover, after the accumulation of rules, the maintenance cost is extremely high, and the operating efficiency will become lower and lower. Secondly, the text classification algorithm based on deep learning, the current solutions do not integrate "structural information" into the learning of the model. Although some pre-trained models based on Bert+ can learn the position information of characters, they still cannot effectively learn structural features. When distinguishing between corpus such as chat and business corpus (primary bidding, secondary current bond transactions, etc.), the effect is very good, but when it comes to the classification of semi-structured texts with relatively high similarity (primary bidding and secondary current bond transactions, etc.), the effect will be significantly reduced.

[0004] Existing solutions do not make good use of the structured information within the data. Therefore, if this information can be incorporated into the judgment plan, it should be significantly improved. Summary of the invention

[0005] The purpose of the present invention is to provide a semi-structured text classification method based on graph attention in view of the defects and shortcomings of the prior art, so as to solve the problems raised in the above-mentioned background technology. The present invention aims at the text classification problem in the financial field, especially in the case of semi-structured text, ordinary classifiers cannot be combined with structured information for recognition, and provides a method for extracting structural information and integrating it into the classification system, which is effective in improving the accuracy of the final classifier. At the same time, the model structure, especially the steps of taking the last few layers of transformer, making the cross-sentence structure weight higher than that within the same sentence, and multi-headattention, greatly improves the ability of information representation and extraction, thereby increasing the accuracy of the final softmax classification, and has great market promotion value.

[0006] To achieve the above purpose, the present invention comprises the following steps: step 1, text preprocessing, data cleaning, forming a graph matrix M; step 2, forming a coding layer, and using the Albert pre-training model to obtain an embedding matrix; step 3, performing an attention operation on the embedding matrix, and performing weighted calculation based on the graph matrix; step 4, compressing the feature vector matrix and performing text classification.

[0007] The text in step 1 usually contains a large amount of noise data. First, the text is preprocessed and cleaned, such as full-width and half-width conversion, English uppercase and lowercase unification, multiple blank characters are merged, stop words are removed, etc. Text segmentation is performed. According to the judgment of text line breaks, space keys, commas, semicolons, Tab keys and other separators, the text data is cut into multiple words or short sentences. The position of the short sentences and words in the text, the position information of the rows and columns in which they are located, and their connection relationship are judged. Simple logical rules are used. In the horizontal direction, a word is directly associated with the words on the left and right. In the vertical direction, if they are in the same position in the horizontal direction, they are associated, so as to form the following Figure 4 As shown in the graph matrix M, in the expression of the graph matrix, this scheme uses different values ​​to express different associations. The association between adjacent characters in a single row is marked with a value of 1, the association between elements is marked with 2, and the remaining unmarked areas are set to 0 by default.

[0008] In step 2, the text is passed through a pre-trained model that has been trained on professional corpus (here we use Albert as an example, and other pre-trained models such as Bert, XLNet, Roberta, etc. can also be used instead), and the output vector of the transformer of the penultimate y layer corresponding to each character is recorded as We concatenate the outputs of the last four layers of transformers, and the concatenated vector corresponding to each character is These vectors are combined to form the embedding matrix.

[0009] The step 3 described above is similar to self-attention, the vector corresponding to each character in the embedding matrix As input, by comparing with the query matrix W Q , key matrix W K , value matrix W V Multiply to get the query vector Key Vector Value vector The calculation formula is as follows:

[0010]

[0011] Calculate the inner product of the query vector and the key vector and scale them to get the weight λ ij , aggregate all weights to get the initial weight matrix Φ, specifically:

[0012]

[0013] Then the initial weight matrix is ​​multiplied by the graph matrix M to obtain the weighted weight matrix Then multiply it with the value vector to output the representation vector matrix X. The calculation process is:

[0014]

[0015] For the above operations, you can also perform multi-head and norm operations like transformer and then output.

[0016] After obtaining the feature vector matrix in step 4, the matrix is ​​compressed into a vector through a pooling layer such as max or mean. Then it is connected to a fully connected layer, and finally enters a softmax layer to output the classification result. In training, a penalty term is added to the loss function to prevent the loss from decreasing due to an increase in the module length. During training, the recall indicator is mainly used as a metric according to business needs.

[0017] The working principle of the present invention is as follows: first, through preprocessing, the input text is cleaned, the internal structure of the text is split, the relationship graph matrix is ​​extracted according to the connection between the elements, and the adjacency matrix M is obtained according to the different weights of the relationship. Secondly, the mature pre-training model Albert is used to embed the text to obtain the text vector matrix E. On this basis, we add an attention layer to each word vector Respectively through and query matrix W Q , key matrix W K , value matrix WV Multiply to get the query vector Key Vector Value vector pass Calculate the connection weights; Different from other graph attention mechanisms, this scheme directly multiplies the adjacency matrix and the weight matrix obtained after attention to obtain the weighted weights. Finally, the weights are After multiplication, it passes through the pooling layer and the fully connected layer, and is classified by softmax.

[0018] After adopting the above technical scheme, the beneficial effects of the present invention are as follows: for the text classification problem in the financial field, especially in the case of semi-structured text, ordinary classifiers cannot be combined with structured information for identification, and a method of extracting structural information and integrating it into the classification system is provided, which is effective in improving the accuracy of the final classifier. At the same time, for the model structure, especially taking the last few layers of transformer, the cross-sentence structure weight is higher than that within the same sentence, multi-headattention and other steps, the ability of information representation extraction is greatly improved, thereby increasing the accuracy of the final softmax classification, and having great market promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 It is a partial schematic diagram of secondary current bond transaction information in the traditional financial field;

[0021] Figure 2 It is a partial schematic diagram of the first-level bidding information in the traditional financial field;

[0022] Figure 3 It is a schematic diagram of the technical process structure of the present invention;

[0023] Figure 4 It is a schematic diagram of the structure of the graph matrix M of the present invention; DETAILED DESCRIPTION

[0024] See also Figure 1 to Figure 4As shown, this specific implementation method includes the following technical solution steps: step one, text preprocessing, data cleaning, forming a graph matrix M; step two, forming a coding layer, and using the Albert pre-training model to obtain an embedding matrix; step three, performing an attention operation on the embedding matrix, and performing weighted calculation based on the graph matrix; step four, compressing the feature vector matrix and performing text classification.

[0025] Furthermore, the text in step 1 usually contains a large amount of noise data. First, the text is preprocessed and cleaned, such as full-width and half-width conversion, English uppercase and lowercase unification, multiple blank characters are merged, stop words are removed, etc. Text segmentation is performed. According to the judgment of text line breaks, and separators such as space keys, commas, semicolons, and Tab keys, the text data is segmented into multiple words or short sentences. The positions of short sentences and words in the text, the position information of the rows and columns in which they are located, and their connection relationships are judged. Simple logical rules are used. In the horizontal direction, a word is directly associated with the words on the left and right. In the vertical direction, if they are in the same position in the horizontal direction, they are associated. This is formed as follows Figure 4 As shown in the graph matrix M, in the expression of the graph matrix, this scheme uses different values ​​to express different associations. The association between adjacent characters in a single row is marked with a value of 1, the association between elements is marked with 2, and the remaining unmarked areas are set to 0 by default.

[0026] Furthermore, in step 2, the text is passed through a pre-trained model that has been trained on professional corpus (here we use Albert as an example, and other pre-trained models such as Bert, XLNet, Roberta, etc. can also be used instead), and the output vector of the transformer of the penultimate y layer corresponding to each character is recorded as We concatenate the outputs of the last four layers of transformers, and the concatenated vector corresponding to each character is These vectors are combined to form the embedding matrix.

[0027] Furthermore, the step 3 is similar to self-attention, and the vector corresponding to each character in the embedding matrix is As input, by comparing with the query matrix W Q , key matrix W K , value matrix W V Multiply to get the query vector Key Vector Value vector The calculation formula is as follows:

[0028]

[0029] Calculate the inner product of the query vector and the key vector and scale them to get the weight λ ij , aggregate all weights to get the initial weight matrix Φ, specifically:

[0030]

[0031] Then the initial weight matrix is ​​multiplied by the graph matrix M to obtain the weighted weight matrix Then multiply it with the value vector to output the representation vector matrix X. The calculation process is:

[0032]

[0033] For the above operations, you can also perform multi-head and norm operations like transformer and then output.

[0034] Furthermore, after obtaining the feature vector matrix in step 4, the matrix is ​​compressed into a vector through a pooling layer such as max or mean, and then connected to a fully connected layer, and finally enters a softmax layer to output the classification result. In training, a penalty term is added to the loss function to prevent the loss from decreasing due to an increase in the module length. During training, the recall indicator is mainly used as a metric according to business needs.

[0035] Furthermore, the present invention mainly aims at the classification problem of text data in the financial field, especially when processing semi-structured text, the effect is better. The pre-training model involved in obtaining the embedding matrix in the solution can be replaced by an existing language model, a pre-training model or related technologies that will appear in the future.

[0036] The working principle of the present invention is as follows: first, through preprocessing, the input text is cleaned, the internal structure of the text is split, the relationship graph matrix is ​​extracted according to the connection between the elements, and the adjacency matrix M is obtained according to the different weights of the relationship. Secondly, the mature pre-training model Albert is used to embed the text to obtain the text vector matrix E. On this basis, we add an attention layer to each word vector Respectively through and query matrix W Q , key matrix W K , value matrix W V Multiply to get the query vector Key Vector Value vector pass Calculate the connection weights; Different from other graph attention mechanisms, this scheme directly multiplies the adjacency matrix and the weight matrix obtained after attention to obtain the weighted weights. Finally, the weights are After multiplication, it passes through the pooling layer and the fully connected layer, and is classified by softmax.

[0037] After adopting the above technical scheme, the beneficial effects of the present invention are as follows: for the text classification problem in the financial field, especially in the case of semi-structured text, ordinary classifiers cannot be combined with structured information for identification, and a method of extracting structural information and integrating it into the classification system is provided, which is effective in improving the accuracy of the final classifier. At the same time, for the model structure, especially taking the last few layers of transformer, the cross-sentence structure weight is higher than that within the same sentence, multi-headattention and other steps, the ability of information representation extraction is greatly improved, thereby increasing the accuracy of the final softmax classification, and having great market promotion value.

[0038] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A semi-structured text classification method based on graph attention, characterized by: It includes the following steps: Step 1, text preprocessing, data cleaning, forming a graph matrix M; Step 2, forming a coding layer, using the Albert pre-training model to obtain the embedding matrix; Step 3, performing attention operation on the embedding matrix, and performing weighted calculation based on the graph matrix; Step 4, compressing the feature vector matrix and performing text classification; The step 1 includes: firstly, pre-processing and cleaning the text, performing full-width and half-width conversion, unifying English uppercase and lowercase, merging multiple blank characters, removing stop words, and cutting the text. According to the judgment of the text line break, space bar, comma semicolon, and Tab key separator, the text data is cut into the form of multiple words or short sentences, and the position of the short sentences and words in the text, the position information of the rows and columns in which they are located, and their connection relationship are judged. Simple logical rules are used. In the horizontal direction, a word is directly associated with the words on the left and right. In the vertical direction, if they are in the same position in the horizontal direction, they are associated, so as to form a graph matrix M. In the expression of the graph matrix, different values ​​are used to express different associations. The association between adjacent characters in a single row is marked with a value of 1, and the association between elements is marked with 2. The remaining unmarked areas are all set to 0 by default; The second step includes: passing the text through the Albert pre-trained model trained on professional corpus in advance, and recording the output vector of the transformer of the penultimate y layer corresponding to each character as The outputs of the last four layers of transformers are concatenated, and the concatenated vector corresponding to each character is These vectors are combined to get the embedding matrix; The step three includes: the vector corresponding to each character in the embedding matrix As input, by comparing with the query matrix W Q , key matrix W K , value matrix W V Multiply to get the query vector Key Vector Value vector The calculation formula is as follows: Calculate the inner product of the query vector and the key vector and scale them to get the weight λ ij , aggregate all weights to get the initial weight matrix Φ, specifically: Then the initial weight matrix is ​​multiplied by the graph matrix M to obtain the weighted weight matrix Then multiply it with the value vector to output the representation vector matrix X. The calculation process is: Perform multi-head and norm operations on the above operations and then output; The step 4 includes: after obtaining the feature vector matrix, compressing the matrix into a vector by reducing its dimension through a max pooling layer or a mean pooling layer; then connecting to a fully connected layer, and finally entering a softmax layer to output the classification result. In training, a penalty term is added to the loss function to prevent the loss from decreasing due to an increase in the module length. During training, the recall indicator is used as a metric according to demand.

Images