Chinese semantic matching method based on RoBERTa-wwm-ext-large pre-training model

Abstract

The invention discloses a Chinese semantic matching method based on a RoBERTa-wwm-ext-lane pre-training model, and the method comprises the steps: obtaining the initial vector representation of a matched text through the pre-training model RoBERTa-wwm-ext-lane, and enhancing the semantic interaction between two text vectors through feature attention; then, the text vector is sent into a convolutional neural network to extract more feature information, then the feature information is output to a long and short-term memory network, and context information of the text vector is enriched through self-attention; and establishing a fine tuning representation model to process the initial text connection vector to obtain a fine-tuned text pair connection vector. And finally, inputting the text vectors into a multi-layer perceptron network, and calculating a text matching result. Compared with an existing text matching algorithm based on a pre-training model, the method has higher accuracy and better generalization.

Description

Technical Field

[0002] The present invention relates to the field of computer applications, and in particular to a Chinese semantic matching method based on a RoBERTa-wwm-ext-large pre-training model. Background Art

[0003] Text matching is one of the basic tasks in natural language processing, and its purpose is to determine whether two different texts express similar semantics. The core task of text matching is to mine the deep semantic information of the text and the semantic relationship between different texts, so as to determine whether two texts match. The research on text matching can be applied to many fields such as machine translation, information retrieval, interpretation recognition and intelligent question answering. The currently disclosed text matching methods based on the BERT pre-training model still have problems such as limited interaction of semantic information between text sequences, insufficient encoding ability of sentence pairs, and the need to further improve the matching accuracy. Summary of the invention In order to solve the above-mentioned problems existing in the text matching method based on the BERT pre-training model, the present invention provides an improved Chinese semantic matching method based on the RoBERTa-wwm-ext-large pre-training model. Compared with the existing text matching algorithm based on the pre-training model, the present invention has higher accuracy and better generalization.

[0004] First, the initial vector representation of the matching text is obtained through the pre-trained model RoBERTa-wwm-ext-large. Then, feature attention is used to strengthen the semantic interaction between the two text vectors. Then, the text vector is sent to the convolutional neural network to extract more feature information, and then output to the long short-term memory network to extract the time series dependency of these feature information, and then the context information of the text vector is enriched through self-attention. At the same time, a fine-tuning representation model is established to process the initial text connection vector and obtain the fine-tuned text pair connection vector. Finally, these text vectors are input into the multi-layer perceptron network to calculate the text matching results, and the model is applied to the target dataset to complete the text matching task.

[0005] The technical solution of the present invention mainly comprises the following steps: (1) The RoBERTa-wwm-ext-large pre-trained model is optimized for Chinese. During the pre-training process, the entire word is masked instead of a single character. This is very useful for a language like Chinese that is not separated by spaces. Therefore, the present invention uses the RoBERTa-wwm-ext-large pre-trained model as a baseline model to obtain the initial vectors of text pairs and connected texts.

[0006] (2) Then, a CNN-LSTM-SA convolutional fusion interaction structure is constructed, and Feature Attention is introduced before it. This structure combines the convolutional neural network (CNN) and the long short-term memory (LSTM) network, and adds self-attention, which can enhance the contextual information of a single text and the semantic interaction ability between two texts, and extract more semantic information from the text.

[0007] First, the convolutional neural network CNN is used to perform convolution operations on the input data to extract more feature information, and then its output is used as the input of the long short-term memory network LSTM to process the time series dependency of these features; then the intermediate vectors obtained by passing the two Chinese original text S1 and S2 representation vectors through the convolutional neural network CNN and the long short-term memory network LSTM are interacted through SA attention, and finally the three text vectors obtained after the interaction are connected to obtain the final feature vector representation Vec of the two texts S1 and Vec S2 .

[0008] (3) Then, a fine-tuning representation training model is established, and the initial vector generated by the connected text pairs is trained through label supervised training. The Logits layer vector is extracted as the fine-tuning representation vector of the text pair, and the representation of the semantic relationship between the texts is further improved.

[0009] Preferably, the fine-tuning represents the construction of a training model: first, the output vector of the Pooler_out layer of the RoBERTa-wwm-ext-large model is input into the linear transformation layer to obtain a temporary vector Svec_temp; then, through label supervised training, the label semantics are integrated to generate a fine-tuning model P; finally, the fine-tuning model P is used to fine-tune Svec_temp to integrate the label semantics to obtain a fine-tuning vector Vec S12 .

[0010] (4) Finally, the two text vectors processed by the convolutional fusion structure and the attention mechanism are connected, and they and the connected text pair vector extracted from the fine-tuning representation model are sent to the multi-layer perceptron (MLP) matching structure to calculate the final matching result.

[0011] Preferably, the multi-layer perceptron MLP matching structure consists of three layers. First, the two text feature vectors Vec obtained after the CNN-LSTM-SA convolution fusion interaction structure are S1 and Vec S2 Input to the first layer of MLP structure for training; then, the feature vector Vec S12It is injected into the second-layer MLP structure and trained to obtain R2. Finally, R2 is sent to the third-layer MLP structure for training to obtain R3. R3 is calculated by the activation function Sigmoid to obtain the final matching result R. Among them, R is equal to 0 or 1. If R is equal to 0, it means that the text semantics do not match; if R is equal to 1, it means that the text semantics match.

[0012] Technical features and effects of the present invention: By constructing a CNN-LSTM-SA convolutional fusion interaction structure and introducing feature attention before it, plus self-attention, the two text feature vectors processed by feature attention and self-attention are rich in more semantic interaction information, and the processing of CNN and LSTM also extracts more encoding information of sentence pairs. Through the established fine-tuning representation training model, the Logits layer vector is extracted as the fine-tuning representation vector of the text pair, and the representation of the semantic relationship between the texts is further improved by the vector, which enriches the encoding information of the sentence pair, and finally improves the accuracy of text matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall framework diagram of the algorithm of the present invention; Figure 2 This is a schematic diagram of the CNN-LSTM-SA network structure; Figure 3 This is the structure diagram of the MLP multi-layer perceptron. DETAILED DESCRIPTION

[0014] The technical scheme and technical effects of the present invention are described in detail below with reference to the accompanying drawings. The overall structure of the model in the present invention is as follows: Figure 1 As shown in the figure, it mainly includes four structures, namely, text initial vector generation structure, CNN-LSTM-SA network structure, fine-tuning representation structure, and multi-layer perceptron matching structure.

[0015] (1) Text initial vector generation structure: Taking RoBERTa-wwm-ext-large as the baseline model, the original Chinese texts S1, S2, and S after the concatenation of S1 and S2 to be matched are connected to the RoBERTa-wwm-ext-large pre-trained model to complete the vector initialization of the three original texts S1, S2 and S.

[0016] (2) CNN-LSTM-SA network structure: like Figure 2As shown, CNN-LSTM combines the advantages of convolutional neural networks and long short-term memory networks, and can be used to process data with spatial and temporal structures. The convolutional neural network (CNN) part is used to extract local features of the input data, while the long short-term memory network (LSTM) part is used to process time series data and capture temporal dependencies. In this structure, we first use CNN to perform convolution operations on the input data to extract more feature information, and then use the output of CNN as the input of LSTM to process the time series dependencies of these features. The intermediate vectors obtained by passing the two text representation vectors through CNN and LSTM are interacted through SA attention, and finally the three text vectors obtained by CNN, LSTM and after attention interaction are connected to obtain the final feature vector representation Vec of the two texts. S1 and Vec S2 . Vec after feature attention and self-attention processing S1 and Vec S2 It is rich in semantic interaction information, and the processing of CNN and LSTM also extracts more encoding information of sentence pairs.

[0017] (3) Fine-tune the representation structure: The feature vector directly extracted from the RoBERTa-wwm-ext-large model ignores the impact of labels on the text feature vector representation. We built a RoBERTa-wwm-ext-large training model with a fine-tuning mechanism and adjusted the feature vector parameters of the text through label supervision to further enrich the encoding information of the sentence pair. First, the output vector of the Pooler_out layer of RoBERTa-wwm-ext-large is input into the linear transformation layer to obtain the temporary vector Svec_temp, which is calculated as follows: Svec_temp =, Svec * W T +Bvecs , where WT is the weight matrix and Bvecs is the function bias. Then, through label supervised training, the label semantics are integrated to generate a fine-tuning model P. Finally, the fine-tuning model P is used to fine-tune Svec_temp to integrate the label semantics and obtain the fine-tuning vector Vec S12 , the calculation formula is Vec S12 =[ P ( Svec_ temp )] Logits .

[0018] (4) Multi-layer perceptron matching structure: like Figure 3As shown in the figure, the multi-layer perceptron (MLP) matching structure consists of three layers. First, the two text feature vectors Vec obtained after the CNN-LSTM-SA structure are S1 and Vec S2 Input to the first layer of MLP structure for training, the formula is R1=MLP1 ( concat ( Vec S1 ,Vec S2 )). Then, Vec S12 Inject it into the second-layer MLP structure and participate in the training to get R2, the formula is R2=MLP2 ( concat ( R1,Vec S12 )). Finally, R2 is sent to the third-layer MLP structure for training to obtain R3. The vector R3 is calculated by the activation function Sigmoid to obtain the final matching result R. Among them, R is equal to 0 or 1. If R is equal to 0, it means that the text semantics do not match; if R is equal to 1, it means that the text semantics match.

Claims

1. A Chinese semantic matching method based on the RoBERTa-wwm-ext-large pre-training model, characterized in that: The method comprises: (1) Using RoBERTa-wwm-ext-large as the pre-trained model, the original Chinese texts S1, S2 and S, which is the concatenation of S1 and S2, are connected to the RoBERTa-wwm-ext-large pre-trained model to complete the vector initialization of the three original texts S1, S2 and S; (2) Construct a CNN-LSTM-SA convolutional fusion interaction structure. First, use the convolutional neural network CNN to perform convolution operations on the input data to extract more feature information, and then use its output as the input of the long short-term memory network LSTM to process the time series dependency of these features; then, the intermediate vectors obtained by passing the two Chinese original text S1 and S2 representation vectors through the convolutional neural network CNN and the long short-term memory network LSTM are interacted through SA attention. Finally, the three text vectors obtained after the interaction are connected to obtain the final feature vector representation Vec of the two texts. S1 and Vec S2 ; (3) Establish a fine-tuning representation training model and train the initial vector generated by the connected text pairs through label supervised training, extract its Logits layer vector as the fine-tuning representation vector of the text pair, and further improve the vector's representation of the semantic relationship between texts; (4) Finally, the two text feature vectors Vec processed by the CNN-LSTM-SA convolutional fusion interaction structure and attention mechanism are S1 and Vec S2 Connect and send it and the connected text pair vector extracted from the fine-tuning representation training model into the multi-layer perceptron MLP matching structure to calculate the final matching result.

2. The method according to claim 1, characterized in that The fine-tuning represents the construction of the training model: first, the output vector of the Pooler_out layer of the RoBERTa-wwm-ext-large model is input into the linear transformation layer to obtain a temporary vector Svec_temp; then, through label supervision training, the label semantics are integrated to generate a fine-tuning model P; finally, the fine-tuning model P is used to fine-tune Svec_temp to integrate the label semantics, and obtain the fine-tuning vector Vec S12 .

3. The method according to claim 1, characterized in that The multi-layer perceptron MLP matching structure consists of three layers. First, the two text feature vectors Vec obtained after the CNN-LSTM-SA convolution fusion interaction structure are S1 and Vec S2 Input to the first layer of MLP structure for training; then, the feature vector Vec S12 It is injected into the second-layer MLP structure and trained to obtain R2. Finally, R2 is sent to the third-layer MLP structure for training to obtain R3. R3 is calculated by the activation function Sigmoid to obtain the final matching result R. Among them, R is equal to 0 or 1. If R is equal to 0, it means that the text semantics do not match; if R is equal to 1, it means that the text semantics match.

Images