Chinese Semantic Matching Method Based on Twin Interaction and Fine-Tuned Representation

Abstract

The present invention discloses a Chinese semantic matching method based on twin interaction and fine-tuned representation. First, the vector initialization of the text is completed with the RoBERTa-WWM-EXT pre-trained model. A twin structure embedded with a soft alignment attention mechanism (SA-Attention) and a BiLSTM training layer is constructed for the initial feature vectors to enhance the semantic interaction between sentence pairs. Secondly, the two texts to be matched are concatenated and input into the RoBERTa-WWM-EXT pre-trained model for vectorization. The concatenated vectorized results are input into the LSTM-BiLSTM network layer for enhanced training to strengthen the upper and lower semantic relationships within the sentences. Then, a training model for fine-tuning the initial vectors of RoBERTa-WWM-EXT is built to generate text vectors fine-tuned by label supervision, thereby further enhancing the representation strength of the vectors for the semantic relationship between texts, and finally achieving the purpose of improving the accuracy of Chinese semantic matching.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and particularly to a Chinese semantic matching method based on twin interaction and fine-tuned representation. Background Art

[0002] Chinese semantic matching is to judge the semantic matching degree between two different sentences. The core of the Chinese semantic matching task lies in mining the deep semantic information of the text and exploring the semantic relationship between different texts. The technology of text semantic matching can be applied to fields such as intelligent question answering, machine translation, and content retrieval. At present, the publicly disclosed text semantic matching technology still has the problem that the matching accuracy needs to be further improved. Summary of the Invention

[0003] In view of the problems existing in the current Chinese semantic matching technology, the present invention provides a Chinese semantic matching method based on twin interaction and fine-tuned representation of RoBERTa-WWE-EXT to improve the accuracy of the Chinese semantic matching task.

[0004] The method provided by the present invention first initializes the vectors of the text with the RoBERTa-WWM-EXT pre-trained model, constructs a twin structure embedded with a soft alignment attention mechanism (SA-Attention) and a BiLSTM training layer for the initial feature vectors to enhance the semantic interaction between sentence pairs. Secondly, the two texts to be matched are concatenated and input into the RoBERTa-WWM-EXT pre-trained model for vectorization, and the concatenated vectorized results are input into the LSTM-BiLSTM network layer for enhanced training to strengthen the upper and lower semantic relationships within the sentences. Then, a training model that can fine-tune the initial vectors of RoBERTa-WWM-EXT is built to generate text vectors fine-tuned by label supervision, thereby further enhancing the representation of the semantic relationship between texts by the vectors, and finally achieving the purpose of improving the accuracy of Chinese semantic matching.

[0005] The method provided by the present invention mainly includes the following steps:

[0006] (1) Connect two Chinese texts S1 and S2 to be matched to the RoBERTa-WWM-EXT pre-trained model to complete the vector initialization of texts S1 and S2, and extract the Pooler_out layer of the RoBERTa-WWM-EXT pre-trained model. The formula is described as S1vec = [RoBERTaWE([S1])] Pooler_out and S2vec = [RoBERTaWE([S2])] Pooler_out , where S1vec and S2vec are the preliminary feature vector representations of texts S1 and S2 respectively.

[0007] (2) Cross - input the vectors S1vec and S2vec into a siamese structure embedded with a soft - alignment attention mechanism (SA - Attention) and a BiLSTM training layer. During the processing of the soft - alignment attention mechanism, first calculate the attention scoring functions of S1vec and S2vec, then use the SoftMax function to calculate the final attention distribution function P, and then calculate the corresponding weighted distributions by multiplying P with vectors S1vec and S2vec respectively. and To ensure the integrity of the original data and avoid information loss of the vectors, the vectors are added to the original vectors S1vec and S2vec. The final vectors are denoted as Avec1 and Avec2. To further enhance the interaction of the internal information of each vector and further improve the representation ability of the feature vectors for text semantics, the output vectors Avec1 and Avec2 are respectively input into the BiLSTM network layer. Finally, after fusion, the feature vector representations SiaVec1 and SiaVec2 corresponding to texts S1 and S2 in two twin channels are generated. For vectors SiaVec1 and SiaVec2, perform subtraction and multiplication on corresponding items and connect the results. The formula is described as SiaVec = [|SiaVec1 - SiaVec2|, SiaVec1 ⊙ SiaVec2]. SiaVec is the interaction vector generated by texts S1 and S2 through the siamese structure.

[0008] (3) Connect texts S1 and S2 into a single - sentence text. The formula is Sen = [[CLS]S1[SEP]S2[SEP]], where [CLS] is the start identifier of the text and [SEP] is the interval identifier between the two texts. Sen is the single - sentence text formed by connecting the two texts. Input Sen into the RoBERTa - WE model and extract the vector Pvec of the Pooler_out output layer of the model as the vector representation of Sen. Input the vector Pvec into the LSTM layer to get the vector Lvec. To avoid losing the original text information, first connect the vector Lvec with the original vector Pvec, and then input this connected vector into the BiLSTM layer. Finally, the feature vector LBvec generated by texts S1 and S2 through the LSTM - BiLSTM structure is obtained.

[0009] (4) Add a linear transformation layer and a SoftMax activation layer based on the RoBERTa - WWM - EXT pre - trained model to build a sentence - pair pre - classification model that can fine - tune the representation parameters of the initial vectors of texts S1 and S2. First, extract the vector Pvec of the Pooler_out output layer of RoBERTa - WWM - EXT, and then input Pvec into a linear transformation layer. The formula is LWvec = Pvec·W T+Bias, where W is the weight matrix for the linear transformation of the vector Pvec, Bias is the bias of the function, and LWvec is the vector representation of Pvec after linear transformation. Then, the vector LWvec passes through the SoftMax activation layer, and the formula is where P LWvec is the final text pair matching result. Connect the dataset to the completed sentence pair pre-classification structure, train to generate the sentence pair pre-classification model PTM, and extract the Logits output layer as the feature vector for fine-tuning of the text pair. The formula is LGvec = [PTM([Sen])] Logits . LGvec is the fine-tuning representation feature vector of texts S1 and S2.

[0010] (5) In the previous steps, texts S1 and S2 pass through the siamese structure, the LSTM-BiLSTM network layer, and the fine-tuning structure, and respectively generate the corresponding vector representations SiaVec, LBvec, and LGvec. First, connect the vectors SiaVec and LBvec and participate in the training of the first two fully connected layers, layer1 and layer2, of the MLP. The formula is R L2 is the output vector of the two fully connected layers. Then, connect the vector LGvec with R L2 and connect it to the layer3 fully connected layer of the MLP to generate the vector R L3 . For the vector R L3 use the activation function Sigmoid. The formula is R is the language matching result of Chinese texts S1 and S2. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is the algorithm structure diagram related to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] The following further describes the present invention in detail to fully understand the technical solution of the present invention.

[0013] (1) The overall structure of the model in the present invention:

[0014] The overall structure of the model in the present invention is as Figure 1As shown, the overall structure consists of three network channels and an MLP network. In the NC1 channel, two texts are independently fed into the RoBERTa-WE model to obtain their initial vectors. A twin interaction structure with an embedded SA-Attention_BiLSTM is then constructed. The two initial vectors are cross-linked into two twin sub-channels, and fused to produce the twin interaction feature vector SiaVec. In the NC2 channel, the two text sentences are concatenated into a single sentence and fed into the RoBERTa-WE model. The output of the Pooler_out layer is used as the initial vector for the text pair. After passing through the LSTM-BiLSTM network training layer, the second feature vector LBvec for the sentence pair is generated. In the NC3 channel, a sentence pair classification model based on RoBERTa-WE fine-tuning is constructed. This model is trained on the dataset to generate a pre-trained sentence pair classification model PTM. The Logits layer of the PTM is extracted as the vector LGvec for the fine-tuned representation. In the MLP structure layer, the vectors SiaVec and LBvec are connected and input into the first two fully connected layers of the MLP. The output is connected with the vector LGvec and input into the last layer of the MLP. Sigmoid is used as the activation function to generate the final matching result of the sentence pair.

[0015] (2) RoBERTa-WWE-EXT initial vectorization of text:

[0016] The RoBERTa model uses a dynamic masking method and disables the NSP task in the BERT model. At the same time, it uses FULL-SENTENCES. The RoBERTa-WWM-EXT (RoBERTa-WE) model applies the full-word masking technology to the RoBERTa model and performs incremental training for Chinese. In the present invention, RoBERTa-WE is used as the baseline model for preliminary extraction of Chinese text features. The text is input into the RoBERTa-WE Chinese preprocessing model, and the Pooler_out layer of the RoBERTa-WE model is extracted as the initial vector representation of the sentence pair. The two texts S1 and S2 are independently input into the RoBERTa-WE model, and then the Pooler_out output layer vectors S1vec and S2vec of the model are extracted respectively. The formula description is shown in (1)(2).

[0017] S1vec=[RoBERTaWE([S1])] Pooler_out (1);

[0018] S2vec = [RoBERTaWE([S2])] Pooler_out (2);

[0019] (3) Twin interaction network structure:

[0020] Each twin channel in the twin structure embeds two vector processing modules, namely the Soft Alignment Attention mechanism (SA-Attention) and BiLSTM, such as Figure 1 the NC1 network channel in

[0021] In the NC1 channel, the feature vectors of two sentences are respectively input into two twin sub-channels. The two twin channels share weights to enhance the semantic interaction between the two sentences. At the same time, after the feature vector of each sentence is processed by each layer of the network structure, it is connected to the vector before processing to retain the original semantic features of the text and avoid information loss. Finally, the vectors calculated by the two sub-channels are fused to generate the final vector representation of the sentence pair. The network processing process is shown in the figure.

[0022] Score(S1vec,S2vec) = S1vec T ·S2vec (3);

[0023]

[0024] The attention distribution function P is multiplied by the vectors S1vec and S2vec respectively to calculate the corresponding weighted distributions, as shown in formulas (5) and (6). After the original vectors S1vec and S2vec are calculated by the soft alignment attention mechanism, new vector representations are generated

[0025]

[0026]

[0027] To ensure the integrity of the original data and avoid information loss of the vectors, the vectors are added to the original vectors S1vec and S2vec to obtain the final vector representations Avec1 and Avec2, as shown in formulas (7) and (8). Avec1 and Avec2 contain both the processing information of SA-Attention and the original information of the vectors.

[0028]

[0029]

[0030] The output vectors Avec1 and Avec2 are respectively connected to the BiLSTM network layer to enhance the interaction of the internal information of the vectors Avec1 and Avec2, and further improve the representation ability of Avec1 and Avec2 for text semantics. The formulas are shown in (9) and (10).

[0031] Bvec1 = BiLSTM(Concat[Avec1, S1evc]) (9);

[0032] Bvec2 = BiLSTM(Concat[Avec2, S2evc]) (10);

[0033] In formula (9), Concat[Avec1, S1evc] means connecting the vector Avec1 and the vector Svec1 to avoid the loss of information of the original vector. Bvec1 is the vector generated by the BiLSTM model. The same applies to formula (10).

[0034] To fully retain the semantic information of each processing process, the vectors generated in each stage are connected with the initial vector together as the final feature vector of the text, as shown in formulas 11 and 12. SiaVec1 and SiaVec2 respectively represent the final vector representations generated by the text S1 and S2 after being processed by the siamese interaction structure.

[0035] SiaVec1 = Concat[S1vec, Avec1, Bvec1] (11);

[0036] SiaVec2 = Concat[S2vec, Avec2, Bvec2] (12);

[0037] For the vectors SiaVec1 and SiaVec2, corresponding terms are subtracted and multiplied, and the results are connected to generate the final vector representation SiaVec of the sentence pair, as shown in formula (13).

[0038] SiaVec = [|SiaVec1 - SiaVec2|, SiaVec1 ⊙ SiaVec2] (13);

[0039] The vector SiaVec generated by the siamese interaction structure not only retains the original semantic information of the text but also enhances the semantic interaction between the two texts.

[0040] (4) LSTM - BiLSTM network structure:

[0041] The network structure is like Figure 1 the NC2 network channel in. This channel connects the text pair S1 and S2 into one text, as shown in formula (14).

[0042] Sen = [[CLS]S1[SEP]S2[SEP]] (14);

[0043] Input Sen into the RoBERTa-WE model, and extract the vector Pvec of the Pooler_out output layer of the model as the vector representation of Sen, as shown in formula (15).

[0044] Pvec = [RoBERTaWE([Sen])] Pooler_out (15);

[0045] First, connect the vector Pvec to the LSTM layer to obtain the vector Lvec, as shown in formula (16).

[0046] Lvec = LSTM([Pvec]) (16);

[0047] To avoid losing the original text information, first connect the vector Lvec with the original vector Pvec, and then connect the concatenated vector to the BiLSTM layer, as shown in formula (17). LBvec is the final output vector of the NC2 network channel.

[0048] LBvec = BiLSTM(Concat[Lvec,Pvec]) (17);

[0049] The vector LBvec further enhances the semantic interaction within a single text and between two texts based on the sentence pair vector extracted by RoBERTa-WE, and enriches the representation information of the sentence pair contained in the vector.

[0050] (5) Text feature representation structure based on RoBERTa-WWM-EXT fine-tuning:

[0051] The structure is as shown in Figure 1 the NC3 network channel in. The sentence pair vector representation model based on RoBERTa-WE fine-tuning adds a linear transformation layer and a SoftMax activation layer on the basis of RoBERTa-WE, and also has a fine-tuning mechanism for vector parameters.

[0052] Extract the vector Pvec of the Pooler_out output layer of RoBERTa-WE, and then pass Pvec through the linear transformation layer of the sentence pair classification model. The transformation formula is as shown in (18).

[0053] LWvec = Pvec·W T + Bias (18);

[0054] In formula (18), W is the weight matrix for the linear transformation of vector Pvec, Bias is the bias of the function, and LWvec is the vector representation of Pvec after linear transformation. The vector LWvec passes through the SoftMax activation layer to obtain the final text pair matching result P LWvec , as shown in formula (19).

[0055]

[0056] For supervised training of the data, a sentence pair classification pre-training model PTM is generated, and the Logits output layer is extracted as the fine-tuning feature vector of the text pair, as shown in formula 20.

[0057] LGvec = [PTM([Sen])] Logits (20);

[0058] The text feature representation vector LGvec of the sentence pair classification model fine-tuned based on RoBERTa-WE is characterized in that the internal information of the vector directly contains the semantic matching relationship between the text pairs, which plays a key role in the subsequent judgment of the text pair matching degree.

[0059] (6) MLP structure:

[0060] After the analysis in the previous section, the vector LGvec contains the semantic matching information of the sentence pair. Therefore, the vector LGvec does not participate in the training in the first two fully connected layers of the MLP layer to avoid the loss of matching information. After the vectors SiaVec and LBvec are trained for two layers, then let the vector LGvec participate in the subsequent training, as Figure 1 in the MLP structure.

[0061] First, connect the vectors SiaVec and LBvec and participate in the training of the first two fully connected layers of the MLP. The formula description is as shown in (21), and R L2 is the output of the two fully connected layers.

[0062]

[0063] Then connect the vector LGvec with R L2 and participate in the generation of vector R in the third fully connected layer of the MLP L3 , and the vector R L3 passes through the activation function Sigmoid to obtain the final matching result R, and the meaning is as shown in formula (22).

[0064]

[0065] Among them, R = 0 indicates that the semantics of texts S1 and S2 do not match, and S = 1 indicates that the semantics between texts S1 and S2 match.

[0066] Technical features and effects of the present invention: In the invention, a twin interaction structure embedded with a soft alignment attention mechanism and BiLSTM is designed to enhance the semantic interaction between two texts; an LSTM-BiLSTM network layer is built to enhance the expression of vector for text context semantic information; a text feature representation structure based on RoBERTa-WWM-EXT fine-tuning is constructed, and the extracted vector integrates the semantic matching information between sentence pairs, improving the representation strength of the vector for the semantic relationship between texts; a specific MLP network structure is designed for the three vector representations, and finally the purpose of improving the accuracy of Chinese semantic matching is achieved.

Claims

1. A Chinese semantic matching method based on twin interaction and fine-tuning representation, characterized by: The method comprises the following steps: (1) The two texts S1 and S2 to be matched are respectively connected to the RoBERTa-WWM-EXT pre-trained model, and then the output vectors S1vec and S2vec of the Pooler_out layer of the RoBERTa-WWM-EXT pre-trained model are extracted respectively, where S1vec and S2vec are the preliminary feature vector representations of texts S1 and S2 respectively; (2) The vectors S1vec and S2vec are cross-input into the twin structure with the soft alignment attention mechanism and BiLSTM training layer embedded; in the soft alignment attention mechanism processing, the attention scoring function of S1vec and S2vec is first calculated, and then the final attention distribution function P is calculated using the SoftMax function, and then the corresponding weighted distribution vector P is obtained by multiplying the vectors S1vec and S2vec. and vector Add them to the original vectors S1vec and S2vec respectively, and the final output vectors are represented as Avec1 and Avec2; the output vectors Avec1 and Avec2 are connected to the BiLSTM network layer respectively, and finally the feature vectors SiaVec1 and SiaVec2 corresponding to the texts S1 and S2 in the two twin channels are generated after fusion; the corresponding items of vectors SiaVec1 and SiaVec2 are subtracted and multiplied, and the results are connected. The formula is described as SiaVec=[|SiaVec1-SiaVec2|,SiaVec1⊙SiaVec2], where SiaVec is the interaction vector generated by the twin structure of texts S1 and S2; (3) Connect the texts S1 and S2 into a single sentence text, the formula is Sen = [[CLS]S1[SEP]S2[SEP]], [CLS] is the beginning mark of the text, [SEP] is the interval mark of the two texts, and Sen is the single sentence text connected by the two texts; input Sen into the RoBERTa-WWM-EXT model, extract the model's Pooler_out output layer vector Pvec as the vector representation of Sen; connect the vector Pvec to the LSTM layer to obtain the vector Lvec; connect the vector Lvec with the original vector Pvec, and then connect this connection vector to the BiLSTM layer, and finally obtain the feature vector LBvec generated by the text S1 and S2 through the LSTM-BiLSTM structure; (4) Add a linear transformation layer and a SoftMax activation layer to the RoBERTa-WWM-EXT pre-trained model to build a sentence pair pre-classification model that can fine-tune the representation parameters of the initial vectors of texts S1 and S2; first extract the Pooler_out output layer vector Pvec of RoBERTa-WWM-EXT, and then connect Pvec to a linear transformation layer, the formula is LWvec = Pvec·W T +Bias, where W is the weight matrix for linear transformation of vector Pvec, Bias is the bias of the function, and LWvec is the vector representation of Pvec after linear transformation; then the vector LWvec is passed through the SoftMax activation layer, and the formula is Among them, P LWvec The final text pair matching result is obtained by connecting the dataset to the constructed sentence pair pre-classification structure, training the sentence pair pre-classification model PTM, and extracting the Logits output layer as the fine-tuned feature vector of the text pair. The formula is LGvec = [PTM([Sen])] Logits , where LGvec is the fine-tuned representation feature vector of texts S1 and S2; (5) In the previous steps, the texts S1 and S2 are processed through the twin structure, LSTM-BiLSTM network layer and fine-tuning structure to generate corresponding vector representations SiaVec, LBvec and LGvec respectively; first, the vectors SiaVec and LBvec are connected to participate in the training of the first two fully connected layers of layer1 and layer2 of MLP, and the formula is R L2 is the output vector of the two fully connected layers; then the vector LGvec is combined with R L2 Connect and access the layer3 fully connected layer of MLP to generate vector R L3 , for vector R L3 Using the activation function Sigmoid, the formula is R is the semantic matching result of Chinese texts S1 and S2.

Images