A Chinese text matching method based on language pre-training model

Abstract

The invention discloses a Chinese text matching method based on a language pre-training model, which belongs to the technical field of text matching. The pre-trained language model is used as a sequence classifier by fine-tuning on a downstream classification task, two sentences to be matched are spliced ​​together by separating characters as the input of the model, and a prediction result is output; in the fine-tuning process, a DC-Match method is used to improve the prediction accuracy; in the prediction process, for the fine-tuning layer after the Encoder, a multi-layer feature aggregation classification module is used to replace the ordinary classification layer, which can significantly improve the accuracy of the Chinese matching task, and can be combined with most Chinese language pre-training models. Compared with the DC-Match method, on its basis, the attention fusion method proposed in this paper is used, and the loss function is improved, so as to improve the fine-tuning effect of the Chinese pre-training model.

Description

Technical Field

[0001] The present invention belongs to the technical field of text matching, and in particular relates to a Chinese text matching method based on a language pre-training model. Background Art

[0002] In the existing technology, there are many methods for text matching: traditional methods, methods based on deep learning, etc., and the pre-trained language model (PLM) shows very good representation learning ability. When applied to the task of text semantic matching, it is usually fine-tuned by adding a simple classification target. Because the language pre-model has been trained with a large amount of text and multiple methods, it has excellent feature extraction effect.

[0003] When applied to the task of text semantic matching, it is usually fine-tuned by adding a simple classification target and directly performing text comparison by uniformly processing each word, which does not fully utilize the feature extraction potential of the pre-trained model. In the DC-Match method, each sentence is decomposed into content with different matching granularities, providing an idea to disentangle keywords from intents and improving the performance of model fine-tuning.

[0004] The fine-tuning method is relatively simple and fails to take advantage of the powerful feature extraction capabilities of the language pre-trained Chinese model, so the text matching effect needs to be improved. In addition, in the Chinese pre-trained model, the semantic information cannot be fully decomposed and utilized during the fine-tuning training process, and the keywords and intent information in the sentence cannot be distinguished. Although the DC-Match method provides ideas for extracting keywords, its performance in Chinese datasets still needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to provide a Chinese text matching method based on a language pre-training model to solve the technical problems mentioned in the above background technology.

[0006] In order to achieve the above object, the present invention discloses a Chinese text matching method based on a language pre-training model, comprising the following steps:

[0007] (1) Use the pre-trained language model (PLM) as a sequence classifier by fine-tuning it on the downstream classification task. The two sentences to be matched are concatenated by separating characters as the input of the model and the prediction result is output.

[0008] During the fine-tuning process, the DC-Match method is used to separate intent from keywords. The feature prediction keywords and intent information extracted by the pre-trained model encoder are covered with masks respectively. The model is made to learn to distinguish between keywords and intent, thereby improving the prediction accuracy.

[0009] During the prediction process, for the fine-tuning layer after the encoder, the features of the last four layers of the pre-trained language model are extracted separately. For each layer of features, the structure of the extracted four layers of features is [size, token, hidden_states]. For each token dimension, the largest hidden layer vector is taken out and then concatenated. The concatenated structure is [size, hidden_states*4], which is used as the input to the aggregation classifier.

[0010] Furthermore, there are multiple aggregate classification modules in the aggregate classifier, each of which is composed of a linear layer, a GELU activation function, and an attention module. All attention modules share parameters and finally enter the full classification layer to predict the matching results.

[0011] Furthermore, the pre-trained language model also includes a training part.

[0012] Furthermore, the training part obtains the hidden features of the last layer, which are then processed by keyword and intent mask respectively, and then enter the training feature classification module.

[0013] Furthermore, the training feature classification module includes an activation function, an attention mechanism and a fully connected layer, wherein the attention mechanism shares parameters with the attention module and is added to the result of the activation function in a weighted form, and the weight size is a hyperparameter.

[0014] Furthermore, the losses during training include: (1) the total loss L of model classification sm ; (2) According to the remote supervision method, it is used to measure the loss L of the model learning intent and keywords ds ; (3) by L ds The resulting auxiliary training objective cannot be directly associated with the original text matching problem. In order to allow the prediction of keywords and intents to affect the final prediction, the model can be encouraged to learn to distinguish keywords from intents during normal reasoning. The method treats keyword matching and intent matching as two sub-problems of text matching, and assumes that they are independent of each other. The results of the sub-problems are then combined to solve the original problem. Specifically, the method uses the bidirectional KL divergence loss to measure the original prediction distribution P sm Q with keyword intent ds The distance between the predicted distributions, denoted by L dc ;

[0015] The above three losses are weighted and added together. Based on the original loss function, the weight of the intent matching loss is improved and the loss function is expanded. The loss function is as follows:

[0016]

[0017] Furthermore, the two texts S a and S b As a continuous sequence S a,b =[S a ;w sep ; S b ], and the loss function L is obtained sm =-ln P(y|S a ,S b ).

[0018] Furthermore, for L ds , the output state of the pre-trained language model is divided into two parts and Character sequences corresponding to keywords and intents, respectively, where N k 、N i Represents the number of characters, thus deriving the loss function Among them, W ds is a trainable parameter, and It is through Use average pooling to get, where W ds is a trainable parameter, and It is the hidden information (hidden_states) output by the last layer of PLM, which are the hidden information of keywords and the hidden information of intent. Its goal is to drive the encoder to learn the representation of keywords and intents so that they can be separated from each other, thereby modeling different matching levels.

[0019] Furthermore, for L dc ,

[0020] Use the bidirectional KL divergence loss to minimize P(y|S a ,S b ) and Q(y|S a ,S b ), P(y|S a ,S b ) and Q(y|S a ,S b ) represent keyword matching and intent matching respectively, and assume that they are independent of each other.

[0021] Compared with the prior art, the Chinese text matching method based on the language pre-training model of the present invention has the following advantages:

[0022] (1) Fine-tuning uses a multi-layer feature aggregation classification module to replace the ordinary classification layer, which can significantly improve the accuracy of Chinese matching tasks and can be combined with most Chinese language pre-training models.

[0023] (2) The present invention uses the attention fusion method proposed in this paper and improves the loss function, thereby improving the fine-tuning effect of the Chinese pre-training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 : Structure diagram based on pre-trained language model. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described in detail below through specific embodiments.

[0026] The Chinese text matching method based on the language pre-training model includes the following steps:

[0027] Use a pre-trained language model (PLM) as a powerful sequence classifier by fine-tuning it on downstream classification tasks. The two sentences to be matched are concatenated by separating characters as the input of the model, and the output is the prediction result.

[0028] During the fine-tuning process, the DC-Match method is used to separate intent from keywords. The feature prediction keywords and intent information extracted by the pre-trained model encoder are covered with masks respectively. By allowing the model to learn to distinguish between keywords and intent, the prediction accuracy is improved.

[0029] In the prediction reasoning process, for the fine-tuning layer after the Encoder, the features of the last 4 layers of PLM are extracted separately. For each layer of features, the structure of the extracted 4 layers of features is [size, token, hidden_states]. For each token dimension, the largest hidden layer vector is taken out and then concatenated. The concatenated structure is [size,,hidden_states*4], which is used as input to the aggregation classifier.

[0030] There are n aggregate classification modules in the aggregate classifier (the specific number of n varies depending on the language pre-training model, and the default is 2). Each aggregate classifier consists of a linear layer, a GELU activation function, and an attention module. All attention modules share parameters and finally enter the full classification layer to predict the matching results.

[0031] The above is the network structure of the model during prediction and reasoning. During the training process, since keywords and intents need to be predicted, the model structure during training also includes a training part.

[0032] Different from the reasoning structure, the hidden features of the last layer are obtained in the training part, which are then processed by keyword and intent mask respectively, and then enter the training feature classification module.

[0033] The training feature classification module includes an activation function, an attention mechanism, and a fully connected layer. The attention mechanism shares parameters with the attention module and is added to the result of the activation function in a weighted form, with the weight size being a hyperparameter.

[0034] There are three places to calculate the loss during training: (1) The total loss L of the model classification sm ; (2) According to the remote supervision method, it is used to measure the loss L of the model learning intent and keywords ds ; (3) by L ds The resulting auxiliary training objective cannot be directly associated with the original text matching problem. In order to allow the prediction of keywords and intents to affect the final prediction, the model can be encouraged to learn to distinguish keywords from intents during normal reasoning. The method treats keyword matching and intent matching as two sub-problems of text matching, and assumes that they are independent of each other. The results of the sub-problems are then combined to solve the original problem. Specifically, the method uses the bidirectional KL divergence loss to measure the original prediction distribution P sm Q with keyword intent ds The distance between the predicted distributions, denoted by L dc ;

[0035] Finally, the above three losses are weighted and added together, and the original loss function is improved to expand the weight of the intent matching loss. The loss function is as follows:

[0036]

[0037] Put two text S a and S b As a continuous sequence S a,b =[S a ;w sep ; S b ], and the loss function L is obtained sm =-ln P(y|S a ,S b ),

[0038] for We divide the output status of PLM into two parts and Character sequences corresponding to keywords and intents, respectively, where N k 、N i Represents the number of characters, thus deriving the loss function Among them, W ds is a trainable parameter, and is through It is through Use average pooling to get it. Among them, W dsis a trainable parameter, and It is the hidden information (hidden_states) output by the last layer of PLM, which are the hidden information of keywords and the hidden information of intent. Its goal is to drive the encoder to learn the representation of keywords and intents so that they can be separated from each other, thereby modeling different matching levels.

[0039] For L dc ,

[0040] Use the bidirectional KL divergence loss to minimize P(y|S a ,S b ) and Q(y|S a ,S b ), P(y|S a ,S b ) and Q(y|S a ,S b ) represent keyword matching and intent matching respectively, and assume that they are independent of each other.

[0041] When the model is used, the model structure is as follows Figure 1 As shown in the figure, at this stage the model does not need the assistance of the DC-Match method for keyword intent matching, and only the PLM and fine-tuning modules are involved in the prediction task.

[0042] Since only one fully connected layer is used to classify the pre-trained text model in the fine-tuning paradigm, this is not the best choice for feature classification, and only the last layer of feature information of the PLM model is used. In the PLM model, the shallower the Encode layer, the more the sentence vector can represent low-level semantic information, and the deeper it is, the higher-level semantic information it represents. If the information of multiple Encoder layers is spliced, richer semantic feature information will be obtained. Based on the above ideas, the present invention proposes a fine-tuning model based on a pre-trained language model for Chinese text semantic matching.

[0043] In the PLM model structure, excluding the first input layer, it is composed of 12 or 24 layers of Encoder layers stacked, from which the last 4 layers are extracted, and the token dimension of each layer is taken as the max to obtain four hidden layer vectors, which are then concatenated into a feature vector of 4 times the size of the hidden layer. Then, it enters the multi-layer feature concatenation classification module composed of the linear layer, activation function, and self-attention module proposed in this paper, and finally the classification output is obtained.

[0044] During formal reasoning, no training is required. Only the aggregate classifier is needed to classify the features extracted by the PLM decoder and finally predict the text matching results.

[0045] Relying on this method, in multiple dataset tasks, such as Medical-SM matching tasks, the accuracy, F1-score, and precision indicators have been significantly improved, proving the effectiveness of this method.

[0046] The above method was used for experimentation. Text similarity experiments were conducted on a specialized Chinese Medical-SM dataset. This dataset consists of user-generated query pairs collected from Chinese search engines. The dataset contains 48,008 query pairs in the field of medical consultation. Each query pair can be classified into three categories: full match, partial match, or no match. The annotations were done by five independent experts, and the dataset retains label selection.

[0047] The ablation experiment results of this method based on different pre-trained language models are shown in Tables 1 and 2.

[0048] Table 1: ACC index ablation experiment table

[0049]

[0050] Table 2: F1 index ablation experiment table

[0051]

[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the design concept of the present invention should be included in the protection scope of the present invention.

Claims

1. A Chinese text matching method based on a language pre-training model, characterized in that: The steps include: Use the pre-trained language model as a sequence classifier by fine-tuning it on downstream classification tasks. Use the two sentences to be matched as input to the model after concatenating them by separating characters, and output the prediction result. During the fine-tuning process, the DC-Match method is used to separate intent from keywords. The feature prediction keywords and intent information extracted by the pre-trained language model encoder are covered with masks respectively. The model is made to learn to distinguish between keywords and intent, thereby improving the prediction accuracy. During the prediction process, for the fine-tuning layer after the encoder, the features of the last four layers of the pre-trained language model are extracted separately. For each layer of features, the structure of the extracted four layers of features is [size, token, hidden_states]. For each token dimension, the largest hidden layer vector is taken out and then concatenated. The concatenated structure is [size, hidden_states*4], which is used as the input to the aggregation classifier. There are multiple aggregate classification modules in the aggregate classifier, each of which consists of a linear layer, a GELU activation function, and an attention module. All attention modules share parameters and finally enter the full classification layer to predict the matching results. The pre-trained language model also includes a training part; The training part obtains the hidden features of the last layer, which are then processed by keyword and intent mask respectively, and then enter the training feature classification module; The training feature classification module includes an activation function, an attention mechanism and a fully connected layer, wherein the attention mechanism shares parameters with the attention module and is added to the result of the activation function in a weighted form, and the weight size is a hyperparameter.

2. The Chinese text matching method based on the language pre-training model according to claim 1, characterized in that: The losses during training include: (1) the total loss L of the model classification sm ; (2) According to the remote supervision method, it is used to measure the loss L of the model learning intent and keywords ds ; (3) Use the bidirectional KL divergence loss to measure the original predicted distribution P sm Q with keyword intent ds The distance between the predicted distributions, denoted by L dc ; The above three losses are weighted and added together. Based on the original loss function, the weight of the intent matching loss is improved and the loss function is expanded. The loss function is as follows:

3. The Chinese text matching method based on the language pre-training model as claimed in claim 2, characterized in that: Put two text S a and S b As a continuous sequence S a,b =[S a ;w sep ; S b ], and the loss function L is obtained sm =-ln P(y|S a ,S b ), the P(y|S a ,S b ) indicates keyword matching.

4. The Chinese text matching method based on the language pre-training model as claimed in claim 2, characterized in that: For L ds , the output state of the pre-trained language model is divided into two parts and Character sequences corresponding to keywords and intents, respectively, where N k 、N i Represents the number of characters, thus deriving the loss function Among them, W ds is a trainable parameter, and It is the hidden information (hidden_states) output by the last layer of PLM, which are the hidden information of keywords and the hidden information of intent. Its goal is to drive the encoder to learn the representation of keywords and intents so that they can be separated from each other, thereby modeling different matching levels.

5. The Chinese text matching method based on the language pre-training model as claimed in claim 2, characterized in that: For L dc , Use the bidirectional KL divergence loss to minimize P(y|S a ,S b ) and Q(y|S a ,S b ), P(y|S a ,S b ) and Q(y|S a ,S b ) represent keyword matching and intent matching respectively, and assume that they are independent of each other.

Images