Text semantic matching length deviation optimization method, electronic device and storage medium

Abstract

The application discloses a text semantic matching length deviation optimization method, an electronic device and a storage medium, and focuses on solving the length difference deviation problem existing in a text semantic matching task model. In the application, after the main model and the opponent model are jointly trained, the text matching model obtained will be difficult to predict the text length difference value according to the given sample, that is, the text matching model does not need to pay attention to the representation of the text length difference value in the semantic space, so that the text length difference is not excessively relied on when making a prediction on a normal text semantic matching task, thereby reducing the influence of the text length difference deviation on the model generalization.

Description

TECHNICAL FIELD

[0001] The present application relates to text matching technology, in particular to a text semantic matching length bias optimization method based on natural language processing, an electronic device and a storage medium. BACKGROUND

[0002] Text matching technology has very important significance in various fields such as industrial production and life practice. Text matching refers to the process of checking whether there is a certain relationship between the specific order or structure of words in the text. The matching result is usually accurate: matching or not matching.

[0003] Text matching includes many sub-tasks, among which similarity measurement is considered an important part of various tasks such as text summarization, text matching, information retrieval, automatic paper scoring, document clustering and machine translation, which measures the similarity between words, sentences, paragraphs and documents.

[0004] Recent studies have shown that deep learning models may achieve false high performance by learning some heuristic patterns or rules in the data set, and even produce a series of social problems. The bias existing in the model and how to optimize the related problems are the problems to be solved at present.

[0005] Invention patent application CN114281931A introduces a method for optimizing a text matching model according to the distance between two texts in the vector space in text matching, but this method does not consider the length difference bias in text matching, which has a greater impact on the generalization of the model, resulting in limited model accuracy.

[0006] The so-called length difference bias refers to that given two texts, the model tends to consider that the text pair with smaller length difference is similar, and the text pair with larger length difference is not similar, and the reason for this situation is that the proportion of positive samples in samples with smaller length difference in the training data set is larger, and the proportion of negative samples in samples with larger length difference is larger. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a text semantic matching length bias optimization method, an electronic device and a storage medium to reduce the impact of text matching length bias on the generalization of the model in view of the deficiencies of the prior art.

[0008] To solve the above technical problems, the technical solution adopted by the present application is: a text semantic matching length bias optimization method, comprising the following steps:

[0009] S1, generating an adversarial test set using the length difference bias of each sample text pair;

[0010] Train a text matching similarity prediction model using the original training set;

[0011] S2, train a text length difference predictor using the word vector output by the text matching similarity prediction model, the text length difference predictor comprising an encoder and a fully connected layer connected to the encoder;

[0012] S3, the text matching similarity prediction model is used as the main model, and the text length difference predictor trained in step S2 is used as the opponent model; the loss function of the main model is added to the loss function of the opponent model, and the main model is optimized through the confrontation between the main model and the opponent model to obtain a first optimized model; or, a disturbance is added to the loss function, and the main model is optimized through the confrontation between the main model and the opponent model to obtain a second optimized model;

[0013] S4, repeat steps S2 and S3, and test the first optimized model and the second optimized model using the confrontation test set;

[0014] S5, select the model with better performance from the first optimized model and the second optimized model as the final text matching model.

[0015] The present application focuses on solving the problem of length difference bias in the text semantic matching task model. After the main model is trained through confrontation with the opponent model, the text matching model (main model) will be difficult to predict the text length difference value according to the given sample, i.e. the text matching model does not need to pay attention to the representation of the text length difference value in the semantic space, so it will not excessively rely on the text length difference when making predictions for normal text semantic matching tasks, thereby reducing the influence of text length difference bias on model generalization.

[0016] If the model is affected by length difference bias, i.e. tends to predict that samples with low LD values are positive samples and samples with high LD values are negative samples, the confrontation test set generated by the above method will cause great trouble to the model, because the samples with low LD values are all composed of negative samples, and the samples with high LD values are all composed of positive samples. In order to solve the above problem and improve the model precision and generalization performance, the specific implementation process of generating the confrontation test set in step S1 includes:

[0017] Calculate the length difference value of each sample S in the original test set, sort all the length difference values in descending order to obtain the sorted test set D t ′ est ;

[0018] The test set D t ′ estThe data in the original training set is divided into two parts, the front part is a data set D f ′ ront The back part is a data set D b ′ ack All samples in the data set D f ′ ront All samples in the data set D b ′ ack The union of the negative sample set and the positive sample set is the adversarial test set.

[0019] The length difference value of the sample S is calculated according to the following formula: Wherein, The length of the two texts T1 and T2 in the sample S is respectively, The length difference value of the two texts T1 and T2 in the sample S is.

[0020] In step S1, the specific implementation process of training the text matching similarity prediction model using the original training set includes:

[0021] 1) Using the word table provided by the BERT pre-training model, the two texts of the sample S* in the original training set are segmented, and the segmented results T1 ′ and T2 ′ are added to the token of the BERT pre-training model to obtain the input input={ [CLS], T'1, [SEP], T'2, [SEP]}, [CLS] is placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] is placed at the end of the sentence as the end of the sentence.

[0022] 2) Input the text matching model to obtain the similarity probability P of the two texts of the sample S* in the original training set; calculate the loss of the probability P and the true label of the sample S*, and update the weight of the text matching model by back propagation;

[0023] 3) Repeat steps 1) and 2) for all samples in the original training set, and verify the text matching model with updated weights using the adversarial test set to obtain the final text matching similarity prediction model. The specific implementation process of step S2 includes:

[0024] A) Calculate the length difference value of each sample in the original training set, divide the value range of all length difference values into M intervals, and assign an integer in [0, M-1] to each length difference value as its label;

[0025] B) using the word table provided by the BERT pre-training model, tokenizing the two texts of the sample S* in the original training set, and obtaining the tokenized results T1 ′ and T2 ′ into the token of the BERT pre-training model, obtaining input = {[CLS], T'1, [SEP], T'2, [SEP]}, [CLS] is placed at the beginning of the sentence as the sentence vector of the entire sentence, and [SEP] is placed at the end of the sentence as the end of the sentence;

[0026] C) input is input into the encoder layer M enc of the text matching similarity prediction model, obtaining the sentence vector representation h of the sample S*;

[0027] D) the sentence vector representation h is input into the full connection layer, obtaining the probability C of the sentence vector representation h corresponding to the sample S* belonging to each label in [0, M-1], that is, C is a vector with a length of M, and the i-th value in C corresponds to the probability of the length difference value belonging to label i-1, i∈[0, M-1];

[0028] E) calculating the loss of the probability C and the real label of the sample S*, and updating the weights of the full connection layer by backpropagating the loss;

[0029] F) repeating steps D) and E) for all samples in the original training set to obtain the text length difference predictor.

[0030] The specific implementation process of step S2 includes:

[0031] A) calculating the length difference value of each sample in the original training set, dividing the value range of all length difference values into M intervals, and assigning an integer in [0, M-1] to each length difference value as its label;

[0032] B) using the word table provided by the BERT pre-training model, tokenizing the two texts of the sample S* in the original training set, and obtaining the tokenized results T1 ′ and T2 ′ into the token of the BERT pre-training model, obtaining input = {[CLS], T'1, [SEP], T'2, [SEP]}, [CLS] is placed at the beginning of the sentence as the sentence vector of the entire sentence, and [SEP] is placed at the end of the sentence as the end of the sentence;

[0033] C) input is input into the encoder layer M enc of the text matching similarity prediction model, obtaining the sentence vector representation h of the sample S*;

[0034] D) taking the sentence vector representation h as an input of a full connection layer to obtain a predicted value V of the length difference value of the sample S* corresponding to the sentence vector representation h;

[0035] E) calculating a loss of the predicted value V and a true label of the sample S*, back propagating the loss to update weights of the full connection layer;

[0036] F) repeating the above steps D) and E) for all samples in the original training set to obtain a text length difference predictor.

[0037] In step S3, the specific implementation process of obtaining the first optimized model comprises:

[0038] the length difference value of the i-th sample S i in the original training set is modified as: M is the number of intervals of the value range division of all length difference values;

[0039] the loss function Loss main of the full connection layer in the main model is added to the loss function Loss bias of the counterpart model, and a balance parameter β is introduced to construct a final loss function Loss as: Loss=βLoss main +(1-β)Loss bias .

[0040] The main model is trained by using the final loss function Loss to obtain the first optimized model.

[0041] In step S3, the specific implementation process of obtaining the second optimized model comprises:

[0042] The two texts of the sample S* in the original training set are segmented by using a word table provided by the BERT pre-training model, and the segmented results T'1 and T'2 are added to the token of the BERT pre-training model to obtain an input input={[CLS],T′1,[SEP],T′2,[SEP]},[CLS] is placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] is placed at the end of the sentence as the end of the sentence.

[0043] The input input corresponding to the sample S* is input into the counterpart model through the encoder of the main model to obtain a predicted value, and a loss of the predicted value and a true label of the sample S* is calculated;

[0044] The gradient is obtained by taking a partial derivative of the loss with respect to the output h of the encoder of the main model. bias is the loss of V and LD calculated by using a loss function, partial derivative with respect to h;

[0045] The input input corresponding to the sample S* is input into the encoder of the main model again, a sentence vector is obtained, the sentence vector is modified as h+delta, and the modified sentence vector is input into the full connection layer of the main model to calculate a text matching probability; is used to control the disturbance intensity, and ‖·‖ is the Frobenius norm;

[0046] The loss of the text matching probability and the true label of the sample S* is calculated, the weight of the main model is updated after back propagation, and a second optimization model is obtained.

[0047] As an inventive concept, the present application also provides an electronic device, comprising:

[0048] One or more processors;

[0049] A memory having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the steps of the above-mentioned method of the present application.

[0050] As an inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned method of the present application.

[0051] Compared with the prior art, the present application has the beneficial effects that:

[0052] 1. The present application proposes a method for making an adversarial test set for length difference bias, which can help to discover the influence of implicit length difference bias on the model;

[0053] 2. The present application designs two length difference predictors, which can map the sentence vector existing in the semantic space to the length difference space, so as to take optimization measures for the length difference bias;

[0054] 3. The present application combines adversarial samples and generative adversarial networks to design an optimization scheme for the model bias of the two length difference predictors respectively, which can improve the robustness and generalization of the model at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 It is a multi-classification length difference prediction and optimization model of the embodiment of the present application;

[0056] Figure 2 It is a linear regression length difference prediction and optimization model of the embodiment of the present application; DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] Embodiment 1

[0059] The embodiment 1 of the present application provides a text matching length deviation optimization technology based on natural language processing, which comprises the following steps:

[0060] Given an original text matching data training set and test set with an implicit length difference deviation, each sample contains a pair of texts, and a text semantic matching task needs to predict whether each sample text pair is similar in semantics.

[0061] S1. According to the length difference deviation of each sample text pair, an algorithm is designed to generate an adversarial test set;

[0062] S2. A text matching similarity prediction model with length difference deviation is trained on the original training data set with implicit length difference deviation, and the performance of the model on the adversarial test set generated in S1 can be tested to study the influence of the length difference deviation;

[0063] S3. According to the model trained in S2, fix its weight, and only use the output word vector to train a text length difference predictor composed of full connection layers. Since the text length difference value domain is continuous and infinite, the predictor can adopt a multi-classification model with artificial division of intervals and label assignment for label prediction, or a linear regression model for direct value prediction;

[0064] S4. Fix the weight of the predictor trained in step S3, and according to the type of predictor selected in step S3, adjust the loss function of the multi-classification model and combine it into the loss function of the text matching model for joint training, and the linear regression model uses the FGSM (Fast Gradient Sign Method) method to generate adversarial samples for adversarial training;

[0065] S5. Repeat steps S3 and S4 until the performance of the text matching model in the adversarial test set in S1 reaches a relatively satisfactory effect. At this time, according to the two different text length difference predictors in S3 and their corresponding training processes, two text matching models can be obtained in the embodiment of the present application, and the one with better comprehensive performance is selected as the final text matching model without text length difference deviation in the embodiment of the present application.

[0066] The specific implementation process of step S1 includes:

[0067] The embodiment of the application takes the calculation of the similarity of text matching Chinese text as an example, and when it is considered that the given two texts are similar, the embodiment of the application marks it as 1, and vice versa, that is, 0. The sample with a label of 1 is also called a positive sample, and the sample with a label of 0 is also called a negative sample. In order to study the length difference bias of the text, and also to provide a test data set for the effect of the optimized model, the embodiment of the application first needs to generate an adversarial test data set D on the basis of the original test data set adv The embodiment of the application agrees that the calculation method of the length difference of a sample Chinese text pair is as follows:

[0068] 1. For a sample S, it contains two texts {T1, T2};

[0069] 2. For a text T, its length L T is the number of words contained in the text T;

[0070] 3. Then the length difference of the two texts {T1, T2} in the sample S is The calculation method is as follows:

[0071]

[0072] Where min(·) is a function of finding the minimum of two, and |·| is the absolute value symbol. It can be known that the value range of LD is [0, ∞], and the greater the value, the greater the difference in the number of words between the text pairs in the sample.

[0073] With the method of quantifying the length difference of the text pair, the generation steps of the adversarial test set are as follows:

[0074] 1. For the original test set D test , it contains {S1, S2, …, S n} a total of n samples;

[0075] 2. Calculate the LD value of each sample S according to the above formula and sort it from small to large to get the sorted test set D' test ;

[0076] 3. Set the first quarter of the data set in D' test as D' front , the last quarter of the data set in D' test as D' back , the set Neg as all negative samples in D' front , N = {S i | i ∈D' front and S i is a negative sample}, and the set Pos as D'back all positive samples in D, Pos = {S i | S i ∈ D back and S i is a positive sample}.

[0077] 4、The union of Pos and Neg, D adv = Pos U Neg, is the generated adversarial test set.

[0078] The reason for this is that if the model is affected by length difference bias, that is, it tends to predict that samples with low LD values are positive samples and samples with high LD values are negative samples, then the adversarial test set generated using the above method will cause great trouble to the model, because the samples with low LD values are all composed of negative samples, and the samples with high LD values are all composed of positive samples.

[0079] The specific implementation process of step S2 is shown in Figure 1 , which includes:

[0080] In this step, the embodiment of the present application normally trains a text matching model on the original training set as the model to be optimized by the embodiment of the present application for the implicit length difference bias. The model generally consists of an Encoder layer and a fully connected layer, denoted as M enc and M main . Here, the embodiment of the present application takes the BERT model as an example to illustrate the complete process of training a text matching model:

[0081] 1. Given a sample S, containing two texts {T1, T2};

[0082] 2. Use the word table (vocab.txt) provided by the BERT pre-training model to tokenize {T1, T2} into corresponding tokens, i.e. T1 = {w 11 , w 12 , …, w 1n} into T'1 = {t 11 , t 12 , …, t 1n}, and T2 is transformed in the same way into T'2, and T'1 and T'2 are combined and added to the token specific to BERT to form the final input form input = {[CLS], T'1, [SEP], T'2, [SEP]}, [CLS] is placed at the beginning of the sentence and the vector after training can be used as the sentence vector of the entire sentence, and [SEP] is placed at the end of the sentence as the end of the sentence;

[0083] 3. Input into the text matching model to obtain the probability P that {T1, T2} is similar;

[0084] 4. Use the CrossEntropyLoss function to calculate the loss between probability P and the true label Y of sample S, and then update the weights of the text matching model after backpropagation.

[0085] 5. For all training sets D train Sample S in i ∈D train Repeat steps 2-4 above, performing them on the test set D in each iteration. test Test the performance once until the model reaches a relatively good performance, which is the final model obtained in this step.

[0086] In this embodiment of the invention, the model can also be run on the adversarial test set obtained in step S1 to check the effect. If a significant performance drop is found, it indicates that the model is indeed affected by the length difference deviation.

[0087] The specific implementation process of step S3 includes:

[0088] To reduce the impact of length difference bias on the model, this embodiment of the invention requires knowledge of how the length difference is encoded in the model, utilizing the encoder layer M of the original biased text matching model obtained in step S2. enc In this embodiment of the invention, an additional fully connected layer M can be trained. bias To predict the value or label of the length difference, there are two ways to achieve this. The first is a multi-class classification implementation, such as... Figure 1 As shown, the specific steps are as follows:

[0089] 1. Calculate the training dataset D train Each sample S i ∈D train LD i The value is divided into M parts, where the value of M is determined according to the specific situation, and is assigned to each S. i LD i The integer in the value assignment [0, M-1] is used as its label L i ;

[0090] 2. Given the text of sample S, and performing the same processing as in step 2 of step S2 on S, obtain the input that the model can recognize. Input the input into the model's M. enc After partial processing, embodiments of the present invention can obtain the sentence vector representation h of sample S;

[0091] 3. Use the sentence vector h as M bias Given the input, predict the probability C of sample S corresponding to h belonging to each label in [0, M-1] with respect to LD;

[0092] 4. Fixed model Menc For some weights, the CrossEntropyLoss function is used to calculate the loss on probability C and the true label L of sample S, and then M is updated after backpropagation. bias The weights;

[0093] 5. For all training sets D train Sample S in i ∈D train Repeat steps 3 and 4 above until the model achieves a relatively good performance, which is one of the final models obtained in this step.

[0094] Considering the LD i Manually dividing the value range incurs certain costs. Alternatively, linear regression can be used to directly predict the values ​​of length differences, such as... Figure 2 As shown:

[0095] 1. Calculate the training dataset D train Each sample S i ∈D train LD i value;

[0096] 2. Given the text of sample S, perform the same processing as in step S2.2 on S to obtain the input that the model can recognize, and input the input into the model's M. enc In some embodiments of the present invention, the sentence vector representation h of sample S can be obtained;

[0097] 3. Use the sentence vector h as M bias The input is used to obtain the predicted value V of sample S with respect to LD for the corresponding h.

[0098] 4. Fixed model M enc The partial weights are used to calculate the loss on the true LD values ​​of V and sample S using the MeanSquaredErrorLoss function, and then updated after backpropagation. bias The weights;

[0099] 5. For all training sets D train Sample S in i ∈D train Repeat steps 3 and 4 above until the model performance no longer improves significantly; this is one of the final models obtained in this step.

[0100] The main purpose of this step is to study how the sentence vectors of the samples are mapped from the semantic space to the length difference space, and to obtain M through the two methods mentioned above. bias This is equivalent to a mapping method, and the embodiments of the present invention can continue to conduct research on optimizing length difference deviations based on this.

[0101] The specific implementation process of step S4 includes:

[0102] This step is the core of the invention and a crucial step in resolving length difference deviations. This embodiment utilizes the tool M obtained in step S3, which maps sentence vectors from the semantic space to the length difference space. bias To achieve this, due to M bias There are two implementations: multi-class classification and linear regression. The subsequent processing methods differ between these two implementations, but the core idea is the same: to use a training method similar to GANs (Generative Adversarial Networks) to create an adversary model. This allows the main model to gain robustness against length differences in adversarial comparisons with the adversary model. Here, the adversary model is M. bias The main model is M enc +M main Meanwhile, M bias It can also share M enc The intermediate vector output by the layer.

[0103] M bias The implementation method is as follows: subsequent steps of a multi-class classification model Figure 1 As shown:

[0104] 1. Given a sample S containing two texts {T1, T2}, the vector form of the label L corresponding to its LD value is a one-hot vector L = {0, 0, ..., 1, ..., 0};

[0105] 2. All samples S i ∈D train LD i Tag L i , by L i ={0,0,…,1,…,0} changed to M is the number of labels manually defined in step S3;

[0106] 3. Place M main The corresponding loss function Loss main With M bias The corresponding loss function Loss bias Adding them together and introducing a balancing parameter β, the final loss function takes the form of:

[0107] Loss = βLoss main +(1-β)Loss bias ;

[0108] 4. Fix M bias The weights of the layers are adjusted using a new loss function, Loss, on M. enc and M mainConduct training.

[0109] Because M bias It is a well-trained mapping tool from semantic space to length difference space. In step 2 of S4 of this invention, the sample labels are changed from one-hot to a uniform distribution. The loss function of the adversary model is added to the loss function of the main model. Thus, during the training process of the main model, it is necessary not only to continuously optimize on the text semantic matching task, but also to optimize the sentence vector so that it can pass through M bias After mapping, we obtain a result whose probability is as close as possible to the average distribution, and fix M. bias The layer weights are used to maintain the effectiveness of the mapping, while the balancing parameter β is used to control the optimization intensity.

[0110] M bias The implementation method is as follows: subsequent steps of the linear regression model are as follows Figure 2 As shown:

[0111] 1. Fix M bias The layer weights are applied to the input corresponding to sample S through M. enc Layer input opponent model M bias The predicted value V is obtained, and the MeanSquaredErrorLoss function is used to calculate the loss Loss between V and the true LD value of sample S. bias ;

[0112] 2. Use Loss bias For M enc Take the partial derivative of the output h to obtain its gradient.

[0113] 3. Create disturbances ∈ is used to control the perturbation intensity, and ||·|| is the Frobenius norm;

[0114] 4. Input the sample S into M again. enc The layer obtains the sentence vector h, modifies h to h′=h+δ, and then inputs it into M. main And calculate the text matching probability P;

[0115] 5. Use the CrossEntropyLoss function to calculate the loss between the probability P and the true label Y of the sample S, and then backpropagate to update the weights of the text matching model.

[0116] Steps 2 through 4 above constitute FGSM (Fast Gradient Sign Method), a method for generating adversarial examples. It emphasizes adding parameters to the input h to reduce the loss. bias The fastest rising perturbation is used to perplex the model, and the direction of the h gradient is the loss.bias The direction with the fastest change speed. By generating the adversarial sample of h, the embodiment of the present application can make the prediction boundary of the LD value of the main model become more blurred, thereby achieving the purpose of improving the robustness of the model to the length difference bias.

[0117] The specific implementation process of step S5 includes:

[0118] Considering that there are more than one encoding method of the model to the length difference, the embodiment of the present application considers repeating steps S3 and S4, and for the i-th step S3, the embodiment of the present application re-trains a and adds it to the candidate adversary model pool Pool adv In step S4, re-train the main model every step, and alternately select one from Pool adv As an adversary model, for the two implementations of the multi-classification model and the linear regression model, the embodiment of the present application can be independently trained, and the specific process is as follows:

[0119] 1. When entering a new step S3, assume that this time is the i-th time to enter step S3, re-train a length difference predictor and add it to the adversary candidate pool Pool adv ;

[0120] 2. When training the main model in step S4 enters a new step (corresponding to a batch), assume that this time is the j-th step, the embodiment of the present application selects the j'-th adversary model from the candidate pool Pool adv adv .

[0121] 3. Repeat the above 1 and 2 until the performance of the model M main on the adversarial test set D adv generated in step S1 of the embodiment of the present application reaches a relatively satisfactory effect or the entire system reaches the upper limit of the training number set by the human.

[0122] 4. Select the one with better comprehensive performance from the two predictor implementation methods of the multi-classification model and the linear regression model as the final optimized text matching model.

[0123] ​​The experiment is performed on a notebook computer containing an Intel(R) Core(TM) i7-10870H processor, 16G memory, and an NVIDA 3060 RTX graphics card. The experimental data sets are two public data sets of Quora Question Pairs (QQP) and Twitter-URL, wherein the QQP contains 363846 training data and 14885 test data, and the Twitter-URL contains 42200 training data and 9334 test data. Taking the QQP data set as an example, before the method of the embodiment of the application is used, the precision of the original model on the test set is 85.77% in accuracy and 81.15% in F1 score, and the precision of the original model on the adversarial test set generated by the method introduced in the embodiment of the application is 83.05% in accuracy and 72.56% in F1 score. After the method of the embodiment of the application is used, the precision of the optimized model on the test set is 86.72% in accuracy and 81.99% in F1 score, and the precision of the optimized model on the adversarial test set is 85.14% in accuracy and 75.10% in F1 score.

[0124] After comparing and analyzing the results, it can be seen that the method for generating the adversarial test set introduced in the embodiment of the application can indeed effectively help to uncover the influence of the length difference bias implicitly contained in the model, which is specifically manifested as the decrease in the precision of the original model on the adversarial test set; secondly, the optimization technique introduced in the embodiment of the application can indeed help the model to improve the influence of the length difference bias, not only greatly improving the precision on the adversarial test set and enhancing the robustness of the model, but also improving the precision on the original test set and enhancing the generalization of the model.

[0125] Embodiment 2

[0126] The embodiment 2 of the application provides an electronic device corresponding to the above-mentioned embodiment 1, and the terminal device can be an electronic device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0127] The electronic device of the embodiment includes a memory, a processor, and a computer program stored on the memory; the processor executes the computer program on the memory to realize the steps of the method of the above-mentioned embodiment 1.

[0128] In some implementations, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0129] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP), or various types of general-purpose processors, without limitation.

[0130] Embodiment 3

[0131] Embodiment 3 of the present application provides a computer readable storage medium corresponding to the above-mentioned embodiment 1, which has stored thereon computer programs / instructions. The computer programs / instructions are executed by a processor to implement the steps of the method of embodiment 1.

[0132] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0133] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0134] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to produce a computer-implemented process such that the instructions executed by the computer or other programmable data processing apparatus provide the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 steps for performing the functions specified in the flowchart

[0136] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0137] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for optimizing length bias of text semantic matching, characterized in that, The method comprises the following steps: S1, generating an adversarial test set by using the length difference deviation of each sample Chinese text pair; training a text matching similarity prediction model by using the original training set; S2, training a text length difference predictor by using the word vector output by the text matching similarity prediction model, the text length difference predictor comprising an encoder and a fully connected layer connected to the encoder; S3, taking the text matching similarity prediction model as a main model, taking the text length difference predictor trained in step S2 as an opponent model, adding the loss function of the opponent model to the loss function of the main model, optimizing the main model through the confrontation between the main model and the opponent model to obtain a first optimized model, or adding a disturbance to the loss function of the main model, optimizing the main model through the confrontation between the main model and the opponent model to obtain a second optimized model; S4, repeating step S2 and step S3, and testing the first optimized model and the second optimized model by using the adversarial test set; S5, selecting a model with better performance from the first optimized model and the second optimized model as a final text matching model; The specific implementation process of step S2 comprises: A) calculating the length difference value of each sample in the original training set, dividing the value range of all length difference values into M intervals, and assigning an integer in [0, M-1] to each length difference value as its label; B) using the word table provided by the BERT pre-training model, tokenizing the two texts of the sample S* in the original training set, adding the tokenized results T1' and T2' to the token of the BERT pre-training model, obtaining input = {[CLS], T'1, [SEP], T'2, [SEP]}, [CLS] being placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] being placed at the end of the sentence as the end of the sentence; C) input as text matching similarity prediction model encoder layer M enc obtaining a sentence vector representation h of the sample S*; D) taking the sentence vector representation h as the input of the fully connected layer to obtain the probability C of the sentence vector representation h belonging to each label in [0, M-1] with respect to the length difference value of the sample S*; E) calculating the loss of the probability C and the true label of the sample S*, back propagating the loss, and updating the weights of the fully connected layer; F) repeating the above steps D) and E) for all samples in the original training set to obtain the text length difference predictor; Or, The specific implementation process of step S2 comprises: A) calculating the length difference value of each sample in the original training set, dividing the value range of all length difference values into M intervals, and assigning an integer in [0, M-1] to each length difference value as its label; B) using the word table provided by the BERT pre-training model, tokenizing the two texts of the sample S* in the original training set, adding the tokenized results T1' and T2' to the token of the BERT pre-training model, obtaining input = {[CLS], T'1, [SEP], T'2, [SEP]}, [CLS] being placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] being placed at the end of the sentence as the end of the sentence; C) input as text matching similarity prediction model encoder layer M enc obtain sentence vector representation h of sample S*; D) taking the sentence vector representation h as input of a fully connected layer, obtaining a predicted value V of the length difference value of the sample S* corresponding to the sentence vector representation h; E) calculating a loss of the predicted value V and the true label of the sample S*, back-propagating the loss, and updating the weights of the fully connected layer; F) repeating the step D) and the step E) for all samples in the original training set, obtaining the text length difference predictor.

2. The method of claim 1, wherein, The specific implementation process of generating the adversarial test set by using the length difference bias of the Chinese text pair of each sample in the step S1 includes: Calculate the length difference value of each sample S in the original test set, sort all the length difference values in descending order to obtain the sorted test set D' test ; The data in the test set D' test is divided into two parts, the former part is D' front , the latter part is D' back , all samples in D' front are set as a negative sample set, all samples in D' back are set as a positive sample set, and the union of the negative sample set and the positive sample set is an adversarial test set.

3. The method of claim 2, wherein, The calculation formula of the length difference value of the sample S is: Wherein, L T1 , L T2 are the lengths of the two texts T1 and T2 in the sample S, is the length difference value of the two texts T1 and T2 in the sample S.

4. The method of claim 1, wherein, The specific implementation process of training the text matching similarity prediction model by using the original training set in the step S1 includes: 1) using the word table provided by the BERT pre-training model to tokenize the two texts of the sample S* in the original training set, and adding the tokenized results T1' and T2' to the token of the BERT pre-training model to obtain the input input={[CLS], T′1, [SEP], T′2, [SEP]}, [CLS] is placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] is placed at the end of the sentence as the end of the sentence; 2) inputting the input into the text matching model to obtain the probability P of the similarity of the two texts of the sample S* in the original training set; calculating a loss of the predicted value V and the true label of the sample S*, back-propagating the loss, and updating the weights of the fully connected layer; 3) repeating the step 1) and the step 2) for all samples in the original training set, and verifying the text matching model after updating the weights by using the adversarial test set, to obtain the final text matching similarity prediction model.

5. The method of claim 1, wherein, The specific implementation process of obtaining the first optimization model in the step S3 includes: The label of the length difference value of the i-th sample S i in the original training set is modified as: M is the number of intervals of the range division of all length difference values; The loss function Loss corresponding to the full connection layer in the main model is added to the loss function Loss corresponding to the opponent model main The loss function Loss corresponding to the full connection layer in the main model is added to the loss function Loss corresponding to the opponent model bias And introduce the balance parameter β, the final loss function Loss is constructed as: Loss = βLoss main +(1-β)Loss bias ; training the main model by using the final loss function Loss to obtain the first optimization model.

6. The method of claim 1, wherein, The specific implementation process of obtaining the second optimization model in the step S3 includes: using the word table provided by the BERT pre-training model to tokenize the two texts of the sample S* in the original training set, and adding the tokenized results T1' and T2' to the token of the BERT pre-training model to obtain the input input={[CLS], T′1, [SEP], T′2, [SEP]}, [CLS] is placed at the beginning of the sentence as the sentence vector of the whole sentence, and [SEP] is placed at the end of the sentence as the end of the sentence; inputting the input corresponding to the sample S* into the encoder of the main model to obtain a predicted value, and calculating a loss of the predicted value and the true label of the sample S*; Taking partial derivative of the loss with respect to an output h of an encoder of the main model, a gradient is obtained LD is a length difference value of the sample S*, V is a prediction value of LD, Loss bias is a loss calculated by a loss function for V and LD, is a partial derivative with respect to h; Again, input the input corresponding to the sample S* into the encoder of the main model to obtain a sentence vector, modify the sentence vector as h+δ, and input the modified sentence vector into the full connection layer of the main model to calculate the text matching probability; ∈ is used to control the disturbance intensity, and ‖·‖ is the Frobenius norm. calculating a loss of the predicted value V and the true label of the sample S*, back-propagating the loss, and updating the weights of the fully connected layer; 7. An electronic device, comprising: comprises: one or more processors; a memory having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the steps of the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1-6.

Images