A text matching method based on nearest neighbor enhanced contrastive learning pre-training

Abstract

The present invention discloses a text matching method using nearest neighbor enhanced contrastive learning pre-training, comprising: pre-training a source text set and a target text set using contrastive learning to obtain new source text vectors and target text vectors, respectively; retrieving similar instances of the source text vector and the target text vector using a nearest neighbor algorithm to obtain a source similar instance set and a target similar instance set, respectively; calculating a similarity weight set of the source similar instance set and a similarity weight set of the target similar instance set; merging the source text vector and the source similar instance set based on the similarity weight set of the source similar instance set to obtain a source text weighted vector; merging the target text vector and the target similar instance set based on the similarity weight set of the target similar instance set to obtain a target text weighted vector; and training the source text weighted vector and the target text weighted vector using a text matching model to obtain a matching result. The method mines similar instance information of the source and target texts, better constructs a text vector layer, and improves matching accuracy.

Description

Technical Field

[0001] The present invention relates to a text matching method based on nearest neighbor enhanced contrast learning pre-training, and belongs to the technical field of natural language processing. Background Art

[0002] Text matching is a fundamental and crucial task in natural language processing. It aims to learn the semantic relationship between source and target text and is used in many application scenarios, such as information retrieval systems, intelligent recommendation systems, and text data deduplication. The meaning of text matching varies depending on the context. In text data deduplication, the system can measure the similarity between two texts by calculating the semantic match between them. In question-answering systems, text matching can also be used to match appropriate answers to questions, reducing the need for human customer service. Therefore, research on text matching is of great significance. Traditional text matching models, such as BM25 and TF-IDF, primarily address lexical-level matching. These models can lead to semantic or knowledge gaps in the text. To address this, deep learning-based text matching models can learn interaction patterns between texts and uncover complex semantic relationships between them, and have garnered extensive research and attention.

[0003] Deep text matching models can be divided into two categories: representation-based and interaction-based. Representation-based models learn representations for the source and target texts and measure the correlation between them based on the learned representations. They focus more on constructing a semantic vector layer. Typical models include DSSM-RNN, BiRNN, GRU, BiGRU, LSTM, and ESIM. Interaction-based models construct a matching matrix between the source and target texts and estimate the matching score by analyzing the matching matrix. Typical models include AlBERT, BERT, and ROBERTA.

[0004] To improve the matching performance of the above-mentioned deep models, text pre-training (e.g., Word2Vec, GloVe, and BERT) plays a vital role. Pre-training models such as Word2Vec, GloVe, and BERT proposed in recent years have effectively solved the problem of text vectorization representation. Among them, the BERT-based pre-training language model has achieved high performance in many aspects, because the BERT model is used to vectorize the text and solve the polysemy problem of Word2Vec. However, the native text representation obtained from BERT has been shown to be of low quality, and the similarity scores of almost all text pairs are between 0.6 and 1.0, which means that almost all text is mapped to a small area. Therefore, it is inappropriate to directly apply BERT's original sentence representation method for semantic matching or text retrieval.

[0005] To this end, a contrastive learning-based text training objective is generated, which brings two augmented views from the same text closer and keeps views from other texts farther away, reshaping the BERT-derived text representation space. Summary of the Invention

[0006] (1) Purpose: Existing text matching models ignore the similar instance information directly obtained from the training examples. Utilizing this information can help the model make more accurate predictions.

[0007] To address these issues, this paper proposes a text matching method using nearest neighbor enhanced contrastive learning pre-training. Incorporating the concept of nearest neighbors, this method retrieves similar instances between the source and target text vectors after contrastive learning and constructs a weighted vector based on similarity weights. This allows the model to fully exploit similar instance information between the source and target texts, thereby improving text matching accuracy.

[0008] (2) Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0009] In a first aspect, the present invention provides a text matching method using nearest neighbor enhanced contrastive learning pre-training, comprising:

[0010] Establish source text set and target text set;

[0011] Comparative learning of the pre-trained source text set and target text set to obtain new source text vectors and target text vectors respectively;

[0012] The nearest neighbor algorithm is used to retrieve similar instances of the source text vector and the target text vector, and the source similar instance set and the target similar instance set are obtained respectively;

[0013] Calculate the similarity weight set of the source similar instance set and the similarity weight set of the target similar instance set;

[0014] Based on the similarity weight set of the source similar instance set, the source text vector and the source similar instance set are combined to obtain a source text weighted vector;

[0015] Based on the similarity weight set of the target similar instance set, the target text vector and the target similar instance set are combined to obtain the target text weighted vector;

[0016] The text matching model is used to train the source text weighted vector and the target text weighted vector to obtain the matching result.

[0017] In some embodiments, establishing a source text set and a target text set includes:

[0018] make represents the source text set, where p [i] is the i-th source text, 1≤i≤N; let represents the target text set, where q [i] is the i-th target text, 1≤i≤N.

[0019] Let {y1,…,y N} represents the label of the input data, where y i ∈{0,1} represents p [i] and q [i] The true value of the matching result between [i] and q [i] Match, then y i =1, if p [i] and q [i] If it does not match, then y i =0.

[0020] In some embodiments, comparative learning of a pre-trained source text set and a target text set to obtain new source text vectors and target text vectors, respectively, includes:

[0021] The contrastive learning model Simcse is used to pre-train the source text set and target text set, including:

[0022] Use the Dropout method to perform data augmentation on all texts, use the pre-trained language model BERT to encode the input text, and then use the contrastive learning objective to fine-tune all parameters to finally obtain the new text vector;

[0023] Source text set After contrastive learning, the new source text vector is obtained as Target text set After contrastive learning, the new target text vector is obtained as

[0024] In some embodiments, a nearest neighbor algorithm is used to retrieve similar instances of a source text vector and a target text vector to obtain a source similar instance set and a target similar instance set, respectively, including:

[0025] make is a vector containing all source text and the target text vector 2N text vector sets, express The j-th text vector of ;

[0026] Calculate the source text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as source similar instances.

[0027] Among them, the source text vector and The smaller the Euclidean distance of the Chinese text vector, the greater the similarity; the source text vector and The Euclidean distance value of As shown below:

[0028]

[0029] Where, yes The features of the mth dimension, yes The m-th dimension of the feature, L is the dimension of the text vector.

[0030] Calculate the target text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as target similar instances.

[0031] Among them, the target text vector and The smaller the Euclidean distance of the Chinese text vector, the greater the similarity; the target text vector and The Euclidean distance value of As shown below:

[0032]

[0033] Where, yes The features of the mth dimension, yes The features of the mth dimension.

[0034] Through nearest neighbor retrieval, K source similar instances and K target similar instances are obtained, which constitute the source similar instance set. and target similar instance set

[0035]

[0036] Where, is the source text vector The kth (1≤k≤K) source similarity instance, The target text vector The kth (1≤k≤K) target similarity instance.

[0037] In some embodiments, calculating the similarity weight set of the source similar instance set and the similarity weight set of the target similar instance set includes:

[0038] The inverse of the Euclidean distance between the source text vector and the source similar instance set is normalized as the similarity weight set of the source similar instance set, and the inverse of the Euclidean distance between the target text vector and the target similar instance set is normalized as the similarity weight set of the target similar instance set. The sigmoid activation function is used for normalization.

[0039] Furthermore, the inverse of the Euclidean distance between the source text vector and the source similar instance set is normalized to obtain the similarity weight set of the source similar instance set, specifically including:

[0040] Let w p,i ={w p,i,1 ,…,w p,i,K} represents the source similar instance set Similarity weight set, where w p,i,k Represents the source text vector Similar instance to the kth source The similarity weights, 1≤k≤K, are as follows:

[0041]

[0042] Furthermore, the inverse of the Euclidean distance between the target text vector and the target similar instance set is normalized as the similarity weight set of the target similar instance set, specifically including:

[0043] Let w q,i ={w q,i,1 ,…,w q,i,K} represents the target similar instance set Similarity weight set, where w q,i,k Represents the target text vector Similar instances to the kth target The similarity weights, 1≤k≤K, are as follows:

[0044]

[0045] In some embodiments, the source text vectors are merged based on the similarity weights of the source similar instance sets. Similar instance set to the source Get the source text weight vector include:

[0046]

[0047] where w p,i ={w p,i,1 ,…,w p,i,K} represents the source similar instance set Similarity weight set, w p,i,kRepresents the source text vector Similar instance to the kth source Similarity weight, 1≤k≤K.

[0048] In some embodiments, the target text vector is merged based on the similarity weight set of the target similar instance set. and target similar instance set Get the target text weighted vector include:

[0049]

[0050] where w q,i ={w q,i,1 ,…,w q,i,K} represents the target similar instance set Similarity weight set, w q,i,k Represents the target text vector Similar instances to the kth target Similarity weight, 1≤k≤K.

[0051] In some embodiments, a text matching model is used to train a source text weighted vector and a target text weighted vector to obtain a matching result, including:

[0052] Use text matching model to train source text weighted vector and the target text weighted vector Get the matching result y i ′, where y i ′ represents p [i] and q [i] The predicted value of the matching results.

[0053] Furthermore, if p [i] and q [i] Match, then y i ′=1, if p [i] and q [i] If it does not match, then y i ′=0.

[0054] In a second aspect, the present invention provides a text matching device for nearest neighbor enhanced contrastive learning pre-training, comprising a processor and a storage medium;

[0055] The storage medium is used to store instructions;

[0056] The processor is configured to operate according to the instructions to perform the method according to the first aspect.

[0057] In a third aspect, the present invention provides a device comprising:

[0058] Memory;

[0059] processor;

[0060] as well as

[0061] computer programs;

[0062] The computer program is stored in the memory and is configured to be executed by the processor to implement the method described in the first aspect above.

[0063] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0064] (3) Beneficial effects: The text matching method with nearest neighbor enhanced contrastive learning pre-training provided by the present invention has the following advantages: Compared with existing text matching methods, the method of the present invention uses contrastive learning and nearest neighbor algorithms to retrieve high-quality similar training instances, and adds similar information to model training, thereby improving the effect of text matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a method flow according to an embodiment of the present invention;

[0066] Figure 2 A schematic diagram of a method according to an embodiment of the present invention;

[0067] Figure 3 Schematic diagram showing the impact of different K values of the nearest neighbor mechanism on experimental results according to one embodiment of the present invention;

[0068] Figure 4 Schematic diagram of the impact of similarity weight on experimental results when the K value of the nearest neighbor mechanism is different according to one embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0070] In the description of the present invention, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0071] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the exemplary expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0072] Example 1

[0073] In a first aspect, this embodiment provides a text matching method using nearest neighbor enhanced contrastive learning pre-training, including:

[0074] Establish source text set and target text set;

[0075] Comparative learning of the pre-trained source text set and target text set to obtain new source text vectors and target text vectors respectively;

[0076] The nearest neighbor algorithm is used to retrieve similar instances of the source text vector and the target text vector, and the source similar instance set and the target similar instance set are obtained respectively;

[0077] Calculate the similarity weight set of the source similar instance set and the similarity weight set of the target similar instance set;

[0078] Based on the similarity weight set of the source similar instance set, the source text vector and the source similar instance set are combined to obtain a source text weighted vector;

[0079] Based on the similarity weight set of the target similar instance set, the target text vector and the target similar instance set are combined to obtain the target text weighted vector;

[0080] The text matching model is used to train the source text weighted vector and the target text weighted vector to obtain the matching result.

[0081] In some embodiments, establishing a source text set and a target text set includes:

[0082] make represents the source text set, where p [i] is the i-th source text, 1≤i≤N; let represents the target text set, where q [i] is the i-th target text, 1≤i≤N.

[0083] Let {y1,…,y N} represents the label of the input data, where y i ∈{0,1} represents p [i] and q [i]The true value of the matching result between [i] and q [i] Match, then y i =1, if p [i] and q [i] If it does not match, then y i =0.

[0084] In some embodiments, comparative learning of a pre-trained source text set and a target text set to obtain new source text vectors and target text vectors, respectively, includes:

[0085] The contrastive learning model Simcse is used to pre-train the source text set and target text set, including:

[0086] Use the Dropout method to perform data augmentation on all texts, use the pre-trained language model BERT to encode the input text, and then use the contrastive learning objective to fine-tune all parameters to finally obtain the new text vector;

[0087] Source text set After contrastive learning, the new source text vector is obtained as Target text set After contrastive learning, the new target text vector is obtained as

[0088] In some embodiments, a nearest neighbor algorithm is used to retrieve similar instances of a source text vector and a target text vector to obtain a source similar instance set and a target similar instance set, respectively, including:

[0089] make is a vector containing all source text and the target text vector 2N text vector sets, express The j-th text vector of ;

[0090] Calculate the source text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as source similar instances.

[0091] Among them, the source text vector and The smaller the Euclidean distance of the Chinese text vector, the greater the similarity; the source text vector and The Euclidean distance value of As shown below:

[0092]

[0093] Where, yes The features of the mth dimension, yes The m-th dimension of the feature, L is the dimension of the text vector.

[0094] Calculate the target text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as target similar instances.

[0095] Among them, the target text vector and The smaller the Euclidean distance of the Chinese text vector, the greater the similarity; the target text vector and The Euclidean distance value of As shown below:

[0096]

[0097] Where, yes The features of the mth dimension, yes The features of the mth dimension.

[0098] Through nearest neighbor retrieval, K source similar instances and K target similar instances are obtained, which constitute the source similar instance set. and target similar instance set

[0099]

[0100] Where, is the source text vector The kth (1≤k≤K) source similarity instance, The target text vector The kth (1≤k≤K) target similarity instance.

[0101] In some embodiments, calculating the similarity weight set of the source similar instance set and the similarity weight set of the target similar instance set includes:

[0102] The inverse of the Euclidean distance between the source text vector and the source similar instance set is normalized as the similarity weight set of the source similar instance set, and the inverse of the Euclidean distance between the target text vector and the target similar instance set is normalized as the similarity weight set of the target similar instance set. The sigmoid activation function is used for normalization.

[0103] Furthermore, the inverse of the Euclidean distance between the source text vector and the source similar instance set is normalized to obtain the similarity weight set of the source similar instance set, specifically including:

[0104] Let w p,i ={w p,i,1 ,…,w p,i,K} represents the source similar instance set Similarity weight set, where w p,i,k Represents the source text vector Similar instance to the kth source The similarity weights, 1≤k≤K, are as follows:

[0105]

[0106] Furthermore, the inverse of the Euclidean distance between the target text vector and the target similar instance set is normalized as the similarity weight set of the target similar instance set, specifically including:

[0107] Let w q,i ={w q,i,1 ,…,w q,i,K} represents the target similar instance set Similarity weight set, where w q,i,k Represents the target text vector Similar instances to the kth target The similarity weights, 1≤k≤K, are as follows:

[0108]

[0109] In some embodiments, the source text vectors are merged based on the similarity weights of the source similar instance sets. Similar instance set to the source Get the source text weight vector include:

[0110]

[0111] where w p,i ={w p,i,1 ,…,w p,i,K} represents the source similar instance set Similarity weight set, w p,i,k Represents the source text vector Similar instance to the kth source Similarity weight, 1≤k≤K.

[0112] In some embodiments, the target text vector is merged based on the similarity weight set of the target similar instance set. and target similar instance set Get the target text weighted vector include:

[0113]

[0114] where w q,i ={w q,i,1 ,…,w q,i,K} represents the target similar instance set Similarity weight set, w q,i,k Represents the target text vector Similar instances to the kth target Similarity weight, 1≤k≤K.

[0115] In some embodiments, a text matching model is used to train a source text weighted vector and a target text weighted vector to obtain a matching result, including:

[0116] Use text matching model to train source text weighted vector and the target text weighted vector Get the matching result y i ′, where y i ′ represents p [i] and q [i] The predicted value of the matching results.

[0117] Furthermore, if p [i] and q [i] Match, then y i ′=1, if p [i] and q [i] If it does not match, then y i ′=0.

[0118] The present invention proposes a text matching method based on nearest neighbor enhanced contrastive learning pre-training. The invention uses contrastive learning and nearest neighbor mechanism to obtain similar training examples of texts and adds similar information to model training, such as Figure 2 shown.

[0119] The specific implementation of the present invention is as follows:

[0120] To verify the prediction performance of the model, experiments were conducted on both the interactive text matching model and the representational text matching model. This experiment used the following benchmark models:

[0121] 1) Representation type: ESIM: The underlying bidirectional LSTM acts as a text encoder and obtains the interaction results between two texts from the interaction matrix between the two text words.

[0122] 2) Interactive: BERT: This utilizes the bidirectional encoder structure of the Transformer to resolve the polysemy problem. The BERT model is used for text matching, which is easy to operate and produces excellent results, making it suitable for industrial applications.

[0123] In order to enhance the comparison effect, the present invention also similarly adds RNN, BiRNN, BiLSTM, AlBERT, and BERT as comparative experiments.

[0124] Taking the Chinese Question Matching Corpus LCQMC as an example, this dataset is a collection of question matching data built based on a massive amount of Baidu questions.

[0125] Evaluation Metrics: The text matching task in this experiment falls under the binary classification category. We use Area Under the Curve (AUC) and Accuracy (ACC) as metrics to evaluate model performance. ACC refers to the ratio of the number of correctly classified examples to the total number of examples across all categories. AUC is the area under the receiver operating characteristic (ROC) curve, where the horizontal axis represents the false positive rate (FPR) and the vertical axis represents the true positive rate (TPR). The closer the ACC and AUC are to 1.0, the more reliable the detection method.

[0126] Simulation parameters: N = 200000, K = 10, L = 300. The specific implementation method is as follows Figure 1 As shown,

[0127] Step 1: Create source text set and target text set. For text matching task, let Represents the source text set. Represents the target text set. {y1,…,y 200000} is the label of the input data.

[0128] Step 2: Compare and learn the pre-trained source text set and the target text set to obtain new source text vectors. and the target text vector

[0129] The source text set and target text set are trained using the contrastive learning model Simcse to obtain new source text vectors and target text vectors respectively. After contrastive learning, the new source text vector is obtained as Target text set After contrastive learning, the new target text vector is obtained as

[0130] make is a vector containing all source text and the target text vector

[0131] Step 3: Retrieve the source text vector using the nearest neighbor algorithm and the target text vector Similar instances of the source similar instance set are obtained respectively and target similar instance set

[0132] This embodiment uses the nearest neighbor algorithm to Retrieve K source similar instances from . Among them, the source text vector and Chinese version The Euclidean distance value of As shown in formula (1):

[0133]

[0134] Where, yes The features of the mth dimension, yes The mth dimension of the feature, 300 is the dimension of the text vector.

[0135] This embodiment uses the nearest neighbor algorithm to l Retrieve K target-similar instances from the target text vector With G cl Text in The Euclidean distance value of As shown in formula (2):

[0136]

[0137] Where, yes The features of the mth dimension, yes The features of the mth dimension.

[0138] In this embodiment, the source similar instance set and the target similar instance set are obtained by nearest neighbor retrieval, as shown in formula (3):

[0139]

[0140] Where, is the source text vector The kth (1≤k≤10) source similarity instance, The target text vector The kth (1≤k≤10) target similar instance.

[0141] Step 4: Calculate the source similarity instance set Similarity weight set w p,i and target similar instance set Similarity weight set w q,i .

[0142] In this embodiment, the source text vector Set of instances similar to the source The inverse of the Euclidean distance value is normalized as the similarity weight set of the source similar instance, where the normalization uses the sigmoid activation function.

[0143] Let w p,i ={w p,i,1 ,…,w p,i,10} represents the source similar instance set Similarity weight set, where w p,i,k (1≤k≤10) represents the source text vector The similarity weight of the k-th source similar instance. Calculate the source text vector Set of instances similar to the source The similarity weight w p,i,k , as shown in formula (4):

[0144]

[0145] Let w q,i ={w q,i,1 ,…,w q,i,10} represents the target similar instance set Similarity weight set, where w q,i,k (1≤k≤10) represents the target text vector The similarity weight of the k-th target similar instance. Calculate the target text vector Similar instances to the target The similarity weight w q,i,k , as shown in formula (5):

[0146]

[0147] Step 5: Merge the source text vectors based on the similarity weights of the source similar instance set Similar instance set to the source Get the source text weight vector In this embodiment, the vector merging adopts the cumulative average method, as shown in formula (6):

[0148]

[0149] Where, is the source text vector The source text weight vector.

[0150] Step 6: Merge the target text vector based on the similarity weight set of the target similar instance set and target similar instance set Get the target text weighted vector In this embodiment, the vector merging adopts the cumulative average method, as shown in formula (7):

[0151]

[0152] Where, The target text vector The target text weight vector.

[0153] Step 7: Use the text matching model to train the source text weighted vector and the target text weighted vector Get the matching result y i ′.

[0154] Weight the source text vector and the target text weighted vector Substitute into the benchmark model (ESIM / ROBERTA) training to produce the matching result y i ′. If p [i] and q [i] Match, then y i ′=1, if p [i] and q [i] If it does not match, then y i ′=0.

[0155] Table 1 Comparison of model effects

[0156]

[0157] On the dataset, when the two baseline models were combined with contrastive learning and nearest neighbor retrieval, respectively, they both showed improved experimental results compared to the traditional baseline model (K = 10). We verified the superiority of combining the two in improving model performance (see Table 1).

[0158] In order to analyze the impact of K similar training instances of the nearest neighbor algorithm on the model matching effect, this embodiment selects the range of K to be [0, 20], and tests the impact of the change of K value on the matching effect of each model in the data set.

[0159] from Figure 3 The variation of the middle curve shows that, as K changes, the combination of this method and the deep learning model shows improved matching performance compared to the baseline model (K = 0). As the K value continues to increase, too many similar instances will also introduce more noise, resulting in a decrease in matching performance. However, as long as K > 0, the model experimental results are consistently better than the baseline deep learning model. This also verifies that this method can better extract text information during model prediction, directly confirming the effectiveness of this model in improving model performance.

[0160] In order to verify the superiority of text object weighting in improving the classification effect of the model, the experiment selected the range of K to be [0, 20], and designed the effects of constructing weighted text and simple text (unweighted) vectors on the model experimental effect ACC on the data set. Figure 4 Comparing the performance curves of each model shows that, as K initially increases, the difference between the experimental performance of weighted text objects on each model and the performance of simple text on the model is relatively small. As the value of K continues to increase, and the number of similar instances retrieved increases, the gap between the experimental performance of the weighted text objects on each model and the performance of the simple text objects on the model gradually becomes more significant. However, overall, the weighted text objects in this application are superior in improving model performance.

[0161] Example 2

[0162] In a second aspect, based on Example 1, this embodiment provides a text matching device for nearest neighbor enhanced contrastive learning pre-training, including a processor and a storage medium;

[0163] The storage medium is used to store instructions;

[0164] The processor is configured to operate according to the instructions to perform the method according to embodiment 1.

[0165] Example 3

[0166] In a third aspect, based on embodiment 1, this embodiment provides a device, including:

[0167] Memory;

[0168] processor;

[0169] as well as

[0170] computer programs;

[0171] The computer program is stored in the memory and is configured to be executed by the processor to implement the method described in embodiment 1.

[0172] Example 4

[0173] In a fourth aspect, based on Example 1, this embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in Example 1 is implemented.

[0174] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text matching method based on nearest neighbor enhanced contrastive learning pre-training, characterized in that: The method comprises: Establish source text set and target text set; Comparative learning of the pre-trained source text set and target text set to obtain new source text vectors and target text vectors respectively; Use the nearest neighbor algorithm to retrieve similar instances of the source text vector and the target text vector, and obtain the source similar instance set and the target similar instance set respectively; Calculating the similarity weight set of the source similar instance set and the similarity weight set of the target similar instance set includes: normalizing the inverse of the Euclidean distance value between the source text vector and the source similar instance set as the similarity weight set of the source similar instance set, and normalizing the inverse of the Euclidean distance value between the target text vector and the target similar instance set as the similarity weight set of the target similar instance set, wherein the normalization uses the sigmoid activation function, specifically including: Let w p,i ={w p,i,1 ,…,w p,i,K } represents the source similar instance set Similarity weight set, where w p,i,k Represents the source text vector Similar instance to the kth source The similarity weights, 1≤k≤K, are as follows: Let w q,i ={w q,i,1 ,…,w q,i,K } represents the target similar instance set Similarity weight set, where w q,i,k Represents the target text vector Similar instances to the kth target The similarity weights, 1≤k≤K, are as follows: Combine the target text vector based on the similarity weight set of the target similar instance set and target similar instance set Get the target text weighted vector include: where w q,i ={w q,i,1 ,…,w q,i,K } represents the target similar instance set Similarity weight set, w q,i,k Represents the target text vector Similar instances to the kth target Similarity weight, 1≤k≤K; Combine the target text vector based on the similarity weight set of the target similar instance set and target similar instance set Get the target text weighted vector include: where w q,i ={w q,i,1 ,…,w q,i,K } represents the target similar instance set Similarity weight set, w q,i,k Represents the target text vector Similar instances to the kth target Similarity weight, 1≤k≤K; The text matching model is used to train the source text weighted vector and the target text weighted vector to obtain the matching result.

2. The text matching method of nearest neighbor enhanced contrastive learning pre-training according to claim 1 is characterized in that Establish source and target text sets, including: make represents the source text set, where p [i] is the i-th source text, 1≤i≤N; let represents the target text set, where q [i] is the i-th target text, 1≤i≤N.

3. The text matching method of nearest neighbor enhanced contrastive learning pre-training according to claim 2, characterized in that: Comparative learning of the pre-trained source text set and target text set to obtain new source text vectors and target text vectors, including: The contrastive learning model Simcse is used to pre-train the source text set and target text set. After contrastive learning, the new source text vector is obtained as Target text set After contrastive learning, the new target text vector is obtained as 4. The text matching method of nearest neighbor enhanced contrastive learning pre-training according to claim 1, characterized in that Use the nearest neighbor algorithm to retrieve similar instances of the source text vector and the target text vector, and obtain the source similar instance set and the target similar instance set respectively, including: make is a vector containing all source text and the target text vector 2N text vector sets, express The j-th text vector, 1≤j≤2N; Calculate the source text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as source similar instances. Calculate the target text vector using the Euclidean distance formula and Except The similarity of the remaining text vectors is calculated, and the top-K similarity texts are selected as target similar instances. Through nearest neighbor retrieval, K source similar instances and K target similar instances are obtained, which constitute the source similar instance set. and target similar instance set 5. The text matching method of nearest neighbor enhanced contrastive learning pre-training according to claim 1, characterized in that: Use the text matching model to train the source text weighted vector and the target text weighted vector to obtain the matching results, including: Use text matching model to train source text weighted vector and the target text weighted vector Get the matching result y i ′, where y i ′ represents p [i] and q [i] The predicted value of the matching results.

6. A text matching device with nearest neighbor enhanced contrastive learning pre-training, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1 to 5.

Images