A hint-based masked exact matching dense retrieval method

By introducing a hint-based mask-exact match intensive search method in the information retrieval task, using pre-trained language model and two-way long and short memory neural network to generate semantic word vectors, the barrier between pre-training tasks and downstream tasks is solved, and information retrieval is improved The accuracy and efficiency of the word can be effectively processed by multiple meanings and multiple words.

CN115391505BActive Publication Date: 2024-12-03NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210989838.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-18
Publication Date
2024-12-03
Estimated Expiration
2042-08-18

AI Technical Summary

Technical Problem

In the prior art, due to the barrier between the pre-training task and the downstream task in the information retrieval task, there are bottlenecks in the model performance, and it is impossible to effectively solve the problem of multiple meanings of one word and multiple words.

Method used

Using a hint-based mask precise matching intensive search method, by adding prompt words to the input text of the original task, using a pre-trained language model to generate word vectors with semantic information, and combining a bidirectional long and short memory neural network and a multi-layer perceptron to generate prompts Vectors, perform word vector splicing and encoding, calculate the exact match and semantic matching scores of query and document, and achieve the improvement of the accuracy of information retrieval tasks.

Benefits of technology

It effectively solves the performance bottleneck problem of pre-trained language models in downstream tasks, improves the accuracy and efficiency of information retrieval, and significantly improves the correlation matching effect between query and document.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115391505B_ABST
    Figure CN115391505B_ABST
Patent Text Reader

Abstract

The present invention discloses a prompt-based masked exact matching dense retrieval method, which introduces prompt learning into information retrieval tasks, unifies the form of information retrieval tasks and the task form of pre-trained language models, and proposes a prompt-based masked exact matching dense retrieval model ProbMEM to perform masked exact matching dense retrieval, which can solve the problem of performance bottleneck in task effects of the pre-trained language model used in the third paradigm due to the gap between pre-training tasks and downstream tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of information retrieval, and in particular relates to a prompt-based masked exact matching intensive retrieval method. Background Art

[0002] The BM25 algorithm is a traditional information retrieval algorithm based on exact word matching. Specifically, given a query, when searching for documents related to the query in the document library, the BM25 algorithm only considers whether each word in the query statement has appeared in the document, and models the relevance between the query and the document based on the words that have appeared in the document. This approach is relatively simple and efficient, but it also has obvious shortcomings. There are multiple meanings of a word and multiple words for the same meaning in the language. It only considers the exact matching of words, not semantics, and cannot deal with problems such as searching for Apple and getting Apple phones.

[0003] To solve the above problems, the pre-trained language model is introduced into the intensive retrieval method of the traditional retrieval algorithm. With the rise of deep learning, the emergence of pre-trained language models has transformed the original feature engineering and structural engineering into the third paradigm: the paradigm of pre-training + fine-tuning. Under this paradigm, the model is first pre-trained on a large-scale corpus, and then fine-tuned on different downstream target tasks to achieve good task results. This technology is introduced into the field of information retrieval, and the language model encodes the words and generates the corresponding word vectors. The pre-trained language model uses the self-attention mechanism to fully consider the semantic information. Each generated word vector contains a certain amount of contextual information about the word. Therefore, by combining this method with the traditional retrieval algorithm, a word vector with semantic information can be generated, and then the correlation between the query and the document can be modeled based on the word vector, which can effectively alleviate the problem of polysemy and multiple words for one meaning. However, this method also has flaws, because pre-training is performed on a general large-scale corpus and has its own task objectives, while downstream tasks are diverse and have different training objectives from pre-training tasks. There is a gap between the two. During fine-tuning, the pre-training tasks are adapted to the downstream tasks, resulting in a bottleneck in model performance. When the training objectives of the upstream and downstream tasks are very different, satisfactory results cannot be achieved.

[0004] The emerging paradigm of natural language processing, the fourth paradigm, is also known as prompt-based learning. The fourth paradigm is different from the third paradigm mentioned above. It changes from the original pre-training + fine-tuning to pre-training + prompt + prediction. Under this paradigm, it is no longer the pre-training task that adapts to the downstream task. On the contrary, by adding additional prompt words to the input text of the original task and masking the target to be predicted, the learning form of the downstream task is reconstructed to make it consistent with the form of the pre-training task, so that the downstream task can adapt to the pre-training task. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a hint-based masked exact matching intensive retrieval method in view of the deficiencies of the above-mentioned prior art, introduce the idea of ​​the fourth paradigm into the field of information retrieval, further improve the accuracy of information retrieval tasks, and solve the problem that the pre-trained language model used in the third paradigm has a performance bottleneck in task effects due to the gap between the pre-training task and the downstream task.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A hint-based masked exact matching dense retrieval method adopts a dense retrieval model ProbMEM based on a pre-trained language model Encoder to perform masked exact matching dense retrieval, wherein the Encoder pre-adds a CLS vector bit to the front of each input query sentence and document sentence;

[0008] The specific process of dense retrieval model ProbMEM for mask exact matching dense retrieval is as follows:

[0009] Step 1: The prompt encoder generates a prompt vector of a specified length and concatenates the prompt vector with the word vector of the query sentence as a prefix;

[0010] Step 2: Find the words in the document that are repeated with the words in the query statement, and replace the location of the repeated words in the document with MASK, which is recorded as a MASK document;

[0011] At the same time, copy a document that has not been processed by MASK, replace the CLS position in the document with MASK, and record it as a CLS document;

[0012] Step 3: The query statement of the concatenated prompt vector, the MASK document, and the CLS document are encoded by the pre-trained language model Encoder to obtain the encoded word vector sequence, and then the encoded word vector is subjected to dimensionality reduction processing;

[0013] Step 4: Select the word in the query statement that has appeared in the MASK document as the target, find all occurrences of the target word in the MASK document, do a vector dot product with the corresponding query word in the query statement, and get the exact match score of the query statement and the document;

[0014] At the same time, take the vector of the CLS bit in the query statement and do a dot product with the CLS bit replaced by MASK in the CLS document to obtain the overall semantic matching score cls_match_score between the query statement and the document.

[0015] Step 5: Add exact_match_score and cls_match_score to obtain the final score of the query statement and document, final_match_score, and determine the optimal search result based on the final score.

[0016] To optimize the above technical solutions, the specific measures taken also include:

[0017] The prompt encoder described in step 1 above uses a combination of a bidirectional long short-term memory neural network and a two-layer multi-layer perceptron to generate a prompt vector of a specified length, and concatenates the prompt vector as a prefix and the word vector of the query sentence.

[0018] The hint vector of the specified length in step 1 above is:

[0019] E=MLP(BiLSTM(P0:P n ))

[0020] Among them, MLP is a two-layer multi-layer perceptron, BiLSTM is a bidirectional long short-term memory neural network, P0:P n is a randomly generated vector.

[0021] The above step 3 performs dimensionality reduction processing on the encoded word vector as follows:

[0022] h token =W token Encoder(input)+b token ,

[0023] h cls =W cls Encoder(input)+b cls ,

[0024] Among them, Encoder is encoder BERT;

[0025] W token The matrix for transforming the word vector sequence of the MASK document after BERT encoding;

[0026] b token is the threshold of the activation function corresponding to the MASK document;

[0027] W cls The matrix for transforming the word vector sequence of the CLS document after BERT encoding;

[0028] b cls is the threshold of the activation function corresponding to the CLS document.

[0029] The calculation formula for the exact match score of the query statement and the document in step 4 above is:

[0030]

[0031] Among them, h q wi is the word w in the query statement i The encoded vector, h d wi is the word w in the MASK document i The encoded vector, dot represents the vector dot product, max represents the maximum value, and mean represents the average value.

[0032] The calculation formula of the overall semantic matching score cls_match_score of the query statement and the document in step 4 above is:

[0033]

[0034] Among them, h q cls is the vector of the CLS position of the query statement, h d cls A vector of CLS locations for the CLS document.

[0035] The dense retrieval model ProbMEM is trained before being put into use. There are two training objectives: the matching loss between query statements and documents and the model loss;

[0036] The model loss is defined as follows:

[0037]

[0038] x j =W·h w

[0039] Among them, h w is the word vector at MASK in the MASK document;

[0040] W is a mapping matrix used to map word vectors to the entire BERT vocabulary;

[0041] x j is the score of each word in the vocabulary;

[0042] x* is the score of the actual word corresponding to MASK in the MASK document.

[0043] The matching loss is defined as follows:

[0044]

[0045] Among them, d+ is a positive example document;

[0046] gs is the number of positive and negative example documents corresponding to each query statement in a training batch.

[0047] The present invention has the following beneficial effects:

[0048] The present invention introduces prompt learning into information retrieval tasks, unifies the form of information retrieval tasks and the task form of pre-trained language models, and proposes a prompt-based masked exact match dense retrieval model ProbMEM (Prompt-based Masked Exact Match Dense Retriever) for masked exact match dense retrieval, which can solve the problem of performance bottleneck in task effects of pre-trained language models used in the third paradigm due to the gap between pre-training tasks and downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the structure of the dense retrieval model ProbMEM of the present invention;

[0050] Figure 2 This is a schematic diagram of dense indexing of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0052] The present invention discloses a prompt-based masked exact matching dense retrieval method, which uses a dense retrieval model ProbMEM based on a pre-trained language model Encoder to perform masked exact matching dense retrieval, wherein the Encoder pre-adds a CLS vector bit to the front of each input query sentence and document sentence;

[0053] Information retrieval task description: Given a query statement, you need to search the document library for a batch of documents that are most relevant to the query.

[0054] 1. Brief description of the model:

[0055] The query statement is apple tree, corresponding to Figure 1 in the Query section.

[0056] The query document is apple, Inc..., Figure 1 The Doc(cls mask) part (Doc(token mask)) is also from the document, but has a different purpose). It only lists the words apple and Inc, and other words are replaced by ellipsis. The word apple after the ellipsis represents other occurrences of the word apple in this document.

[0057] Encoder is a pre-trained language model. The BERT model is selected here. The encoders in the three parts are the same.

[0058] CLS is unique to the BERT model. BERT will add a CLS vector bit in front of each input sentence. After encoding, this vector can represent the overall semantic information of the input text.

[0059] 2. The specific process of dense retrieval model ProbMEM for mask exact matching dense retrieval is as follows:

[0060] Step 1: Generate a prompt encoder for the prompt word ( Figure 1 The E0…En, P0…Pn, H0…Hn, PromptEncoder part) uses a combination of a bidirectional long short-term memory neural network and a two-layer multilayer perceptron to generate a prompt vector of a specified length: E=MLP(BiLSTM(P0:P n )), and concatenate the hint vector as a prefix with the word vector of the query sentence;

[0061] Among them, MLP is a two-layer multi-layer perceptron, BiLSTM is a bidirectional long short-term memory neural network, P0:P n is a randomly generated vector.

[0062] Step 2: Find the words in the document that are repeated with the words in the query statement, and replace the positions of the repeated words in the document with MASK. Figure 1 The Doc (token mask) part is recorded as the MASK document; (MASK is a special word in the BERT vocabulary).

[0063] At the same time, copy a document that has not been processed by MASK, and replace the CLS position in the document with MASK. Figure 1 The Doc (cls mask) part is recorded as CLS documentation;

[0064] Step 3: Encode the query statement of the concatenated prompt vector, the MASK document, and the CLS document through the pre-trained language model Encoder to obtain the encoded word vector sequence, and then perform dimensionality reduction processing on the encoded word vector:

[0065] h token =W token Encoder(input)+b token ,

[0066] h cls =W cls Encoder(input)+b cls ,

[0067] Among them, Encoder is encoder BERT;

[0068] W t oken is the matrix for transforming the word vector sequence of the MASK document after BERT encoding;

[0069] b token is the threshold of the activation function corresponding to the MASK document, which is used to reduce the dimension of the vector;

[0070] W cls The matrix for transforming the word vector sequence of the CLS document after BERT encoding;

[0071] b cls is the threshold of the activation function corresponding to the CLS document, which is used to reduce the dimension of the vector;

[0072] This can reduce the generated word vectors and CLS vectors to the specified dimension, which is convenient for subsequent vector storage.

[0073] Step 4: Select the word in the query statement that has appeared in the MASK document as the target, find all occurrences of the target word in the MASK document, do a vector dot product with the corresponding query word in the query statement, and get the exact match score of the query statement and the document;

[0074] When the same query word appears multiple times in a document, take the maximum dot product value and the average dot product value, add the two together, and use them as the score of the exact match between the query and the document.

[0075]

[0076] Among them, h q wi is the word w in the query statement i The encoded vector, h d wi is the word w in the MASK document i The encoded vector, dot represents the vector dot product, max represents the maximum value, and mean represents the average value.

[0077] At the same time, take the vector of the CLS bit in the query statement and do a dot product with the CLS bit replaced by MASK in the CLS document to obtain the overall semantic matching score cls_match_score between the query and the document.

[0078]

[0079] Among them, h q cls is the vector of the CLS position of the query statement, hd cls A vector of CLS locations for the CLS document.

[0080] Step 5: Add exact_match_score and cls_match_score to obtain the final score of the query statement and document, final_match_score, and determine the optimal search result based on the final score.

[0081] The calculation formulas for the two scores are as follows:

[0082] final_match_score=exact_match_score+cls_match_score

[0083] The dense retrieval model ProbMEM is trained before being put into use, and the training objectives are two: the matching loss between the query statement and the document and the model loss;

[0084] The model loss is defined as follows:

[0085]

[0086] x j =W·h w

[0087] Among them, h w is the word vector at MASK in the MASK document;

[0088] W is a mapping matrix used to map word vectors to the entire BERT vocabulary;

[0089] x j is the score of each word in the vocabulary;

[0090] x* is the score of the actual word corresponding to MASK in the MASK document.

[0091] It is understandable that the formula x j =W·h w It maps the word vector obtained at MASK to the entire vocabulary. Assuming the vocabulary size is V, the word vector h w If W is a V×d matrix, then the obtained x j is a V-dimensional vector. The value of each dimension of this vector corresponds to the score of each word in the vocabulary, and x* is h w The score for this word.

[0092] The matching loss is defined as follows:

[0093]

[0094] Among them, d + is a positive example document;

[0095] gs is the number of positive and negative example documents corresponding to each query statement in a training batch (for example, gs = 8, of which 1 is a positive example document and 7 are negative example documents)

[0096] In the training set, a query statement is equipped with a document related to it and several documents that are not related to it (the total number is gs). The document related to it is called a positive document, and the documents that are not related to it are called negative documents.

[0097] After the model training is completed, the trained BERT model is used to encode the query statements and documents in the dataset, and the encoded word vectors are stored after dimensionality reduction for subsequent retrieval and matching.

[0098] Traditional information retrieval systems use inverted lists to index documents. Each word has a corresponding list that records the document number and corresponding word frequency information of the word. For dense retrieval using word vectors, an index can also be established based on the word list, but the information stored is changed from a simple scalar to a low-dimensional vector, which is called a vector index or dense index. This approach can pre-calculate and store the vector representation of the document. When performing online queries, a small amount of matrix operations can be performed based on the index to complete the matching of documents.

[0099] Dense indexes such as Figure 2 As shown, Figure 2 In the figure, the gray part on the left side of the dotted line is the word vector after all document words are encoded. Each vector has a corresponding document number (doc id) where the word appears. The part on the right side of the dotted line is the query word, and the arrow matmul represents matrix multiplication. For a query statement, the word vector of each word in it is matrix multiplied with the corresponding word matrix in the vector index to obtain the score between each query word and the same word in each document. Then, according to the query document matching score formula mentioned above, the total score of each document is calculated, and then all documents are sorted to obtain a list of candidate documents for the query.

[0100] Example verification:

[0101] The experiment uses two datasets, MS MARCO Passage Retrieval and MS MARCO Document Retrieval, which contain 8.8 million passages and 3.2 million documents, respectively. The results of the experiment with the task named FullRanking are reported: given a query, for Passage Retrieval, 1,000 candidate passages related to the query sorted by relevance are returned, and for Document Retrieval, 100 candidate documents related to the query sorted by relevance are returned. The Training queries of these two datasets are selected for training, and the Dev queries and the test queries of TREC DL 2019 (TREC is the name of the conference and DL is deep learning) are selected for effect testing. The experimental results are reported using three indicators: mrr@k (mean reciprocal rank), recall@k (recall rate), and ndcg@k (normalized discounted cumulative gain). The experimental results of the two datasets are shown in the following two tables.

[0102] Effect description:

[0103] Table 1: ProbMEM achieved an MRR score of 0.376 on the Dev queries of Passage Ranking, which greatly surpassed the traditional BM25 algorithm and discrete retrieval model. In the dense retrieval model, it was also about 7 percentage points higher than the single vector model (RepBERT). Among the multi-vector models of the same type, COIL achieved the same effect as ColBERT's full word interaction with a small amount of word interaction, and was more than three times faster. ProbMEM uses the same word interaction and query document matching score calculation method based only on overlapping words as COIL: overlap words + CLS, so it has the same speed advantage as COIL, but has surpassed it in accuracy.

[0104] Compared with ColBERT using full matching, MRR@10 and Recall@1k on Passage Ranking Dev queries also surpassed by 1.6 percentage points and 0.5 percentage points respectively. Using only word-to-word interactions of overlapping words means a certain degree of sacrifice in task accuracy. ProbMEM uses the method of prompt learning. Under the premise of the same amount of word-to-word interactions, it not only successfully makes up for the accuracy defects of COIL, but even outperforms ColBERT using full word-to-word interactions. This shows the effectiveness of our method: the introduction of prompt learning allows downstream tasks to complement pre-training tasks, while retaining the original downstream matching tasks. Downstream tasks can still benefit from pre-training tasks, and the two complement each other.

[0105] Table 2: As in the Passage Ranking task, ProbMEM significantly surpasses the traditional BM25 algorithm and discrete retrieval models DeepCT and DocT5Query. Compared with the single-vector dense retrieval model, each indicator has also been significantly surpassed. Since ColBERT does not provide experimental results for Document Ranking, the multi-vector dense model is only compared with COIL: on the Dev Queries of Document Ranking, mrr@10 surpassed by 2.5 percentage points, recall@1k surpassed by 1.3 percentage points, and on the test queries of DL, NDCG@10 surpassed by 1.4 percentage points, and mrr@1k surpassed by 1.5 percentage points. This confirms the effectiveness of our proposed method on long text tasks (the average document length of Document Ranking is 900 words, compared to the average of 60 words in the Passage Ranking task).

[0106] Table 1 Performance of various indicators on the full ranking task and DL test queries of the MS MARCO Passage Ranking dataset

[0107]

[0108]

[0109] Table 2 Performance of various indicators on the full ranking task and DL test queries of the MS MARCO Document Ranking dataset

[0110]

[0111] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A hint-based masked exact matching dense retrieval method, characterized in that: A dense retrieval model ProbMEM based on a pre-trained language model Encoder is used to perform masked exact matching dense retrieval. The Encoder pre-adds a CLS vector bit to the front of each input query sentence and document sentence; The specific process of dense retrieval model ProbMEM for mask exact matching dense retrieval is as follows: Step 1: The prompt encoder generates a prompt vector of a specified length and concatenates the prompt vector with the word vector of the query sentence as a prefix; Step 2: Find the words in the document that are repeated with the words in the query statement, and replace the location of the repeated words in the document with MASK, which is recorded as a MASK document; At the same time, copy a document that has not been processed by MASK, replace the CLS position in the document with MASK, and record it as a CLS document; Step 3: The query statement of the concatenated prompt vector, the MASK document, and the CLS document are encoded by the pre-trained language model Encoder to obtain the encoded word vector sequence, and then the encoded word vector is subjected to dimensionality reduction processing; Step 4: Select the word in the query statement that has appeared in the MASK document as the target, find all occurrences of the target word in the MASK document, do a vector dot product with the corresponding query word in the query statement, and get the exact match score of the query statement and the document; At the same time, take the vector of the CLS bit in the query statement and do a dot product with the CLS bit replaced by MASK in the CLS document to obtain the overall semantic matching score cls_match_score between the query statement and the document. Step 5: Add exact_match_score and cls_match_score to obtain the final score of the query statement and document, final_match_score, and determine the optimal search result based on the final score. The prompt encoder in step 1 uses a combination of a bidirectional long short-term memory neural network and a two-layer multi-layer perceptron to generate a prompt vector of a specified length, and concatenates the prompt vector as a prefix with the word vector of the query sentence; The hint vector of the specified length is: E=MLP(BiLSTM(P0:P n )); Among them, MLP is a two-layer multi-layer perceptron, BiLSTM is a bidirectional long short-term memory neural network, P0:P n is a randomly generated vector; The step 3 performs dimensionality reduction processing on the encoded word vector as follows: h token =W token ·Encoder(input)+b token , h cls =W cls ·Encoder(input)+b cls , Among them, Encoder is encoder BERT; W token The matrix for transforming the word vector sequence of the MASK document after BERT encoding; b token is the threshold of the activation function corresponding to the MASK document; W cls The matrix for transforming the word vector sequence of the CLS document after BERT encoding; b cls is the threshold of the activation function corresponding to the CLS document.

2. The hint-based masked exact matching dense retrieval method according to claim 1, characterized in that: The calculation formula for the exact match score of the query statement and the document in step 4 is: Among them, h q wi is the word w in the query statement i The encoded vector, h d wi is the word w in the MASK document i The encoded vector, dot represents the vector dot product, max represents the maximum value, and mean represents the average value.

3. The hint-based masked exact matching dense retrieval method according to claim 1, characterized in that: The calculation formula of the overall semantic matching score cls_match_score of the query statement and the document in step 4 is: Among them, h q cls is the vector of the CLS position of the query statement, h d cls A vector of CLS locations for the CLS document.

4. The hint-based masked exact matching dense retrieval method according to claim 1, characterized in that: The dense retrieval model ProbMEM is trained before being put into use, and the training objectives are two: the matching loss between the query statement and the document and the model loss; Among them, the model loss is defined as follows: x j =W·h w Among them, h w is the word vector at MASK in the MASK document; W is a mapping matrix used to map word vectors to the entire BERT vocabulary; x j is the score of each word in the vocabulary; x* is the score of the corresponding word at MASK in the MASK document; The matching loss is defined as follows: Among them, d + is a positive example document; gs is the number of positive and negative example documents corresponding to each query statement in a training batch.