A Dense Retrieval Method Based on Syntactic Contrastive Learning
By constructing a dense search method of dual encoder model and syntax masking strategy, the problems of large hardware consumption and low efficiency in the training of dense search model are solved, and efficient training and accurate retrieval in low data conditions are achieved.
Patent Information
- Application Number
- CN202211087538.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-07
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2042-09-07
AI Technical Summary
The dense search model has problems such as high hardware consumption and low training efficiency during the training process, especially when the batch volume and training cycle number are limited, resulting in poor performance in the face of new queries.
Using a dense search method based on syntactic comparison learning, a dual encoder model and a pretrained model is constructed, combined with the syntactic masking strategy of query and paragraphs, the total loss function is used to fine-tune the model, reduce the impact of syntactic interference, and improve training efficiency and accuracy.
In the case of low data, the training efficiency and accuracy of dense retrieval is significantly improved, hardware consumption is reduced, and better characterization space mapping capabilities are formed, which is better than existing methods.
Smart Images
Figure CN115374251B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of dense retrieval, and particularly to a dense retrieval method based on syntactic contrast learning. Background Art
[0002] In recent years, the proposed dense retrieval methods have significantly outperformed sparse retrieval methods in terms of effectiveness and have gradually become the mainstream method for relevant passage retrieval. Large-scale pre-trained models have recently become the cornerstone of methods for processing natural language processing tasks. Based on complex transformer structures and a large number of parameters within the model, pre-trained models can encode word and sentence representations with rich context information, which is achieved due to their carefully designed masking strategies.
[0003] Recent research has shown that when semantic information needs to be considered, dense retrieval can outperform traditional sparse retrieval methods that rely on exact lexical matching, such as TF-IDF and BM25. However, most basic pre-trained models are not suitable for semantic similarity search of sentences. It has been found that the semantic representation spaces of Bert and other pre-trained models are non-smooth anisotropic spaces, which is not conducive to the completion of semantic similarity tasks; the sentence vectors of two similar sentences may have a large distance, which will lead to the failure of the sentence vector space mapping task. Therefore, a key step in dense retrieval to solve these problems and improve the performance of dense retrieval is to adopt an appropriate contrast learning method during the representation learning process.
[0004] The performance and training effect of dense retrieval models depend to a large extent on contrastive learning methods. However, previous methods inevitably encounter the following problems:
[0005] Too large a batch size and too many training cycles will result in a large amount of hardware consumption: Previous research has focused on improving the performance of the model. Therefore, during the contrast learning process, the success of mining hard negative samples with a large amount of information has been proven to be effective. Unfortunately, the cost of obtaining complex hard negative samples is huge hardware resource consumption and lengthy calculations, and also because there are unlabeled positive samples in the training set and there are also unlabeled positive samples in the hard negative samples, regarding the correct answer as the wrong answer will undoubtedly damage the model; however, squeezing out unlabeled positive samples like RocketQA can surely mine more complex hard negative samples, but the cost is to train another cross-encoder to complete this task. In addition, methods based on improving the quality of negative samples use extremely large batch sizes and excessive training cycles to train for better representation mapping capabilities, and there is also a large amount of hardware consumption during the training process, such as having to use multiple GPUs for training and requiring extremely large video memory.
[0006] Poor training efficiency: Most previous studies have focused on increasing the similarity of correct query-passage pairs and reducing the similarity of incorrect query-passage pairs. However, the matching of questions and answers is an intuitive object-to-object search process in the human brain, but for machines, it is a chaotic similarity calculation process from object to set. Most previous studies used contrastive learning to train the dual encoder through the loss function proposed by DPR. In this case, when training with limited data in terms of batch size and number of training epochs, the poor training efficiency results in the model performing poorly when encountering completely new queries, and it is uncertain whether the correct answer passage can be found.
[0007] Therefore, finding a better method to represent dense vectors and improving the performance, effectiveness, and training efficiency of the model with fewer computing resources is a major problem that urgently needs to be solved. Summary of the Invention
[0008] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to improve the accuracy and training efficiency of dense retrieval results.
[0009] To solve the above technical problems, the present invention adopts the following technical solutions: A dense retrieval method based on syntactic contrastive learning, comprising the following steps:
[0010] S100: Select a publicly available document retrieval dataset, which includes queries (Query) and positive passages (Passage) + , where the Query and the positive passage (Passage) + correspond one by one, and a corresponding set of Query and Passage + is used as a training sample;
[0011] S200: Select some samples from the dataset as the training set C;
[0012] S300: Construct a dense retrieval model SynC, which includes a dual encoder model and two pre-trained models. The two pre-trained models are Encoder Q and Encoder P ;
[0013] The dual encoder model includes an encoder DualEncoder Q and an encoder DualEncoder P ;
[0014] S310: Randomly select a training sample t from C, process t using the BM25 method to obtain 4 most relevant negative passages corresponding to t, denoted as Passage - ;
[0015] Execute the syntactic masking strategy on the Query in t to obtain the masked query Query Masked ; For the Passage in t + Execute the syntactic masking strategy to obtain the masked positive passage
[0016] S320: Use the dual-encoder model to map the Query in t to a d-dimensional representation space to obtain the query representation matrix E of t q ; Use the dual-encoder model to map the Passage in t + to a d-dimensional representation space to obtain the positive passage representation matrix of t Use the dual-encoder model to map the corresponding Passage of t - to a d-dimensional representation space to obtain the negative passage representation matrix of t
[0017] S330: Use Encoder Q to perform the encoding and masking strategy on the Query Masked to obtain the masked query c q ; Use Encoder P to perform the encoding and masking strategy to obtain the masked positive passage c p ;
[0018] S400: Repeat S310 - S330 to calculate the E corresponding to each training sample in the training set C q 、 c q and c p ; S500: Calculate the total loss of SynC, and the expression is as follows:
[0019] Loss TOTAL = αLoss qp + βLoss qq + γLoss pp；(1)
[0020] where Loss TOTAL represents the total loss function, α, β, γ represent loss coefficients, Loss qp represents the query-passage loss, Loss qq represents the query-query loss, Loss pp represents the passage-passage loss;
[0021] S600: Set the maximum number of training iterations and train SynC, using Loss TOTALThe loss function updates the SynC parameters in reverse through the gradient descent method, and stops training when the maximum number of iterations is reached, obtaining the trained SynC;
[0022] S700: Use the query Q' to be predicted as the data input of the trained SynC, and the output is the dense retrieval result of the query Q'.
[0023] Preferably, the method for obtaining the query representation matrix E q , positive passage representation matrix and negative passage representation matrix E p - in S320 is as follows:
[0024] E q = DualEncoder Q (Query); (2)
[0025]
[0026]
[0027] Apply two existing pre-trained models, Bert and coCondenser, as encoder types and conduct two comparative experiments. The original Bert model has poor representation mapping ability in the dense retrieval task, while coCondenser is a complex dense retrieval pre-trained model with state-of-the-art representation mapping ability. Using two extreme models for testing can well verify the method of the present invention.
[0028] Preferably, the calculation formulas for obtaining the masked query c q and the masked positive passage c p in S330 are as follows:
[0029] c q = Encoder Q (Query masked ); (5)
[0030]
[0031] The encoding process maps all the obtained data into dense vectors in the representation space and then prepares for contrastive learning after encoding.
[0032] Preferably, the specific steps for calculating the total loss of SynC in S500 are as follows:
[0033] S510: Calculate the query-passage loss Loss qp of SynC, and the specific expression is as follows:
[0034]
[0035] Among them, N represents the total number of negative examples in a training batch, i represents the i-th negative example, and sim(·) represents similarity calculation, that is, the dot product of vectors;
[0036] S520: Calculate the query-query loss Loss of SynC qq , and the specific expression is as follows:
[0037]
[0038] Among them, max(·) represents taking the maximum as the output compared with 0, and when it is less than 0, the loss is 0, and a represents a hyperparameter;
[0039] S530: Calculate the passage-passage loss Loss of SynC pp , and the specific expression is as follows:
[0040]
[0041] Among them, b represents a hyperparameter.
[0042] The total loss function is used to fine-tune the dual encoder. The query-to-query loss and passage-to-passage loss are used as auxiliary adjustments to force E q and to get rid of the influence of syntactic interference, so as to generate enhanced syntactic isolation representations. By using the syntactic isolation representations to fine-tune the dual encoder, the training efficiency can be improved.
[0043] Compared with the prior art, the present invention has at least the following advantages:
[0044] 1. Improve the training quality through the query-based syntactic masking strategy and the passage-based syntactic masking strategy: When a new query or a passage with different syntax appears, an inefficient model is sometimes affected by the syntax and encodes it as a distant dense representation in the embedding space. When designing the model, the syntax as the input sentence template is first separated from the input object. The present invention proposes query-based and passage-based masking strategies. In the process of answering questions, nouns are always the key query objects of the questions. Regarding it as the target, while the grammar in the query is a kind of decoration, expressing what type of specific object information the questioner wants to collect. In addition, since the syntax and the object are similar, the dense representations between different queries also have a relatively small distance in the embedding space, which will affect the retrieval performance and damage the representation space. Therefore, applying this method to remove the syntactic information can force the contrast learning process to focus on the valuable parts in the given data and will improve the training efficiency and training quality.
[0045] 2. Improve training efficiency using syntactic contrastive learning: Fine-tune the dual encoder through the proposed contrastive learning loss function. The total loss of the loss function is composed of query-to-query loss, paragraph-to-paragraph loss, and query-to-paragraph loss combined in different proportions. The syntactic contrastive learning method fine-tunes the pre-trained dual encoder through query-to-query loss and paragraph-to-paragraph loss, reducing the similarity between the question and the masked question, and also reducing the similarity between the representations of other queries in the given query batch; reducing the similarity between the positive paragraph and the masked positive paragraph, and also reducing the similarity between the representations of other positive paragraphs in the given query batch. The query-to-paragraph loss shortens the distance between the representations of relevant query-paragraph pairs and pulls apart the representations of irrelevant question-answer pairs. Applying our method helps queries and paragraphs with similar Q&A templates get rid of the influence of syntactic similarity and obtain better representation space mapping ability, and has higher training efficiency. While improving efficiency, we can train the model with low data without excessive hardware consumption. Brief Description of the Drawings
[0046] Figure 1 It is the training process of the SynC model of the present invention.
[0047] Figure 2 It is the comparison between the query-based syntactic masking strategy and the paragraph-based masking strategy in the present invention.
[0048] Figure 3 It is the comparison between the contrastive learning method of the present invention and the traditional contrastive learning method. Detailed Description of the Invention
[0049] The present invention will be further described in detail below.
[0050] The present invention discloses a syntactic contrastive learning method for dense retrieval, which is based on a specially designed masking strategy. The model proposes a query-based syntactic masking strategy and a paragraph-based syntactic masking strategy to retain and highlight the syntactic information of the data to be processed; then implement a syntactic-based contrastive learning method through the masked data to obtain syntactic isolation representations of queries and paragraphs; this model reduces the impact of the syntax of queries and paragraphs on the representation quality, and at the same time achieves good results without consuming huge training cycles and hardware resources; the efficiency and quality of training are proven, and it is hoped to obtain better performance without low data. The present invention improves the masking strategy and adjusts the model structure to obtain better representations.
[0051] See Figures 1-3 , a dense retrieval method based on syntactic contrastive learning, comprising the following steps:
[0052] S100: Select a publicly available document retrieval dataset, where the dataset includes queries (Query) and positive passages (Passage). + , where the Query and the Passage + are in one-to-one correspondence, and a corresponding pair of Query and Passage + is used as a training sample;
[0053] S200: Select a part of the samples from the dataset as the training set C;
[0054] S300: Build a dense retrieval model SynC, where SynC includes a dual-encoder model and two pre-trained models. The two pre-trained models are Encoder Q and Encoder P , and the pre-trained models are prior arts;
[0055] The dual-encoder model includes encoder DualEncoder Q and encoder DualEncoder P ;
[0056] S310: Randomly select a training sample t from C, and process t using the BM25 method to obtain 4 most relevant negative passages corresponding to t, denoted as Passage - , and the BM25 method is prior art;
[0057] The preprocessed data will form query-passage pairs. Finally, contrastive learning is applied to increase the similarity of correct query-passage pairs and reduce the similarity of incorrect query-passage pairs in the last step.
[0058] Execute the syntactic masking strategy on the Query in t to obtain the masked query Query Masked ; Execute the syntactic masking strategy on the Passage + in t to obtain the masked positive passage The syntactic masking strategy is prior art;
[0059] The syntactic masking strategy is a preliminary data preprocessing for the final contrastive learning process. Different from the pre-trained model based on Bert that predicts randomly masked words as cloze tasks, the syntactic masking strategy does not perform any prediction work. Its ultimate goal is to hide the semantic information that scatters the model's attention and prepare the syntactic isolation representations required for contrastive learning. The query-based syntactic masking strategy masks all nouns in the query. If the nouns do not reach 25% of the total length, other words in the sentence are randomly masked. Since the query is a short question sentence, it is very likely that the nouns exceed 25% of the query. For example, the masked query can be: "Where is [mask] [mask]? Which part of the [mask] do [mask] prefer?" In this case, the syntactic structure of the question sentence will be maximally preserved and highlighted.
[0060] For the passage-based masking strategy, first of all, the semantic information of the masked tokens in the query cannot be leaked. Otherwise, the model's attention will be directed to semantics rather than syntax, and its training efficiency will also be impaired. Therefore, a key step in the masking strategy is to mask the words that appear in the corresponding query. To demonstrate how to obtain the most representative syntactic isolation representations, two passage-based strategies are divided into two conditions. The first condition is that if the words masked in the query account for more than 25% of the passage words, then all query words in the passage are masked. The second case is that if the words shown in the query account for less than 25% of the passage words, then other words are randomly masked until 25% of the passage words are masked.
[0061] S320: Use a dual-encoder model to map the Query in t to a d-dimensional representation space to obtain the query representation matrix E of t q ; Use a dual-encoder model to map the Passage in t + to a d-dimensional representation space to obtain the positive passage representation matrix of t Use a dual-encoder model to map the corresponding Passage of t - to a d-dimensional representation space to obtain the negative passage representation matrix of t
[0062] The method to obtain the query representation matrix E of t q , the positive passage representation matrix and the negative passage representation matrix in the above S320 is as follows:
[0063] E q = DualEncoder Q (Query); (2)
[0064]
[0065]
[0066] S330: Use the Encoder Q for the Query Masked to perform encoding and masking strategies to obtain the masked query c q ; Use the Encoder P for to perform encoding and masking strategies to obtain the masked positive passage c p ;
[0067] In the above S330, the masked query c q and the masked positive passage c p are calculated as follows:
[0068] c q = Encoder Q (Query masked ); (5)
[0069]
[0070] S400: Repeat S310 - S330 to calculate E corresponding to each training sample in the training set C q , c q and c p ;
[0071] S500: Calculate the total loss of SynC, and the expression is as follows:
[0072] Loss TOTAL = αLoss qp + βLoss qq + γLoss pp；(1)
[0073] where Loss TOTAL represents the total loss function, α, β, γ represent loss coefficients, Loss qp represents the query - passage loss, Loss qq represents the query - query loss, Loss pp represents the passage - passage loss;
[0074] The specific steps for calculating the total loss of SynC in the above S500 are as follows:
[0075] S510: Calculate the query - passage loss Loss qp of SynC, and the specific expression is as follows:
[0076]
[0077] Among them, N represents the total number of negative examples in a training batch, i represents the i-th negative example, and sim(·) represents similarity calculation, that is, the dot product of vectors;
[0078] By fine-tuning the dual encoder with the query-passage loss, the distance between the representations of relevant query-passage pairs is shortened, and the representations of irrelevant question-answer pairs are pulled apart; this loss function is the core of the contrastive learning method in previous dense retrieval, and it shows the fine-tuning effect in the representation space. See Figure 3 .
[0079] S520: Calculate the query-query loss Loss of SynC qq , and the specific expression is as follows:
[0080]
[0081] Among them, max(·) represents the maximum as the output compared with 0, and the loss is 0 when it is less than 0, and a represents a hyperparameter;
[0082] Loss qq The program of the function is to reduce the similarity between the representation E q of the given query and the representation c Masked of the masked syntactic template Query q , and at the same time reduce the similarity between the given query and the representations of queries in other batches; it not only reduces the distance between the given query and its syntactic template, but also reduces the distance between different query samples in the same training batch, thus making full use of the existing data.
[0083] S530: Calculate the passage-passage loss Loss of SynC pp , and the specific expression is as follows:
[0084]
[0085] Among them, b represents a hyperparameter.
[0086] Loss pp reduces the similarity between the representation + of the given Passage and the representation c of the masked syntactic template p , and reduces the similarity between the given passage and the representations of passages in other batches. The hyperparameters a and b limit the loss function so that it overly separates syntactic information when the similarity of the query-masked pair and the passage-masked pair is too high. In this case, the loss function will be less than 0 and automatically turn to 0 to avoid damaging the representation space.
[0087] S600: Set the maximum number of training iterations and train SynC, using Loss TOTAL The loss function updates the SynC parameters backward through the gradient descent method. The gradient descent method is a prior art. When the training reaches the maximum number of iterations, the training stops, and the trained SynC is obtained;
[0088] S700: Use the query Q' to be predicted as the data input of the trained SynC, and the output is the dense retrieval result of the query Q'.
[0089] Experimental verification
[0090] Dataset
[0091] Experiments were conducted on the well-known document retrieval benchmark dataset MSMARCO Passage Ranking. The detailed statistics of this dataset are shown in Table 1 below. MSMARCO is a large-scale reading comprehension and question answering dataset. In MSMARCO, all questions are mined from real anonymous users and collected from Bing search logs. This dataset has more than 880k paragraphs, and the answers to each query are labeled. Here, "p" and "q" are abbreviations for question and paragraph respectively. The length is in tokens.
[0092] Table 1 Detailed statistics of MSMARCO
[0093] dataset #q(training) #q(development) #q(test) #p MSMARCO 502,939 6,980 6,837 8,841,823
[0094] Evaluation metrics
[0095] According to the mainstream evaluation method, MRR@10 and Recall@k are used as the evaluation metrics of the present invention. MRR is called the mean reciprocal rank, which is a general mechanism for evaluating search algorithms. It refers to the average of the mutual rankings of each test question when the relevant paragraph is retrieved for the first time. Recall@k refers to the recall rate at the first k positions, which is the proportion of questions whose retrieved paragraphs in the first k positions contain the correct answer.
[0096] Implementation details
[0097] Experiments were conducted on four NVIDA RTX3060 GPUs (12G video memory), and the deep learning framework Pytorch was used. In the first experiment, all encoders were initialized by Bert base ; the batch size of each device was set to 8, so the total batch size was 32, and the number of training epochs was set to 2. Since the representation mapping ability of Bert was poor, the learning rate was set to 5e -5, which is larger than the second experiment; Use coCondenser to initialize all the encoders in the second experiment. It is a pre-trained model particularly suitable for dense retrieval tasks; Therefore, to avoid the performance degradation caused by a large learning rate, the present invention sets the learning rate to 5e -6 , the batch size for each device is 8, so the total batch size is 32. To test whether the method of the present invention can improve a pre-trained model specifically designed with excellent representation ability, the number of training epochs is set to 1; In the BM25 negative mining part, each query will generate 4 hard negative examples applying BM25, and the loss hyperparameters a and b are set to 10, α is set to 0.75, and β and γ are set to 0.25. In this experiment, FAISS is used to index the dense representations of paragraphs, and the present invention implements InxdexFlatIP to complete the indexing task, and the queries are performed through exact maximum inner product search.
[0098] Comparative Experiments and Results
[0099] The SynC model is compared with previous benchmark sparse retrieval methods and dense retrieval methods. For sparse retrieval models, including the most classic BM25 based on lexical matching, as well as doc2qury, DeepCT, and docTTTTTquery, all of which are enhanced by neural networks; The selection of dense retrieval models is ANCE, ME-BERT, RocketQA, PAIR, and coCondenser. ANCE uses Roberta to initialize the dual encoders and uses a SynChronously updated ANN index to select global negative samples from the entire corpus; DPR implements in-batch random sampling and BM25 to obtain hard negative samples. However, DPR has not been experimented on the MSMARCO dataset, and a concurrent work, DPR-PAQ, is also considered in the comparison; RocketQA introduces a cross-batch negative sampling strategy and applies a trained cross-encoder to denoise the hard negatives; PAIR considers not only passage-centered similarity but also query-centered similarity when performing contrastive learning; Condenser and coCondenser achieve a better representation space by adjusting the structure of the transformer and defining a corpus-aware contrastive loss, shifting their focus to pre-training methods.
[0100] The experiments are conducted through two extreme pre-trained models to verify the effectiveness of the method of the present invention. Since the original Bert model has poor representation mapping ability in dense retrieval tasks, while coCondenser is a complex dense retrieval pre-trained model with state-of-the-art representation mapping ability, two comparative experiments are carried out using the Bert and coCondenser pre-trained models as encoder types.
[0101] Since the experiments of the present invention were conducted under low data volume, the training settings of the above-mentioned model, especially the batch size and the number of training epochs, were compared with the method proposed by the present invention. The detailed settings are shown in Table 2. The pre-training processes of Condenser and coCondenser are not shown in the table because their actual hardware consumption is relatively large:
[0102] Fine-tuning settings of the DR model compared in Table 2
[0103]
[0104]
[0105] It can be concluded from Table 2 that since all the fine-tuning settings were obtained when the best performance occurred, and the pre-training hardware consumption of the models condenser and coCondenser, which is extremely large, is not shown in this table, compared with the above-mentioned models, SynC has the lowest hardware consumption, a relatively small batch size, and the fewest number of training epochs.
[0106] Table 3 Experimental results on MSMARCO
[0107]
[0108] Compared with the baseline model, the best performance of the proposed method on the MSMARCO dataset is shown in Table 3. The results are divided into three parts according to the pre-trained models: sparse retrieval, Bert-based dense retrieval, and dense retrieval with more complex pre-trained models; the dense passage retrieval method is significantly better than all sparse retrieval methods, even those sparse retrievals enhanced by neural networks such as doc2query, DeepCT, and docTTTTTquery; in addition, applying more complex pre-trained models to initialize the encoder has obvious performance improvement, which obtains better representation mapping ability due to more parameters, advanced training strategies, or specially designed structures.
[0109] Compared with the baseline, by Bert baseThe MRR@10 value of the initialized SynC reaches 35.0, and the R@1000 value reaches 96.5. Compared with the best sparse retrieval baseline, the MRR performance is improved by 7.3% (27.7 / 35.0); compared with the best Bert-based dense retrieval baseline, the MRR performance is improved by 1.2% (33.8 / 35.0); the performance of R@1000 is 1.8% higher than the best sparse retrieval baseline (94.7 / 96.5); even in the case of less data, SynC can still form a better representation space, perform best among the Bert-based baselines, and even outperform the more complex pre-trained model Roberta base ANCE.
[0110] SynC initialized by coCondenser also has the highest performance among the dense models initialized by better pre-trained language models. Its MRR@10 value is 35.0 and its R@1000 value is 96.5. Since coCondenser is a state-of-the-art pre-trained model designed specifically for dense retrieval tasks and has the best representation mapping ability, SynC still has a 0.2% (98.4 / 98.6) gain in R@1000 in the low-data case, and only uses 1 training cycle.
[0111] Model Performance and Training Efficiency Evaluation
[0112] Compared with the Bert-based DR model, SynC has the best performance in MRR and recall@1000. DPR uses traditional contrast learning methods to improve the similarity of correct query-passage pairs and reduce the similarity of incorrect query-passage pairs without data processing and structural adjustment; SynC has a gain of more than 7.3% (27.7 / 35.0) in MRR with less than 1 / 4 of the batch size and 1 / 10 of the number of training cycles; since their training methods are the same, except for the improvement of the masking strategy and syntactic contrast learning methods, this experiment has achieved significant performance improvement with extremely low data. Therefore, the method proposed in the present invention is proven to be effective in promoting the training process and improving the training quality, the training efficiency is verified, and the proposed method can handle low-data settings.
[0113] Compared with the Bert-based model, the DR model using more complex models and data processing methods significantly improves performance. However, the cost of resource occupancy and training consumption is huge. RocketQA violently uses a batch size of 4096 and has 60 training cycles. Such training resources are impossible for most researchers to achieve. However, the Condensor and coCondenser models aim to adjust the transformer structure of Bert to obtain better sentence representations. After 8 epochs of pre-training, Condensor only uses a batch size of 64 and 3 epochs for fine-tuning. Since their ultimate goal of adjusting the architecture is to obtain better representations, which is the main contribution of SynC, and coCondenser has the best performance in the baseline, SynC was tested on the coCondenser model with a batch size of 24 for 1 hour to see if the method of the present invention contributes to the representation. SynC increased the recall@1000 from 98.4 to 98.6, proving that even in low-data situations, it can still obtain better representations than strong baselines. Combining the information in Table 2 and Table 3, SynC has the best training efficiency and the least hardware consumption with a smaller batch size and duration.
[0114] Table 4 Influence of the number of training cycles on the performance of SynC
[0115]
[0116] Table 4 shows the influence of the training cycle on the performance of SynC. Batch size = 32, α = 0.75, β = 0.25, learning rate = 5e -5 (Bert-based SynC), learning rate = 5e -6 (coCondenser-based SynC). SynC was tested in low-data situations, so 3 training cycles are the maximum training settings carried out by the present invention. Since the original Bert model has an anisotropic space with non-smooth semantic representations, which is not conducive to semantic similarity tasks, the performance is worse than sparse retrieval. However, after one training cycle of SynC training, the MRR result is 31.4, reaching the best performance of DPR. The training effect and efficiency far exceed DPR. After 2 training cycles, the performance reaches the best, and the performance slightly decreases after 3 training cycles. Therefore, SynC has a good training efficiency, and its best performance is achieved in 2 durations.
[0117] Although training for one epoch on coCondenser only results in a small improvement in R@1000, it still shows that the method proposed in the present invention can form a better representation space on the existing state-of-the-art dense retrieval pre-trained model. During the experiment, the pre-trained model was not damaged, but rather the model's representation space mapping ability was further improved.
[0118] Table 5 Influence of Hyperparameters on SynC Performance
[0119]
[0120] Table 5 shows the experiments previously used to compare the performance affected by hyperparameters. In the low-data scenario, the experiments of the present invention considered different results obtained under different settings to evaluate the hyperparameters: it was found from Experiments 1 and 2 that better results could be obtained by not setting α and β to 1 in the total loss function, but arranging the three losses in a certain proportion (α = 0.75, β = 0.25, γ = 0.25); in addition, applying a low learning rate would reduce the training efficiency, and the best performance of an appropriate learning rate was 5e -5 . And during this efficient training process, the performance reached its peak at the 2nd training epoch.
[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.
Claims
1. A dense retrieval method based on syntactic contrastive learning, characterized in that: It includes the following steps: S100: Select a publicly available document retrieval dataset, which includes queries (Query) and positive passages (Passage). + , where the Query and the positive passage Passage + correspond one-to-one, and a corresponding pair of Query and Passage + is used as a training sample; S200: Select a part of the samples from the dataset as the training set C; S300: Build a dense retrieval model SynC, where SynC includes a dual-encoder model and two pre-trained models, namely Encoder Q and Encoder P ; The dual encoder model includes the encoder DualEncoder Q and the encoder DualEncoder P ; S310: Randomly select a training sample t from C, process t using the BM25 method, and obtain the 4 most relevant negative paragraphs corresponding to t, denoted as Passage - ; Execute the syntactic masking strategy on the Query in t to obtain the masked query Query Masked ; For the Passage in t + Execute the syntactic masking strategy to obtain the masked positive passage S320: Map the Query in t to a d-dimensional representation space using a dual-encoder model to obtain the query representation matrix E of t q ; Use the dual-encoder model to process the Passage in t + Map it to a d-dimensional representation space to obtain the positive passage representation matrix of t Use the dual-encoder model to map the Passage corresponding to t - To a d-dimensional representation space to obtain the negative passage representation matrix of t S330: Use the Encoder Q For the Query Masked Perform encoding and masking strategies to obtain the masked query c q ; Use the Encoder P For Perform encoding and masking strategies to obtain the masked positive passage c p ; S400: Repeat S310 - S330 to calculate E corresponding to each training sample in the training set C q 、 c q and c p ; S500: Calculate the total loss of SynC, and the expression is as follows: Loss TOTAL = αLoss qp + βLoss qq + γLoss pp ; (1) Among them, Loss TOTAL represents the total loss function, α, β, and γ represent loss coefficients, and Loss qp represents the query-passage loss, and Loss qq represents the query-query loss, and Loss pp represents the passage-passage loss; S600: Set the maximum number of training iterations and train SynC, using Loss TOTAL The loss function updates the SynC parameters backward through the gradient descent method, stops training when the training reaches the maximum number of iterations, and obtains the trained SynC; S700: Use the query Q' to be predicted as the data input of the trained SynC, and the output is the dense retrieval result for the query Q'.
2. The dense retrieval method based on syntactic contrast learning according to claim 1, wherein: Obtain the query representation matrix E of t in S320 q , the positive paragraph representation matrix and the negative paragraph representation matrix The method is as follows: E q = DualEncoder Q (Query); (2) 3. The dense retrieval method based on syntactic contrast learning according to claim 2, wherein: The masked query c obtained in S330 q and the masked positive paragraph c p are calculated as follows: c q = Encoder Q (Query masked ); (5) 4. The dense retrieval method based on syntactic contrast learning according to claim 3, characterized in that: The specific steps for calculating the total loss of SynC in S500 are as follows: S510: Calculate the query-passage loss Loss of SynC qp , and the specific expression is as follows: Among them, N represents the total number of negative examples in a training batch, i represents the i-th negative example, sim(·) represents similarity calculation, that is, the dot product of vectors; S520: Calculate the query-query loss of SynC qq , and the specific expression is as follows: Among them, max(·) represents the maximum compared with 0 as the output, and the loss is 0 when it is less than 0, and a represents a hyperparameter; S530: Calculate the paragraph-paragraph loss of SynC pp , and the specific expression is as follows: Among them, b represents a hyperparameter.