Semantic retrieval model fusion method and system based on adaptive weights

Abstract

The application provides a semantic retrieval model fusion method based on adaptive weights, comprising: performing sparse retrieval on retrieval information to obtain a sparse retrieval result and a corresponding sparse ranking score; performing dense retrieval on the retrieval information to obtain a dense retrieval result and a corresponding dense ranking score; constructing a weight prediction model, and obtaining a fusion weight through the weight prediction model based on the retrieval information, the sparse retrieval result and the dense retrieval result; generating a fusion ranking score based on the fusion weight, the sparse ranking score and the dense ranking score; and sorting the sparse retrieval result and the dense retrieval result according to the fusion ranking score to generate a final retrieval result. The application also provides a semantic retrieval model fusion system based on adaptive weights and a data processing device for semantic retrieval model fusion based on adaptive weights.

Description

Technical Field

[0001] This invention relates to the field of information retrieval, and in particular to a method and system for integrating semantic-based dense retrieval and term-matching-based sparse retrieval. Background Technology

[0002] With the development of the internet, a massive amount of content is generated online every day. Numerous websites have emerged; as of June 2022, my country had 3.98 million websites. Many websites, such as Zhihu and Xiaohongshu, have users creating a vast amount of content daily. As of December 31, 2021, Zhihu had accumulated 490 million pieces of content, including 420 million questions and answers.

[0003] Search engines are becoming increasingly diverse, encompassing traditional web search engines, e-commerce searches for goods, searches for local services, and searches for user-generated content. This variety of search scenarios presents greater challenges, demanding a higher level of understanding of the semantic relevance between user queries and documents. Word-matching-based retrieval technologies have limitations, failing to handle synonyms, polysemous words, and the overall semantics of sentences. Therefore, semantic-based retrieval technologies have become a crucial issue in the field of information retrieval.

[0004] With the introduction of pre-trained language models, current deep semantic retrieval methods based on dense vector retrieval have completely surpassed the traditional BM25 algorithm on datasets such as MS MARCO and NaturalQuestions. For example, on NaturalQuestions, the semantic retrieval model based on dense vectors only requires 1000 training data points to outperform the BM25 algorithm.

[0005] However, researchers have found that current semantic-based retrieval models are not good at capturing the semantics of entities in text, resulting in poor performance in queries containing entities. For example, for the query "Who plays Thoros of Myr in Game of Thrones?", the BM25 algorithm successfully retrieved the entity "Thoros of Myr" and found text containing the correct answer. In contrast, the dense vector retrieval model retrieved an unrelated encyclopedia page for another actor.

[0006] Current retrieval systems primarily operate on a retrieval-ranking model. The first stage, retrieval, aims to select a small subset (typically thousands) of candidate documents from a large corpus. Then, a more complex model is used to further rank these candidate documents. The first stage of retrieval forms the basis for subsequent ranking; if the retrieval stage fails to recall the required documents, the subsequent ranking becomes meaningless.

[0007] Based on how the text is represented, it can be divided into sparse retrieval and dense retrieval.

[0008] Sparse retrieval primarily represents text using sparse vectors the size of a vocabulary, and indexing is mainly based on inverted indexes. The main difference between different sparse retrieval algorithms lies in how they determine the weight of each term. Traditional sparse retrieval algorithms mainly rely on statistical information, such as term frequency and inverse document frequency, to determine word weights, such as the BM25 algorithm.

[0009] Dense retrieval primarily uses dense vectors to represent text, which can represent the semantics of the text. Retrieval is usually achieved through nearest neighbor search of vectors.

[0010] A typical architecture for dense vector retrieval uses two encoders to encode the query and the document respectively. The document can be pre-encoded and indexed using an Approximate Nearest Neighbor (ANN) index. For a new query, it is first encoded using the query encoder to obtain a query-encoded vector. Then, relevant documents are found through Maximum Inner-Product Search (MIPS). The core of dense retrieval lies in how to encode the query and document into vectors that effectively represent semantics and are discriminative between different texts. Existing dense retrieval models primarily use two BERTs to encode the question and paragraph respectively, obtaining their respective vector representations.

[0011] Sparse retrieval models use words as the unit of representation, and because the matching score is calculated based on exact matching signals, they have strong discriminative power. On the other hand, dense retrieval models learn the continuous embeddings of queries and documents to encode their semantic information and evaluate semantic relevance. Therefore, it is natural to consider mixing sparse and dense retrieval models to ensure that retrieval takes into account both the semantic relevance between texts and the semantics of key entities. Hybrid retrieval refers to using different retrieval models to obtain relevance scores between documents and queries, and then ranking them according to a predefined fusion rule to obtain a fused relevance score.

[0012] Existing methods fuse the BM25 model and the BERT-based dense retrieval model using fixed-weight score fusion. The current mainstream approach to determining weights involves testing on a development set and selecting a fusion weight specific to the current dataset. This fixed-weight fusion method across the entire dataset falls significantly short of the upper bound. This implies that dynamically setting the model's fusion weights for each query could further improve retrieval performance. To focus on improving the model's performance on queries where word-matching algorithms excel, a semantic retrieval model is trained using residual learning to enhance its performance on samples where the BM25 model is less effective. Finally, the scores from the semantic retrieval model and the BM25 model are weighted and summed.

[0013] Current common sparse and dense retrieval model score fusion weights are still obtained by parameter tuning on the validation set, without individually determining the weights for each query. Since each query has different characteristics, it is unreasonable for all queries to share a single fusion weight.

[0014] Existing retrieval methods can be mainly divided into two categories: sparse retrieval and dense retrieval. Sparse retrieval excels at word matching and has strong generalization ability, while dense retrieval mainly models the semantic similarity between queries and documents. However, dense retrieval is not good at capturing the semantics of key entities. To address this issue, current researchers have attempted to fuse sparse and dense models with fixed weights through static weights and to ensure complementarity between models during training by fitting residuals. However, these methods all have a problem: they cannot consider the characteristics of different queries to select different weights for different model scores. Different queries are suitable for different retrieval models. For queries with relatively fixed expressions, such as drug names, "compound acetaminophen and amantadine tablets," which have no other expressions, sparse retrieval based on term matching can achieve good results. For queries with varied expressions, it is more dependent on capturing their semantic similarity. For example, when searching for "hotel," names containing "hotel," "inn," or "guesthouse" should be recalled. In this case, semantically based dense retrieval would be more advantageous.

[0015] Therefore, the fusion weights of different model retrieval results should be determined based on the characteristics of the query itself. The method for determining the learning objective of the fusion weights is the key problem that this invention aims to solve. Summary of the Invention

[0016] To address the problem in existing retrieval model fusion methods where all queries are fused with the same weight, making it impossible to adaptively determine fusion weights for each query, this invention proposes a method for fusing retrieval results from different retrieval models by generating adaptive fusion weights based on query content. The method includes: performing a sparse retrieval using the retrieval information to obtain sparse retrieval results and corresponding sparse ranking scores; performing a dense retrieval using the retrieval information to obtain dense retrieval results and corresponding dense ranking scores; constructing a weight prediction model to obtain fusion weights based on the retrieval information, the sparse retrieval results, and the dense retrieval results; generating a fusion ranking score using the fusion weights, the sparse ranking scores, and the dense ranking scores; and ranking the sparse and dense retrieval results according to the fusion ranking scores to generate the final retrieval results.

[0017] The semantic retrieval model fusion method of the present invention includes the step of obtaining the fusion weights, which comprises: obtaining the vector representation (emb) of the retrieval information respectively. query Vector representation of the sparse search result and the vector representation of the dense search results emb query , and The concatenated weights are used as input to the weight prediction model and output as the fused weights.

[0018] The semantic retrieval model fusion method of the present invention further includes: obtaining historical retrieval information and its corresponding historical retrieval results; applying sparse retrieval to the historical retrieval information to obtain historical sparse retrieval results and corresponding historical sparse ranking scores; applying dense retrieval to the historical retrieval information to obtain historical dense retrieval results and corresponding historical dense ranking scores; and, based on the historical retrieval information, the historical sparse retrieval results, the historical sparse ranking scores, the historical dense retrieval results, the historical dense ranking scores, and the historical retrieval results, obtaining the fusion weight corresponding to the ranking index having the maximum value as the learning target for training the weight prediction model, and training the weight prediction model; the ranking index includes MRR, nDCG, or MAP.

[0019] The semantic retrieval model fusion method of the present invention includes a multilayer perceptron as the weight prediction model.

[0020] The semantic retrieval model fusion method of the present invention includes the step of generating the fusion ranking score by: normalizing the sparse ranking score and the dense ranking score using Min-Max normalization or Z-Score normalization, and then generating the fusion ranking score.

[0021] This invention also proposes a semantic retrieval model fusion system based on adaptive weights, comprising: a retrieval module for performing preliminary retrieval; wherein, sparse retrieval is performed using the retrieval information to obtain sparse retrieval results and corresponding sparse ranking scores; dense retrieval is performed using the retrieval information to obtain dense retrieval results and corresponding dense ranking scores; a weight acquisition module for obtaining fusion weights based on the retrieval information, the sparse retrieval results, and the dense retrieval results through a weight prediction model; and a result ranking module for generating a fusion ranking score using the fusion weights, the sparse ranking scores, and the dense ranking scores; and ranking the sparse retrieval results and the dense retrieval results according to the fusion ranking scores to generate the final retrieval results.

[0022] The semantic retrieval model fusion system of the present invention includes a weight acquisition module comprising: a weight output module, used to acquire the fusion weight through the weight prediction model; wherein, the vector representation (emb) of the retrieval information is acquired respectively. query Vector representation of the sparse search result and the vector representation of the dense search results emb query , and The concatenated weights are used as input to the weight prediction model and output as the fused weights.

[0023] The semantic retrieval model fusion system of the present invention further includes a weight acquisition module: a model training module for training the weight prediction model; wherein, historical retrieval information and its corresponding historical retrieval results are acquired; sparse retrieval is performed on the historical retrieval information to acquire historical sparse retrieval results and their corresponding historical sparse ranking scores; dense retrieval is performed on the historical retrieval information to acquire historical dense retrieval results and their corresponding historical dense ranking scores; based on the historical retrieval information, the historical sparse retrieval results, the historical sparse ranking scores, the historical dense retrieval results, the historical dense ranking scores, and the historical retrieval results, the fusion weight corresponding to the ranking index having the maximum value is obtained as the learning target for training the weight prediction model, and the weight prediction model is trained; the ranking index includes MRR, nDCG, or MAP.

[0024] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the semantic retrieval model fusion method based on adaptive weights as described above is implemented.

[0025] The present invention also proposes a data processing apparatus, including the computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes the computer-executable instructions in the computer-readable storage medium, it implements a semantic retrieval model fusion method based on adaptive weights. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the semantic retrieval fusion method based on adaptive weights of the present invention.

[0027] Figure 2 This is a schematic diagram of the fusion weight prediction based on vector features of the present invention.

[0028] Figure 3 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0030] While researching the fusion of different types of retrieval models in information retrieval, the inventors discovered that existing retrieval model fusion techniques merely obtain a weight by testing on a validation set and then fuse the results retrieved for all queries across different retrieval systems. They ignored the fact that each query is best suited for a different retrieval system, and crudely used the same weight for fusion.

[0031] The inventors analyzed the results retrieved by existing retrieval systems and discovered that, with existing relevance labeling, a weighted search method can be used to find the fusion weight that maximizes the ranking metrics for each query. Ranking metrics can include MRR, nDCG, and MAP. Furthermore, using adaptive fusion weights resulted in approximately a 10% improvement in various retrieved metrics such as MRR@10, NDCG, and MAP. This demonstrates the importance of using adaptive weights to fuse query results from different retrieval systems for each query. However, in actual retrieval processes, relevance labels are unavailable. Nevertheless, fusion weights for each query in the training set can be obtained based on the relevance labels, allowing the construction of a query content-based fusion weight prediction model. During actual retrieval, the fusion weight can be adaptively determined for each query's content; for example, models adept at handling such queries can receive higher weights. This allows different retrieval models to leverage their strengths and compensate for their weaknesses, improving the final retrieval performance.

[0032] The semantic retrieval model fusion method of this invention adopts the construction of fusion weight learning objectives. For a query, based on existing relevance labels, the fusion weight that maximizes the ranking index is obtained. The ranking index can be MRR, nDCG, MAP, etc. It also adopts fusion weight prediction based on vector representation. During retrieval, the fusion weight is obtained based on the vector representation of the query and the first k retrieved documents. The vector representation directly reuses the vectors in the dense vector retrieval model. The fusion weight is used to fuse the retrieval results of different models, which can further improve the retrieval effect.

[0033] Existing retrieval model fusion methods are mainly based on the scores S of queries and documents in sparse retrieval. sparse (q, d) and the score S of dense retrieval output dense (q, d) are weighted and fused using weight α to obtain the fused score S. (qd) Then sort them. See the formula below.

[0034] S (q，d) =α*S sparse (q, d) + (1-α)*S dense (q, d)

[0035] The semantic retrieval model fusion method based on adaptive weights mainly determines the fusion weights of the semantic retrieval (dense retrieval) model and the term-matching-based retrieval (sparse retrieval) model dynamically based on the characteristics of each query. Its main structure is illustrated as follows: Figure 1 As shown.

[0036] There are two key points in the fusion process. One is the fusion score S. sparse (q, d) and S dense (q, d) Different score sources can significantly impact the final fusion result, mainly including the relevance score output by the model and the inverse of the document's ranking in the query. Simultaneously, the scores need to be normalized using appropriate methods to eliminate the influence of different units of measurement. Secondly, the generation of adaptive fusion weights α requires, firstly, for queries with relevant document annotations in the training set, determining the fusion weight that maximizes the ranking metric for the current query based on the relevance annotations. The ranking metric can be MRR, nDCG, MAP, etc. Then, a model is built to learn the mapping relationship between the query and the weights, so that in actual retrieval, even without knowing the relevance labels, the adaptive weights can be predicted, resulting in better fusion performance.

[0037] Construction of fusion weighted learning objectives

[0038] Different retrieval models output scores with varying numerical ranges. For example, the BM25 score has no upper bound. In dense retrieval, dot product or cosine similarity might be used as the final relevance score. To prevent the influence of inconsistent score dimensions, relevance scores need to be normalized. Min-Max normalization or Z-Score normalization can be used.

[0039]

[0040]

[0041] Min-Max and Z-Score normalization require calculating the maximum, minimum, mean, and variance of the set. There are two ways to partition the set: global and local. Local normalization only normalizes the first N documents (N is usually 1000) within a single retrieval system for each query, treating them as a single set. Global normalization normalizes all recalled documents across all queries within a single retrieval system, treating them as a single set.

[0042] Previous fusion methods for search results were mainly based on global static weights, which means that the same fusion weight α is used for all queries. α is usually a hyperparameter, which is usually obtained by trying on the validation set to maximize the fusion weight of the ranking index. The ranking index can be MRR, nDCG, MAP, etc.

[0043] The fusion weight is determined for each query, given the relevant document lists and scores retrieved in both sparse and dense retrieval systems. The goal is to find the fusion weight that maximizes the ranking metric, which can be MRR, nDCG, MAP, etc. The search method involves calculating the ranking metric (nDCG, MAP, MRR, etc.) of the fused document list for each weight. Weight values ​​are between 0 and 1, and a fixed step size (e.g., 0.01) is used to search for the fusion weight that maximizes the ranking metric. Its formal representation is: for each query... i The fusion weight is α i D Sparse (Query i ) and D ddense (Query i ) represents the query for the i-th query. i The document list obtained through sparse retrieval and dense retrieval.

[0044] α i =argmax α∈[0，1] RankMetric(Merger(D Sparse (Queryi ), D dense (Query i ), α))

[0045] Construction of weighted prediction model

[0046] For each query in the training set, a fusion weight is obtained given that relevant documents are known. This weight is derived from the knowledge of these documents, but in actual retrieval, the correct answer cannot be known in advance to calculate the weight. The goal is to predict the fusion weight using features relevant to the query. This problem is defined as a regression problem. Depending on the type of input features, weight prediction is divided into vector-based feature prediction and text-based feature prediction.

[0047] Weight prediction based on vector features

[0048] Dense retrieval primarily achieves vectorized recall by encoding queries and documents into dense vectors of fixed dimensions. The trained vectors can effectively represent text. Therefore, existing query vector representations from dense retrieval can be directly used to represent queries, saving time and resources spent on text encoding.

[0049] To represent a query, we need more than just the query itself. The top k candidate documents retrieved by the retrieval system are highly relevant to the query. Furthermore, the characteristics of the relevant documents themselves are crucial when merging results from different retrieval systems. Therefore, the vectors of the top k candidate documents retrieved by the retrieval system can also be used as features for weight prediction.

[0050] like Figure 2 As shown, for the query... i The vector representations of the top k documents from both the sparse and dense retrieval methods in the dense retrieval are taken, concatenated with the vector representation of the query, and then input into the model for prediction.

[0051]

[0052]

[0053] The model takes a vector as input and outputs a number between 0 and 1; this task can be accomplished using any regression model. This invention uses a Multilayer Perceptron (MLP) to train the weight predictor. MLPs can handle high-dimensional data and, by incorporating non-linear activation functions, can learn non-linear relationships between outputs.

[0054] Specifically, by acquiring historical retrieval information and its corresponding historical retrieval results, a training set is constructed, and supervised learning is performed on the weight prediction model. This includes: using sparse retrieval on the historical retrieval information to obtain historical sparse retrieval results and corresponding historical sparse ranking scores; using dense retrieval on the historical retrieval information to obtain historical dense retrieval results and corresponding historical dense ranking scores; and based on the obtained historical retrieval information, historical sparse retrieval results, historical sparse ranking scores, historical dense retrieval results, historical dense ranking scores, and historical retrieval results, obtaining the fusion weight corresponding to the ranking index having its maximum value as the learning target for training the weight prediction model, and training the weight prediction model. In embodiments of this invention, the ranking index includes one of MRR, nDCG, or MAP.

[0055] Figure 3 This is a schematic diagram of the data processing apparatus of the present invention. Figure 3 As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, the aforementioned semantic retrieval model fusion method based on adaptive weights is implemented. Those skilled in the art will understand that all or part of the steps in the above method can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by implementing its corresponding function through an integrated circuit, or it can be implemented in the form of a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0056] Compared with existing technologies, this invention can adaptively determine the fusion weights for each query, thereby achieving better fusion results. In terms of efficiency, this invention has lower overhead than traditional re-ranking algorithms, requiring only the prediction of fusion weights. In terms of practicality, this invention can be easily integrated as a plugin into existing retrieval models.

[0057] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A semantic retrieval model fusion method based on adaptive weights, characterized in that, include: Sparse retrieval is performed using the retrieved information to obtain sparse retrieval results and corresponding sparse ranking scores. Perform a dense search using the search information to obtain dense search results and corresponding dense ranking scores; Construct a weight prediction model, and obtain the fusion weight based on the retrieval information, the sparse retrieval results, and the dense retrieval results. ; , For hyperparameters, For the nth query The list of documents obtained through sparse retrieval Representatives targeting The document list is obtained through dense retrieval; historical retrieval information and its corresponding historical retrieval results are obtained; sparse retrieval is applied to the historical retrieval information to obtain historical sparse retrieval results and their corresponding historical sparse ranking scores; dense retrieval is applied to the historical retrieval information to obtain historical dense retrieval results and their corresponding historical dense ranking scores; based on the historical retrieval information, the historical sparse retrieval results, the historical sparse ranking scores, the historical dense retrieval results, the historical dense ranking scores, and the historical retrieval results, the fusion weight corresponding to the ranking index having the maximum value is obtained as the learning objective for training the weight prediction model, and the weight prediction model is trained; the ranking index is MRR, nDCG, or MAP. A fusion ranking score is generated using the fusion weight, the sparse ranking score, and the dense ranking score. Based on the fusion ranking score, the sparse search results and the dense search results are sorted to generate the final search results.

2. The semantic retrieval model fusion method as described in claim 1, characterized in that, The steps to obtain the fusion weights include: Obtain the vector representation of the retrieved information respectively. Vector representation of the sparse search results and the vector representation of the dense search results ,Will , and The concatenated weights are used as input to the weight prediction model and output as the fused weights.

3. The semantic retrieval model fusion method as described in claim 1, characterized in that, The weighted prediction model is a multilayer perceptron.

4. The semantic retrieval model fusion method as described in claim 1, characterized in that, The steps to generate this fusion ranking score include: After normalizing the sparse sort score and the dense sort score using Min-Max normalization or Z-Score normalization, the fused sort score is generated.

5. A semantic retrieval model fusion system based on adaptive weights, characterized in that, include: The retrieval module is used to perform preliminary retrieval; it performs sparse retrieval using the retrieval information to obtain sparse retrieval results and corresponding sparse ranking scores; and it performs dense retrieval using the retrieval information to obtain dense retrieval results and corresponding dense ranking scores. The weight acquisition module is used to obtain the fusion weight based on the search information, the sparse search results, and the dense search results through a weight prediction model. ; , For hyperparameters, For the nth query The list of documents obtained through sparse retrieval Representatives targeting The document list is obtained through dense retrieval; the weight acquisition module includes a model training module for training the weight prediction model; wherein, historical retrieval information and its corresponding historical retrieval results are obtained; sparse retrieval is applied to the historical retrieval information to obtain historical sparse retrieval results and their corresponding historical sparse ranking scores; dense retrieval is applied to the historical retrieval information to obtain historical dense retrieval results and their corresponding historical dense ranking scores; based on the historical retrieval information, the historical sparse retrieval results, the historical sparse ranking scores, the historical dense retrieval results, the historical dense ranking scores, and the historical retrieval results, the fusion weight corresponding to the ranking index having the maximum value is obtained as the learning objective for training the weight prediction model, and the weight prediction model is trained; the ranking index is MRR, nDCG, or MAP. The result sorting module is used to generate a fusion sorting score based on the fusion weight, the sparse sorting score, and the dense sorting score; and to sort the sparse search results and the dense search results according to the fusion sorting score to generate the final search results.

6. The semantic retrieval model fusion system as described in claim 5, characterized in that, The weight acquisition module includes: The weight output module is used to obtain the fusion weight through the weight prediction model; wherein, the vector representation of the retrieval information is obtained respectively. Vector representation of the sparse search results and the vector representation of the dense search results ,Will , and The concatenated weights are used as input to the weight prediction model and output as the fused weights.

7. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the semantic retrieval model fusion method based on adaptive weights as described in any one of claims 1 to 4 is implemented.

8. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 7, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it implements a semantic retrieval model fusion method based on adaptive weights.

Images