A query representation enhancement method in cross-language dense retrieval

Abstract

The application provides a query representation enhancement method in cross-language dense retrieval, comprising the following steps: collecting a source language query input by a user and corresponding query results, i.e., target language documents clicked by the user in target language candidate documents, from a search log of a cross-language retrieval system to construct labeled data; retrieving the first K queries with the highest similarity scores with the source language query input by the user, taking K relevant documents corresponding to the first K queries as feedback documents; training a cross-language dense retrieval model, obtaining dense representation vectors according to the K feedback documents and the source language query input by the user, and fusing the dense representation vectors to obtain an enhanced source language query representation; calculating the similarity between the query and the document; sorting the target language candidate documents to output the sorted results as final retrieval results.

Description

Technical Field

[0001] This invention relates to a query representation enhancement method in cross-language dense retrieval, and more particularly to a query representation enhancement method in cross-language dense retrieval. Background Technology

[0002] With the rapid global development of internet applications, an increasing number of applications need to provide cross-language services, such as video platforms serving users in multiple countries, overseas shopping platforms, and multilingual news websites. Overseas users may need to use their native language to search for resources in another language. Therefore, cross-language retrieval technology has become crucial. Compared to traditional sparse matching-based retrieval methods, dense vector retrieval can map text from different languages ​​into the same representation space, allowing it to break free from its dependence on translation models. Therefore, dense vector retrieval is a key technology in cross-language retrieval.

[0003] Dense vector retrieval generates fixed-dimensional representation vectors for queries and documents, and ranks documents by calculating the similarity between representations during retrieval. Although dense vector retrieval models can map queries and documents in different languages ​​to the same vector space, spatial distribution differences in the generated representation vectors still exist between languages ​​because queries and documents use different languages ​​and vocabularies.

[0004] In cross-language information retrieval, methods based on pseudo-relevance feedback typically require obtaining relevant feedback documents in a preliminary search. The language and vocabulary mismatch between the source language query and the target language documents leads to poor results in this initial search. This results in a large number of documents in the initial search that are irrelevant to the original query. Modifying the query using these irrelevant feedback documents further deviates the semantics or representation of the query from the original query, ultimately negatively impacting the performance of the retrieval system. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a query representation enhancement method in cross-language dense retrieval, which addresses the shortcomings of the prior art.

[0006] To address the aforementioned technical problems, this invention discloses a query representation enhancement method for cross-language dense retrieval, comprising the following steps:

[0007] Step 1: Collect the source language query input by the user and the corresponding query results, i.e. the target language candidate documents clicked by the user, from the search logs of the cross-language retrieval system, and construct labeled data.

[0008] Step 2: From the labeled data, retrieve the top K queries with the highest similarity scores to the source language query input by the user, and use the K relevant documents corresponding to the top K queries as feedback documents.

[0009] Step 3: Train a cross-language dense retrieval model using labeled data, and obtain dense representation vectors based on K feedback documents and the source language query input by the user, and fuse them to obtain the enhanced source language query representation;

[0010] Step 4: Calculate the dot product of the enhanced source language query representation and the target language candidate document representation vector as the similarity score to obtain the similarity score between the query and the document.

[0011] Step 5: Use the similarity between the query and the document calculated in Step 4 to sort the candidate documents in the target language, and output the sorted results as the final search results.

[0012] Beneficial effects:

[0013] From a technical perspective, the technical solution of this invention (1) utilizes the input source language query to retrieve other similar source language queries, and obtains feedback documents from the relevant target language documents annotated by the retrieved similar queries. This process replaces the pre-round cross-language retrieval of target language documents in the traditional PRF method. This can avoid problems such as word mismatch and semantic drift encountered in the initial retrieval of target language documents by source language queries, thereby obtaining higher quality relevant feedback documents. (2) A query representation enhancement method based on weighted fusion of representation vectors is used to narrow the distance between source language queries and relevant target language documents in the representation space, thereby improving the results of cross-language dense retrieval.

[0014] From an application perspective, the technical solutions of this invention (1) can utilize single-language retrieval of similar queries, eliminating the reliance on translation resources for obtaining relevant feedback documents in cross-language retrieval, and lowering the application threshold of cross-language retrieval systems. (2) By using relevant documents labeled with queries similar to the original query as high-quality feedback text and enhancing the query representation, users can retrieve target language documents that are more relevant to the query, thus improving the multilingual user experience. This has good application prospects in multilingual web applications and cross-language search engines. (3) By using vector weighted fusion to modify the query representation, there is no need to retrain the dense retrieval model, which reduces the complexity of retrieval system construction while improving retrieval results. Attached Figure Description

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0016] Figure 1 This is a schematic diagram of the execution flow of a cross-language dense retrieval model.

[0017] Figure 2 This is a flowchart illustrating the process of retrieving feedback documents using similarity queries.

[0018] Figure 3 This is a schematic diagram illustrating the process of obtaining the representation vectors of the candidate document set for the target language. Detailed Implementation

[0019] This invention utilizes a monolingual retrieval method. First, it obtains queries similar to the original query from the labeled <source language query, target language document> data. Second, it takes the target language documents corresponding to the similar queries obtained in the initial retrieval as feedback documents, and uses the dense representation of the feedback documents to modify the dense representation of the input source language query. Finally, it uses the modified enhanced query representation vector to re-retrieve the candidate set of target language documents and outputs the retrieval results.

[0020] This invention proposes a query representation enhancement method in cross-language dense retrieval, such as... Figure 1 As shown, it includes the following steps:

[0021] Step 1: Collect the source language query input by the user and the corresponding query results, i.e. the target language candidate documents clicked by the user, from the search logs of the cross-language retrieval system, and construct labeled data.

[0022] Step 2: From the labeled data, retrieve the top K queries with the highest similarity scores to the source language query input by the user, and use the K relevant documents corresponding to the top K queries as feedback documents.

[0023] Step 3: Train a cross-language dense retrieval model using labeled data, and obtain dense representation vectors based on K feedback documents and the source language query input by the user, and fuse them to obtain the enhanced source language query representation;

[0024] The cross-language dense retrieval model described above uses the multilingual pre-trained model XLM-R as the encoder.

[0025] The enhanced source language query representation is obtained by inputting K feedback documents and the original source language query into the cross-language dense retrieval model, obtaining dense representation vectors for the K feedback documents and the original source language query respectively, and fusing the obtained dense representation vectors to obtain the enhanced source language query representation, specifically including:

[0026] Step 3-1, obtain the dense representation vector, the specific methods include:

[0027] The encoder in the cross-language dense retrieval model encodes the K feedback documents and the original source language query respectively, resulting in a dense representation vector of the original source language query and a dense representation vector of the K feedback documents.

[0028] Step 3-2, fuse dense representation vectors, specifically including:

[0029] The aggregated feedback document representation vector is obtained by summing the mean of the dense representation vectors of the K feedback documents; the expanded query representation V is obtained by weighted summing of the dense representation vector of the user-input source language query and the aggregated feedback document representation vectors. q_new Specifically, it is expressed as follows:

[0030] V q_new =α*V q +(1-α)*V d_agg

[0031] Where α represents the weights of vector fusion, V q V represents the dense representation vector of the source language query. d_agg This represents the representation vector of the aggregated feedback document.

[0032] Step 4: Calculate the dot product of the enhanced source language query representation and the target language candidate document representation vector as the similarity score to obtain the similarity score between the query and the document.

[0033] The representation vector of the target language candidate document is obtained by encoding using an encoder in a cross-language dense retrieval model.

[0034] The representation vectors of the candidate documents in the target language are reused in subsequent searches, and the representation vectors of the candidate documents are calculated only once offline.

[0035] Step 5: Use the similarity between the query and the document calculated in Step 4 to sort the candidate documents in the target language, and output the sorted results as the final search results.

[0036] The number of searches, K, mentioned in step 2 can be adjusted in practical applications.

[0037] The weight α for vector fusion described in step 3-2 is adjusted in practical applications.

[0038] Step 2 involves retrieving the top K queries with the highest similarity scores to the source language query input by the user, using the BM25 method.

[0039] Example:

[0040] This invention proposes a query representation enhancement method for cross-language dense retrieval. It obtains relevant documents corresponding to similar queries as feedback documents by retrieving similar queries; aggregates the dense representation vectors of the feedback documents and weightedly fuses them with the initial query vector to enhance the dense representation of the source language query. The specific process is as follows: Figure 1 As shown, it includes the following steps:

[0041] Step 101, Input source language query, which refers to the query text written in the source language. The goal of this retrieval system is to retrieve results related to the source language query from the target language candidate document set.

[0042] Step 102: From the labeled <source language query - target language document> data, use the BM25 method to retrieve the top K queries most similar to the original query, and use the K relevant documents corresponding to the retrieved top K queries as feedback documents. The BM25 method is a common monolingual retrieval method in the field of information retrieval, which calculates the similarity between texts based on word frequency information. Furthermore, the value of K can be determined after validation on a validation set in a real-world application scenario. The labeled data is constructed from the search logs of the cross-language retrieval system by collecting the source language queries entered by the user and the corresponding target language documents clicked by the user.

[0043] Step 103: Input K target language feedback documents and the initial source language query. Utilize the cross-lingual dense retrieval model to obtain the corresponding dense representation vectors for each language, and fuse them to obtain the enhanced source language query representation. The cross-lingual dense retrieval model uses the multilingual pre-trained model XLM-R as the cross-lingual encoder (reference: Unsupervised Cross-lingual Representation Learning at Scale. https: / / arxiv.org / pdf / 1911.02116.pdf). XLM-R is a Transformer-based multilingual pre-trained model that can map text from different languages ​​to the same representation space. The first token input into the encoder is taken. <s>The output of the last layer of the model is used as the text representation vector output by the encoder. The similarity calculation function is defined as follows:

[0044] Sim(q,d) = dot(Encoder) Q (q),Encoder D (d))

[0045] Encoder Q With Encoder D These are the query encoder and document encoder, respectively, with `dot` representing the dot product function. During training, positive samples are documents related to the query from the labeled data, while negative samples are constructed by randomly selecting one negative sample from the target language candidate document set for each query's corresponding positive sample using a random negative sampling method. Similar to the Dense Passage Retrieval (DPR) model in monolingual retrieval (see: Dense PassageRetrieval for Open-Domain Question Answering. https: / / arxiv.org / pdf / 2004.04906v3.pdf), this method uses in-batch negative sampling to increase negative samples and calculates the negative log-likelihood loss to update the model parameters during backpropagation. The specific form of the loss function is:

[0046]

[0047] Where, q i For the source language query input, d + With d - These are positive and negative samples, respectively, where n is the total number of negative samples in a training batch, including negative samples randomly sampled from the document set and negative samples obtained through the in-batch negative sampling method. The specific execution flow of this step is as follows: Figure 2 As shown:

[0048] Step 301: Input K target language feedback documents and the source language query entered by the user.

[0049] Step 302: The representation vector V of the source language query is obtained by encoding the code of the cross-language dense retrieval model. q and the representation vectors {V} of K target language feedback documents d1 V d2 ,…,V dK }

[0050] Step 303: Sum the mean values ​​of the representation vectors of the K feedback documents to obtain an aggregated feedback document representation vector, i.e., V. d_agg =Average(V d1 V d2 ,…,V dK ), where V d_agg Let V be the representation vector of the aggregated feedback document. The expanded query representation is obtained by weighted summing of the representation vector of the source language query and the representation vector of the aggregated feedback document. This process can be formalized as: V q_new =α*V q +(1-α)*V d_agg , where V q_new V is the modified query representation vector. q The representation vector of the source language query input by the user. The weight α of the vector fusion can be adjusted to an appropriate value after validation on a validation set in a real-world application.

[0051] Step 104: Calculate the dot product similarity between the fused query representation and the representation vectors of the target language candidate documents. The process for obtaining the representation vectors of the target language candidate document set is as follows: Figure 3 As shown:

[0052] Step 401: Input candidate documents in the target language.

[0053] Step 402: The encoder of the cross-language dense retrieval model is used to encode the representation vector of the candidate documents in the target language.

[0054] Step 403: Output the representation vectors of the candidate documents in the target language. The representation vectors obtained in this step can be reused in subsequent searches, and the representation vectors of all candidate documents only need to be calculated offline once.

[0055] Step 105: Sort the candidate documents in descending order using the calculated similarity between the query and the document, and output the sorted results as the final search results.

[0056] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a query representation enhancement method for cross-language dense retrieval, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0057] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0058] This invention provides an idea and method for query representation enhancement in cross-language dense retrieval. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.< / s>

Claims

1. A query representation enhancement method in cross-language dense retrieval, characterized in that, Includes the following steps: Step 1: Collect the source language query input by the user and the corresponding query results, i.e. the target language candidate documents clicked by the user, from the search logs of the cross-language retrieval system, and construct labeled data. Step 2: From the labeled data, retrieve the top K queries with the highest similarity scores to the source language query input by the user, and use the K relevant documents corresponding to the top K queries as feedback documents. Step 3: Train a cross-language dense retrieval model using labeled data, and obtain dense representation vectors based on K feedback documents and the source language query input by the user, and fuse them to obtain the enhanced source language query representation; Step 4: Calculate the dot product of the enhanced source language query representation and the target language candidate document representation vector as the similarity score to obtain the similarity score between the query and the document. Step 5: Use the similarity between the query and the document calculated in Step 4 to sort the candidate documents in the target language, and output the sorted results as the final search results. In step 3, obtaining the enhanced source language query representation involves inputting K feedback documents and the original source language query into the cross-language dense retrieval model, obtaining dense representation vectors for the K feedback documents and the original source language query respectively, and fusing the obtained dense representation vectors to obtain the enhanced source language query representation. Specifically, this includes: Step 3-1, obtain the dense representation vector, the specific methods include: The encoder in the cross-language dense retrieval model encodes the K feedback documents and the original source language query respectively, resulting in a dense representation vector of the original source language query and a dense representation vector of the K feedback documents. Step 3-2, fuse dense representation vectors, specifically including: The aggregated feedback document representation vector is obtained by summing the mean of the dense representation vectors of the K feedback documents; the expanded query representation is obtained by weighted summing of the dense representation vector of the user-input source language query and the aggregated feedback document representation vectors. Specifically, it is expressed as follows: ； in, The weights represent the vector fusion weights. A dense representation vector representing the source language query. This represents the representation vector of the aggregated feedback document.

2. The query representation enhancement method in cross-language dense retrieval according to claim 1, characterized in that, The representation vector of the target language candidate document mentioned in step 4 is obtained by encoding using an encoder in a cross-language dense retrieval model.

3. The query representation enhancement method in cross-language dense retrieval according to claim 2, characterized in that, The representation vectors of the target language candidate documents described in step 4 are reused in subsequent searches, and the representation vectors of the candidate documents are calculated only once offline.

4. The query representation enhancement method in cross-language dense retrieval according to claim 3, characterized in that, The number of searches, K, mentioned in step 2 can be adjusted in practical applications.

5. The query representation enhancement method in cross-language dense retrieval according to claim 4, characterized in that, The cross-language dense retrieval model described in step 3 uses the multilingual pre-trained model XLM-R as the encoder.

6. The query representation enhancement method in cross-language dense retrieval according to claim 5, characterized in that, The weights for vector fusion described in step 3-2 Adjustments are made in practical applications.

7. The query representation enhancement method in cross-language dense retrieval according to claim 6, characterized in that, Step 2 involves retrieving the top K queries with the highest similarity scores to the source language query input by the user, using the BM25 method.

Images