Zero sample cross-language reordering method based on large language model, electronic equipment and medium

Through the zero-sample cross-language reordering method based on large language model, documents of different languages ​​are directly reordered, which solves the problems of translation quality limitations and system complexity in the existing technology, and achieves more efficient and accurate cross-language information retrieval Effect.

CN118939757APending Publication Date: 2024-11-12ZHEJIANG UNIV +1

Patent Information

Application Number
CN202410990608.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing cross-language information retrieval methods are limited by translation quality, resulting in information loss and deviation, and the system complexity and computing resource requirements are high, so it is impossible to effectively solve the problem of cross-language information retrieval.

Method used

Using the zero-sample cross-language reordering method based on the large language model, the multi-language query and document data are obtained, the inverted index is established, and the query is decomposed and sliding window strategy is used to directly reorder documents of different languages. Combining query decomposition and sliding window strategies, it handles complex queries and large amounts of documents.

Benefits of technology

Using the powerful semantic understanding ability of the large language model, directly reordering documents in different languages ​​has improved the search effect, reduced the system complexity and computing resource requirements, and achieved more accurate and efficient cross-language information retrieval.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0004958766560000031
    Figure BDA0004958766560000031
  • Figure BDA0004958766560000032
    Figure BDA0004958766560000032
  • Figure BDA0004958766560000033
    Figure BDA0004958766560000033
Patent Text Reader

Abstract

The invention discloses a zero-sample cross-language reordering method based on a large language model, electronic equipment and a medium. The zero-sample cross-language reordering method comprises the steps of obtaining queries corresponding to a plurality of language types and document data corresponding to the plurality of language types; mapping from entries to documents is established for the document data corresponding to each language type, and corresponding inverted indexes are obtained; inputting the query into the inverted index corresponding to the language type to obtain a first rearrangement document list corresponding to each query; for each query, decomposing the query into a plurality of sub-queries, and setting a unique identifier for each document in the first rearrangement document list corresponding to the query; setting a cue word template, inputting the query, the sub-query, the first rearrangement document list, the unique identifier corresponding to each document and the cue word template into a large language model, and outputting a correlation identifier sequence of each document and the query to obtain a second rearrangement document list corresponding to each query.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval, and in particular relates to a zero-sample cross-language reordering method based on a large language model, an electronic device, and a medium. Background Art

[0002] Cross-language information retrieval (CLIR) aims to establish connections between texts in different languages ​​to achieve efficient search and information extraction. Traditional CLIR methods usually rely on translating queries or documents into the target language for retrieval and ranking. However, this method is often limited by the quality of translation, and the conversion process between different languages ​​will cause information loss and deviation, affecting the final retrieval effect. In addition, the independent existence of translation models and retrieval models also brings problems of system complexity and computing resource requirements.

[0003] Existing CLIR methods are mainly divided into three categories: query translation, document translation, and language-independent representation. Query translation methods translate user queries into the target language to match documents; document translation methods translate target documents into the query language; and language-independent representation methods attempt to represent texts in different languages ​​in a common embedding space. Although these methods have alleviated the difficulty of cross-language information retrieval to a certain extent, there are still bottlenecks in translation quality and system integration. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a zero-sample cross-language reordering method based on a large language model, an electronic device, and a medium.

[0005] In a first aspect, an embodiment of the present invention provides a zero-shot cross-language reordering method based on a large language model, the method comprising:

[0006] Get queries corresponding to several languages ​​and document data corresponding to several languages;

[0007] Establish a mapping from terms to documents for the document data corresponding to each language type, and obtain a corresponding inverted index; input a query into the inverted index corresponding to the language type, and obtain a first re-ranked document list corresponding to each query;

[0008] For each query, the query is decomposed into sub-queries, and a unique identifier is set for each document in the first rearranged document list corresponding to the query; a prompt word template is set, and the query, sub-query, first rearranged document list, the unique identifier corresponding to each document, and the prompt word template are input into the large language model, and the relevance identifier ranking of each document and the query is output to obtain a second rearranged document list corresponding to each query.

[0009] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned zero-sample cross-language reordering method based on a large language model.

[0010] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned zero-sample cross-language reordering method based on a large language model.

[0011] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned zero-sample cross-language reordering method based on a large language model.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] The present invention discloses a zero-shot cross-language re-ranking method based on a large language model. The present invention utilizes the powerful semantic understanding ability of the large language model to directly re-rank documents in different languages. By combining query decomposition and sliding window strategies, the large language model can handle complex queries and ultra-long ranking requests consisting of a large number of documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0015] Figure 1 A flowchart of a zero-shot cross-language reordering method based on a large language model provided by an embodiment of the present invention;

[0016] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0019] like Figure 1 As shown, an embodiment of the present invention provides a zero-sample cross-language reordering method based on a large language model, and the method specifically includes the following steps:

[0020] Step S1, obtaining queries corresponding to several languages ​​and document data corresponding to several languages.

[0021] Specifically, in this example, the NeuCLIR public dataset is used as a corpus, and English queries in the NeuCLIR public dataset are translated into queries corresponding to several languages ​​including Chinese, Russian, and Persian queries, thereby constructing a query set.

[0022] The process of preprocessing the document data includes: limiting the length of each document to within the first 500 characters. At the same time, the garbled symbols and stop words in the query and document data are deleted.

[0023] Step S2, establish a mapping from terms to documents for the document data corresponding to each language type, and obtain a corresponding inverted index; input a query into the inverted index corresponding to the language type, and obtain a first re-ranked document list corresponding to each query.

[0024] Specifically, in this example, the Pyserini toolkit is used to process the document data corresponding to each language type, establish a mapping from terms to documents, and create an inverted index corresponding to the document data corresponding to each language type. Each inverted index records the position and frequency of each word in the document.

[0025] The BM25 algorithm is used to calculate the matching score between each document and the query, and each query is set to return a list of the top N documents with the highest BM25 scores. In this example, N is 100, which is the first re-ranked document list for the current query.

[0026] The BM25 algorithm is implemented based on term frequency (TF) and inverse document frequency (IDF), and the expression is as follows:

[0027] TF represents the frequency of a word in a document, which is usually calculated as the number of times the word appears in the document divided by the total number of words in the document. As shown in the formula:

[0028]

[0029] Among them, f t,d represents the number of times word t appears in document d, t′ represents each word in document d, ∑ t′∈ d ft′,d Represents the total number of times all words appear in document d.

[0030] IDF measures the rarity of a word in the entire corpus. As shown in the formula:

[0031]

[0032] Where N is the total number of documents in the corpus, n t is the number of documents containing the term t.

[0033] BM25 combines TF and IDF, allowing the information retrieval model to better balance word frequency and word importance, thereby improving the accuracy of document and query matching. As shown in the formula:

[0034]

[0035] In the formula, k1 is a parameter for adjusting the saturation degree of word frequency, which usually takes a value between 1.2 and 2.0, and is used to control the influence of word frequency on the final score; b is a parameter for adjusting the influence of document length on word frequency, which takes a value between 0 and 1, usually 0.75, and is used to control the normalized influence of document length on word frequency; avgdl is the average length of all documents in the document collection.

[0036] Step S3, for each query, decompose the query into several sub-queries, set a unique identifier for each document in the first rearranged document list corresponding to the query; set a prompt word template, input the query, sub-query, the first rearranged document list, the unique identifier corresponding to each document, and the prompt word template into the large language model, output the relevance identifier ranking of each document and the query, and obtain the second rearranged document list corresponding to each query.

[0037] Step S301, decomposing the query into several sub-queries.

[0038] In this example, a prompt word template corresponding to query decomposition is set, and the prompt word template corresponding to query decomposition and the query to be decomposed are input into the large language model, and parsed by the output parser (RAG Fusion) to obtain several sub-queries after decomposition.

[0039] For example, the prompt word template corresponding to the query decomposition is shown in Table 1 below:

[0040] Table 1: Prompt word template corresponding to query decomposition

[0041]

[0042] Step S302, setting a unique identifier for each document in the first rearranged document list corresponding to the query;

[0043] Specifically, for each query q, the large language model LLM provides a first re-ranked document list, each document in the first re-ranked document list has a corresponding unique identifier (e.g., [1], [2], etc.). In this example, the LLMs are required to generate a ranking order of documents based on their relevance to the query. Documents are sorted using identifiers in a format such as [2]>[3]>[1]>, etc. This example directly ranks documents without generating intermediate relevance scores.

[0044] Step S303: setting a prompt word template.

[0045] Exemplarily, in this example, a prompt word template is set according to the query, the subquery, the first rearranged document list, and the unique identifier corresponding to each document; as shown in Table 2 below:

[0046] Table 2: Input prompts and generated completion examples

[0047]

[0048]

[0049] Furthermore, in this example, external knowledge information can also be input into the large language model; for example, the content of resources such as PDF documents or professional books can be encoded into prompt information that LLMs can understand for reference during re-ranking. This includes converting key concepts, terms, and common problems or solutions related to the field into a form that the model can understand, and incorporating them into the input prompts. At the same time, this example also obtains more extensive and real-time information through web crawlers, especially crawling websites, forums, or news in a specific language to obtain the latest information and context in the language environment. These data are integrated into the input prompts to help LLMs better understand search queries in a specific language environment, thereby providing more accurate and relevant re-ranking results.

[0050] The prompt word template for introducing external knowledge information can be set as shown in Table 3 below.

[0051] Table 3: Input prompts and generated completion examples

[0052]

[0053] Step S304, input the query, subquery, first rearranged document list, unique identifier corresponding to each document, and prompt word template into the large language model, output the relevance identifier ranking of each document and the query, and obtain the second rearranged document list corresponding to each query.

[0054] In this example, a sliding window is used to process the first re-ranked document list, which is input into a large language model in batches for re-ranking; including:

[0055] The number of documents in the first rearranged document list is recorded as M;

[0056] Set the window size w and step size s;

[0057] Rank the documents from (Mw)th to Mth using the large language model; then, slide the window with a step size s and re-rank the documents in the range from (Mws)th to (Ms);

[0058] And so on, until all documents are re-ranked.

[0059] Exemplarily, in this example, the window size w is set to 4, and the step size s is set to 2. The document lists P1, P2, P3, P4, P5, P6, P7, and P8 are sorted.

[0060] The first sorting: P1, P2, P3, P4, {P5, P6, P7, P8}. Sort {P5, P6, P7, P8} to get P1, P2, P3, P4, P8, P5, P6, P7.

[0061] Second sorting: P1, P2, {P3, P4, P8, P5}, P6, P7. Sort {P3, P4, P8, P5} to get P1, P2, P8, P3, P4, P5, P6, P7.

[0062] The third sorting: {P1, P2, P8, P3}, P4, P5, P6, P7. Sorting {P1, P2, P8, P3}, we get P2, P8, P1, P3, P4, P5, P6, P7.

[0063] In this example, DCG and NDCG indicators are used to measure the quality of the results of the annotation sorting. The calculation of DCG includes three parts: accumulation process, gain calculation and position discount. The accumulated content is the gain, and this gain value will be multiplied by a position discount. The later the position, the greater the discount and the smaller the overall score. Finally, the product results of each item are added up to get the DCG value. In order to better compare different sorting results, DCG is normalized to obtain NDCG (normalized discounted cumulative gain). NDCG makes it easier to compare the results of different sortings, even if the number of documents is different. As shown in the formula:

[0064]

[0065] Among them, rel i represents the relevance score of the ith result. p is the first p results.

[0066]

[0067] in, is the relevance score of the result at the ith position under ideal conditions.

[0068] For text rearrangement results, recall is a very common evaluation method. Recall measures the proportion of all relevant documents that the model correctly finds. It is the ratio of true positives (retrieved relevant documents) to all actual relevant documents. As shown in the formula:

[0069]

[0070] Among them: TP (True Positives) is the number of documents correctly identified as relevant by the retrieval system, and FN (False Negatives) is the number of documents that are actually relevant but not identified as relevant by the retrieval system.

[0071] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned zero-sample cross-language reordering method based on a large language model. Figure 2 As shown, a hardware structure diagram of any device with data processing capability in which the zero-sample cross-language reordering method based on a large language model provided by an embodiment of the present invention is located, except Figure 2 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0072] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the zero-sample cross-language reordering method based on a large language model as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0073] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A zero-shot cross-language re-ranking method based on a large language model, characterized in that: The method comprises: Get queries corresponding to several languages ​​and document data corresponding to several languages; Establish a mapping from terms to documents for the document data corresponding to each language type, and obtain a corresponding inverted index; input a query into the inverted index corresponding to the language type, and obtain a first re-ranked document list corresponding to each query; For each query, the query is decomposed into several sub-queries, and a unique identifier is set for each document in the first rearranged document list corresponding to the query; a prompt word template is set, and the query, sub-query, first rearranged document list, the unique identifier corresponding to each document, and the prompt word template are input into the large language model, and the relevance identifier ranking of each document and the query is output to obtain a second rearranged document list corresponding to each query.

2. The zero-shot cross-language re-ranking method based on a large language model according to claim 1, characterized in that: Get the query corresponding to several languages. The document data corresponding to several languages ​​also includes: Delete garbled symbols and stop words in query and document data; The length of the document data corresponding to each language type is limited, and the first K characters of each document data are taken.

3. The zero-shot cross-language re-ranking method based on a large language model according to claim 1, characterized in that: The process of inputting a query into the inverted index corresponding to the language type and obtaining a first re-ranked document list corresponding to each query includes: Use the BM25 algorithm to calculate the matching score between each document and the query, and set each query to return the top N document lists with the highest BM25 scores as the first re-ranked document list for the current query; The calculation process of BM25 score is as follows: In the formula, f t,d represents the number of times word t appears in document d, t′ represents each word in document d, ∑ t′∈d f t′,d Represents the total number of times all words appear in document d; Where N is the total number of documents, n t is the number of documents containing word t; Wherein, k1 is a parameter for adjusting the saturation degree of word frequency, which is used to control the influence of word frequency on the final score; b is a parameter for adjusting the influence of document length on word frequency, which is used to control the normalized influence of document length on word frequency; avgdl is the average length of all documents in the document collection.

4. The zero-shot cross-language re-ranking method based on a large language model according to claim 3, characterized in that: The value of k1 is 1.2~2.0; the value of b is 0~1.

5. The zero-shot cross-language re-ranking method based on a large language model according to claim 1, characterized in that: The method further comprises: The first re-ranked document list is processed using a sliding window and input into the large language model in batches for re-ranking; including: The number of documents in the first rearranged document list is recorded as M; Set the window size w and step size s; Rank the documents from (Mw)th to Mth using the large language model; then, slide the window with a step size s and re-rank the documents in the range from (Mws)th to (Ms); And so on, until all documents are re-ranked.

6. The zero-shot cross-language re-ranking method based on a large language model according to claim 1, characterized in that: The prompt word template is set as: You are the ranking model, an intelligent assistant that ranks documents by relevance based on a query. I will provide you with {number} documents, each with an ID identifier []. Sort these documents by relevance based on the query: {query, first subquery, second subquery,,,}. [1]{Document 1} [2]{Document 2} ... [Quantity]{Number of documents} Search query: {query} Please sort the {number} documents above by relevance to your query. Documents should be sorted in descending order of relevance, using the identifier. The most relevant document should be at the top. The output format should be []>[], for example, [1]>[2]. Just respond to the sort result without explaining or saying anything else.

7. The zero-shot cross-language re-ranking method based on a large language model according to claim 1 or 6, characterized in that: The method further comprises: Input external knowledge information into the large language model; set the prompt word template to: You are the ranking model, an intelligent assistant that ranks documents by relevance based on a query. I will provide you with {number} documents, each with an ID identifier []. According to the query: {query, first subquery, second subquery,,,}, refer to {external knowledge information}, and sort these documents by relevance. [1]{Document 1} [2]{Document 2} ... [Quantity]{Number of documents} Search query: {query} Please sort the {number} documents above by relevance to your query. Documents should be sorted in descending order of relevance, using the identifier. The most relevant document should be at the top. The output format should be []>[], for example, [1]>[2]. Just respond to the sort result without explaining or saying anything else.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the zero-sample cross-language reordering method based on a large language model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the zero-sample cross-language reordering method based on a large language model is implemented as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the zero-sample cross-language reordering method based on a large language model described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Language model-based sorting algorithm

    CN101826075A

  • Cross-language information retrieval method based on bilingual dictionary mapping guidance

    CN113076398A

  • Natural language processing system and method based on large language model

    CN117251551A

  • Cross-lingual search re-ranking

    US20090222437A1

  • Cross-lingual language models and pretraining of cross-lingual language models

    US20220318255A1

Cited By

  • Data reordering retrieval method and system based on RAG

    CN120086307A