Multilingual generative retrieval method based on cross-language semantic compression

Abstract

This invention relates to a multilingual generative retrieval method based on cross-language semantic compression, belonging to the field of information retrieval technology. The invention includes the following steps: constructing a multilingual document retrieval dataset; extracting keywords from multilingual documents from multiple perspectives using a keyword extraction model, and calculating the extracted keywords using semantic similarity to construct a similarity matrix; performing semantic clustering based on the similarity matrix, representing clusters using atomic IDs, and then assigning document identifiers to each multilingual document by the cluster containing the keywords; in the inference stage, after inputting a query, employing a dynamic multi-complement constraint decoding method, gradually narrowing the decoding range of the document identifier in the current step based on the decoding results of previous steps, thereby obtaining the final document identifier. The retrieval capability of this invention is significantly improved compared to other models.

Description

Technical Field

[0001] This invention relates to a multilingual generative retrieval method based on cross-language semantic compression, belonging to the field of information retrieval technology. Background Technology

[0002] Multilingual Information Retrieval (MIR) aims to achieve semantic alignment between different languages, enabling users to access information resources across languages. The core of this task lies in establishing semantic connections between queries and documents through language modeling or representation learning, even when they are in different languages. Early MIR primarily relied on translation techniques for cross-language retrieval. These methods typically involved translating queries into the target language, documents into the source language, or translating both simultaneously. Translation could mitigate semantic differences caused by language inconsistencies to some extent, thus balancing retrieval efficiency and accuracy in practical systems. With the rise of multilingual pre-trained language models, such as mBERT and XLM-R, the research paradigm of MIR has shifted significantly. These models possess the ability to uniformly model multilingual text, making vector representations mainstream. New-generation vector retrieval methods no longer rely on traditional translation steps; instead, they encode queries and documents into vector representations in the same semantic space and then perform retrieval by calculating the similarity between vectors. However, despite the advancements in multilingual information retrieval driven by vectorization methods, they still face a number of challenges. Most current methods still follow a fixed "encode-match-rank" process, which lacks an end-to-end overall optimization mechanism, limiting the model's performance in real-world tasks. Furthermore, many methods rely on contrastive learning for training, which typically requires a large amount of high-quality, cross-lingual parallel data—resources that are extremely scarce in many languages.

[0003] Generative Information Retrieval (GIR), as an emerging information retrieval paradigm, differs from traditional retrieval methods that rely on encoding and similarity matching. GIR fully leverages the powerful memory capabilities of pre-trained language models, enabling the direct generation of Document Identifiers (DocIDs) during the inference phase, thus achieving an end-to-end retrieval process. Currently, document identifier representations can be broadly categorized into two types: atomic DocIDs and string DocIDs. Atomic DocIDs typically use randomly assigned identifiers or construct document representations based on document clustering embedding layers. This approach pursues compact and efficient indexing but lacks semantic interpretability. To enhance semantic expression, some studies have introduced progressive training strategies to learn discrete semantic representations, allowing document vectors to better capture content features. In contrast, string DocIDs use semantically meaningful text as document identifiers, such as document titles and keywords. This type of method not only improves the interpretability of document representations but also facilitates generative inference using existing language knowledge. While GIR has made significant progress in monolingual environments, most existing methods are limited to English or other monolingual corpora, and their adaptability to multilingual scenarios remains insufficient. Faced with the challenges of semantic diversity and cross-linguistic inconsistencies in multilingual environments, existing models often struggle to be directly transferred, resulting in significantly reduced retrieval performance.

[0004] Therefore, in order to address this challenge, this invention proposes a multilingual generative retrieval method based on cross-lingual semantic compression, which aims to map the lexical representations of different languages ​​into a shared semantic space, thereby achieving effective modeling of multilingual GIRs. Summary of the Invention

[0005] The technical problem solved by this invention is that it provides a multilingual generative retrieval method based on cross-language semantic compression. Keywords of documents in different languages ​​are clustered through semantic similarity and mapped to atomic IDs with the same semantic meaning. These atomic IDs are then assigned to documents as document identifiers. In the inference stage, a dynamic multi-step constraint decoding method is used to gradually narrow the decoding range, thereby obtaining the final document identifier and improving the ability to retrieve multilingual information.

[0006] The technical solution of this invention is: a multilingual generative retrieval method based on cross-lingual semantic compression, the method comprising:

[0007] Step 1: Construct a multilingual document retrieval dataset;

[0008] Step 2: Extract keywords from multilingual documents from multiple perspectives using a keyword extraction model, and use semantic similarity to calculate the extracted keywords and construct a similarity matrix;

[0009] Step 3: Perform semantic clustering based on the similarity matrix, and represent the clusters using atomic IDs. Then, assign a document identifier to each multilingual document based on the cluster where the keyword belongs.

[0010] Step 4: In the reasoning stage, after inputting the query, a dynamic multi-complement constraint decoding method is adopted. Based on the decoding results of the previous steps, the decoding range of the document identifier in the current step is gradually narrowed to obtain the final document identifier.

[0011] Furthermore, in Step 1, the multilingual document retrieval dataset is derived from Wikipedia data and related web page data. By extracting titles and multiple parts of content from the web pages and filtering out invalid information, title-content pairs are formed, thereby obtaining the multilingual document retrieval dataset.

[0012] Further, Step 1 includes:

[0013] Step 1.1: Automatically obtain Wikipedia webpage content using web scraping technology. Write a Python web scraping program using the Scarpy framework in web scraping technology. By modifying the language field of the Wikipedia URL, obtain documents in different languages.

[0014] Step 1.2: Use code to construct and send a fake web page request, injecting key information including user agent and access key into the request header, thereby successfully establishing a secure connection with the target website server;

[0015] Step 1.3: After obtaining the webpage source code, use the lxml.etree module in Python to parse the page structure; then, use XPath path extraction technology to locate the anchor points of the required information for accurate collection of target text content in the page.

[0016] Step 1.4: Finally, construct title-content information pairs from the obtained target text content to obtain a multilingual document retrieval dataset.

[0017] Furthermore, Step 2 includes:

[0018] Using a prompt-based large language model (LLM) as a keyword extraction model, a fixed number of keywords are extracted from each multilingual document in the multilingual document collection as a semantically compact representation of the multilingual document content.

[0019] The extracted keyword set has a unified format across all languages;

[0020] Then, the keyword sets of all multilingual documents are merged into a global keyword set;

[0021] Each keyword is encoded into a dense vector using a pre-trained text encoder, and cosine similarity is calculated to construct a similarity matrix between keywords for subsequent semantic modeling.

[0022] Furthermore, Step 2 specifically includes the following steps:

[0023] Step 2.1: Use a prompt-based Large Language Model (LLM) to process a collection of multilingual documents D = {d1, d2, ..., d...} i Extract each multilingual document d from ,…} i Keywords; these keywords serve as a semantically compact representation of the content of multilingual documents;

[0024] Step 2.2: Extract a fixed number of m keywords from each multilingual document to obtain the keyword set K of the multilingual document. i The extraction process is represented as follows:

[0025]

[0026] in, Indicates a multilingual document d i The m-th keyword in;

[0027] The keyword extraction process is performed in a standardized manner across all multilingual documents;

[0028] The set of keywords K for all multilingual documents i Merge into a global keyword set K;

[0029]

[0030] The global keyword set K contains n = |K| unique keywords, denoted as K.

[0031] N represents the number of multilingual documents;

[0032] Step 2.4: Use a pre-trained text encoder to encode each keyword into a dense vector v. i ∈R d Calculate the cosine similarity between any two keywords, and construct a keyword similarity matrix S∈R based on the cosine similarity scores. n×n Dense vector v i v j cosine similarity S ij The calculation formula is as follows:

[0033]

[0034] Among them, d is the dimension of the dense vector.

[0035] Furthermore, Step 3 includes: setting a similarity threshold according to the similarity matrix, clustering the pairwise keywords with similarity scores greater than the similarity threshold, those less than the similarity threshold are each a separate cluster, using atomic IDs to represent each cluster, and assigning document identifiers to the multilingual documents according to the clusters where the keywords are located.

[0036] Furthermore, the specific steps of Step 3 include:

[0037] Step 3.1: According to the similarity matrix S of the keywords constructed in Step 2, when there is a path between any two keywords and the similarity of all adjacent keywords on the path satisfies S ij ≥θ, cluster these keywords into one cluster, where the similarity threshold θ ∈ [0, 1]; if the similarity of a certain keyword with any other keyword is lower than the similarity threshold, form an independent cluster;

[0038] Step 3.2: For the clustering results of all keywords, assign a unique atomic identifier, that is, an atomic ID, to each clustering cluster, and obtain the atomic set A = {a1, a2, …, a C}; an atom represents a group of semantically similar keywords, and the number of clusters C satisfies C < N, that is, the number of clusters C is less than the number of documents N;

[0039] Step 3.3: Since each keyword belongs to a clustering cluster, each keyword is replaced with the atomic ID of the cluster it belongs to;

[0040] For the keyword set of the multilingual document d i each of its keywords is mapped to the corresponding atomic ID representation in the atomic ID set A;

[0041] >Step 3.4: Since each multilingual document is represented by its keywords, the document identifier DocID is represented as the set of atomic IDs to which the keywords of the multilingual document belong, that is

[0042]

[0043] Each multilingual document obtains an atomic ID sequence of equal length, forming a consistent document representation in the shared semantic space.

[0044] Furthermore, Step 4 includes the following steps:

[0045] Step 4.1: In the model inference stage, decode the document identifier with a fixed length of m, represent the document as an atomic ID sequence, compress the decoding space into selection from C atoms, and reduce the overall size of the decoding space.

[0046] Step 4.2: During the decoding process, input query q, and the model generates target document d. q The process of document identifiers is described as follows:

[0047] Among them, a t Let P(a) represent the atoms generated in step t, with conditional probability P(a). t |a <t ,q) indicates that the generation of the current step depends on the previously generated atomic sequence a. ＜t And query q, P(DocID|q) means taking query q as input and generating target document d. q The document identifier DocID;

[0048] Step 4.3: In each decoding step, a dynamic multi-step constraint decoding strategy is used, that is, the decoding range of the current step is limited according to the decoding result of the previous step.

[0049] Step 4.4: Perform m iterations through the above steps to generate the complete final document identifier.

[0050] The present invention also provides a multilingual generative retrieval system based on cross-language semantic compression, the system comprising: a module for executing the multilingual generative retrieval method based on cross-language semantic compression.

[0051] The beneficial effects of this invention are:

[0052] 1. This invention first constructs a multilingual document retrieval dataset; then, it uses a keyword extraction model to extract document keywords, calculates the semantic similarity of all multilingual document keywords, clusters keywords with high similarity scores, encodes an atomic ID for each cluster, and assigns the atomic ID to the multilingual document as a document identifier; finally, in the inference stage, after inputting the query, it uses dynamic multi-step constraint decoding to gradually narrow the decoding range and obtain the final document identifier.

[0053] 2. This invention enables queries and documents to be generated and matched in a unified semantic space through cross-language semantic alignment, which significantly improves the adaptability and generalization effect of generative retrieval models in multilingual environments.

[0054] 3. Experiments were conducted on the constructed dataset using different baseline models. The results show that the present invention significantly improves the multilingual retrieval capability compared to other models, and the implementation results verify the effectiveness of the method of the present invention in multilingual retrieval tasks. Attached Figure Description

[0055] Figure 1 This invention provides a multilingual generative retrieval model framework based on cross-lingual semantic compression.

[0056] Figure 2 This illustrates the impact of the number of keywords on model performance in this embodiment of the invention.

[0057] Figure 3 This illustrates the impact of semantic similarity threshold on model performance in this embodiment of the invention. Detailed Implementation

[0058] Example 1: As Figures 1-3 As shown, a multilingual generative retrieval method based on cross-lingual semantic compression includes the following:

[0059] Step 1: Construct a multilingual document retrieval dataset;

[0060] Furthermore, in Step 1, the multilingual document retrieval dataset is derived from Wikipedia data and related web page data. By extracting titles and multiple parts of content from the web pages and filtering out invalid information, title-content pairs are formed, thereby obtaining the multilingual document retrieval dataset.

[0061] Further, Step 1 includes:

[0062] Step 1.1: Automatically obtain Wikipedia webpage content using web crawling technology; obtain documents in different languages ​​by modifying the language field of the Wikipedia URL.

[0063] Specifically, we first use a Wikipedia page in a specific language as a base template. By modifying the language field in the Wikipedia URL programmatically, we can access the content of the same topic in different language versions, thereby building a multilingual content-aligned corpus. The language fields include "fr", "sv", and "mk".

[0064] Step 1.2: In order to improve the success rate and stability of web page requests, code is used to construct and send fake web page requests, injecting key information such as user-agent and access key into the request header, thereby successfully establishing a secure connection with the target website server.

[0065] Specifically, automated scripts written in Python are used to construct and send web page requests that simulate real user behavior. This includes injecting customized fields into the request headers (HTTP headers), such as User-Agent (simulating different browser types), Referer (source page address), Accept-Language (accepted language range), as well as Cookies and access keys, to simulate legitimate client access scenarios.

[0066] Step 1.3: After obtaining the webpage source code, use the lxml.etree module in Python to perform DOM parsing on the HTML document structure. Then, analyze the tag hierarchy of the webpage and use XPath extraction technology to locate the anchor points of the required text content, such as headings, paragraphs, hyperlinks, and other information fields, for accurate extraction of target text content from the page. The lxml.etree module has efficient structured data extraction capabilities and is suitable for deep information extraction from complex pages. To ensure consistency in information extraction across multiple language versions, a general and robust XPath extraction logic needs to be designed based on the actual page structure to ensure accurate extraction of target text in different language versions.

[0067] Step 1.4: Finally, construct title-content information pairs from the obtained target text content to obtain a multilingual document retrieval dataset. The document dataset for each language includes a title, summary, and content.

[0068] Step 2: Extract keywords from multilingual documents from multiple perspectives using a keyword extraction model, and use semantic similarity to calculate the extracted keywords and construct a similarity matrix;

[0069] Furthermore, Step 2 includes:

[0070] Using a prompt-based large language model (LLM) as a keyword extraction model, a fixed number of keywords are extracted from each multilingual document in the multilingual document collection as a semantically compact representation of the multilingual document content.

[0071] The extracted keyword set has a unified format across all languages;

[0072] Then, the keyword sets of all multilingual documents are merged into a global keyword set;

[0073] Each keyword is encoded into a dense vector using a pre-trained text encoder, and cosine similarity is calculated to construct a similarity matrix between keywords for subsequent semantic modeling.

[0074] Furthermore, Step 2 specifically includes the following steps:

[0075] Step 2.1: Use a prompt-based Large Language Model (LLM) to process a collection of multilingual documents D = {d1, d2, ..., d...} i Extract each multilingual document d from ,…} i Keywords are used to capture key semantic information; these keywords serve as a semantically compact representation of the multilingual document content.

[0076] Step 2.2: Extract a fixed number of m keywords from each multilingual document to obtain the keyword set K of the multilingual document. i The extraction process is represented as follows:

[0077]

[0078] in, Indicates a multilingual document d i The m-th keyword in the text; the keywords have significant semantic differences to enhance the ability to express semantics from multiple perspectives;

[0079] The keyword extraction process is performed in a standardized manner across all multilingual documents to ensure consistency in semantic representation;

[0080] The set of keywords K for all multilingual documents i Merge into a global keyword set K;

[0081]

[0082] The global keyword set K contains n = |K| unique keywords, denoted as K.

[0083] N represents the number of multilingual documents;

[0084] Step 2.4: Use a pre-trained text encoder to encode each keyword into a dense vector v. i ∈R d Calculate the cosine similarity between any two keywords, and construct a keyword similarity matrix S∈R based on the cosine similarity scores. n×n Dense vector v i v j cosine similarity S ij The calculation formula is as follows:

[0085]

[0086] Where d is the dimension of the dense vector.

[0087] Step 3: Perform semantic clustering based on the similarity matrix, and represent the clusters using atomic IDs. Then, assign a document identifier to each multilingual document based on the cluster where the keyword belongs.

[0088] Further, the said Step 3 includes: setting a similarity threshold according to the similarity matrix, clustering the paired keywords with similarity scores greater than the similarity threshold, and those less than the similarity threshold form a separate cluster. Each cluster is represented by an atomic ID, and a document identifier is assigned to the multilingual document according to the cluster where the keyword is located.

[0089] Specifically, based on the similarity matrix S between keywords, the keywords with cosine similarity equal to or higher than the similarity threshold θ are clustered into the same cluster through a connected path, and those that do not meet the condition form separate clusters to form semantic clustering.

[0090] Each clustering cluster is assigned a unique atomic identifier (Atom ID), that is, the atomic ID, to form an atomic set A = {a1, a2, …, a C}, which serves as a semantic unit. Here, a C is an atom, and C is the number of clusters.

[0091] Subsequently, each keyword in the multilingual document is mapped to its corresponding atomic ID representation, so that each multilingual document is represented by the sequence of atomic IDs corresponding to its keywords as is the atom corresponding to the m-th keyword of the i-th multilingual document, ]>This process realizes the equal-length and consistent representation of cross-language documents in a unified semantic space.

[0092] Further, the specific steps of the said Step 3 include: <0[000262>Step 3.1: According to the similarity matrix S between keywords constructed in Step 2, when there is a path between any two keywords and the similarity of all adjacent keywords on the path satisfies S ij ≥θ, these keywords are clustered into one cluster, and the similarity threshold θ ∈ [0, 1]; if the similarity of a certain keyword with any other keyword is lower than the similarity threshold, an independent cluster is formed.

[0094] Step 3.2: For the clustering results of all keywords, assign a unique atomic identifier, that is, the atomic ID, to each clustering cluster to obtain the atomic set A = {a1, a2, …, a C}; an atom represents a group of keywords with similar semantics, and the number of clusters C satisfies C < N, that is, the number of clusters C is less than the number of documents N. <00002[66>Step 3.3: Since each keyword belongs to a clustering cluster, each keyword is replaced by the atomic ID of the cluster it belongs to.

[0096] For the multilingual document d i of the keyword set Each of its keywords Map the corresponding atomic ID representation in the atomic ID set A;

[0097] Step 3.4: Since each multilingual document is represented by its keywords, the document identifier DocID represents the set of atomic IDs to which the keywords of the multilingual document belong, i.e.

[0098]

[0099] For the m-th keyword in the i-th multilingual document,

[0100] Each multilingual document receives a sequence of atomic IDs of equal length, forming a consistent document representation in a shared semantic space.

[0101] Step 4: In the reasoning stage, after inputting the query, a dynamic multi-complement constraint decoding method is adopted. Based on the decoding results of the previous steps, the decoding range of the document identifier in the current step is gradually narrowed to obtain the final document identifier.

[0102] Furthermore, Step 4 includes the following steps:

[0103] Step 4.1: In the model inference stage, the document retrieval process is modeled as a multi-step decoding process of length m. Traditional methods require selecting from all N documents at each step, resulting in a search space of O(N^2). m Our method represents documents as sequences of atomic IDs, compressing the decoding space into selecting from C semantic atoms, thus reducing the overall decoding space to O(C). m );

[0104] Step 4.2: During the decoding process, input query q, and the model generates target document d. q The process of obtaining the document identifier DocID is described as follows:

[0105] Among them, a t Let P(a) represent the atoms generated in step t, with conditional probability P(a). t |a <t ,q) indicates that the generation of the current step depends on the previously generated atomic sequence a. ＜t And query q, P(DocID|q) means taking query q as input and generating target document d. q The document identifier DocID;

[0106] Step 4.3: In each decoding step, a dynamic multi-step constraint decoding strategy is used, that is, the decoding range of the current step is limited according to the decoding result of the previous step.

[0107] Specifically, the decoding of each step t adopts a dynamic multi-step constraint mechanism, that is, based on the generated prefix, i.e., the document identifier DocID = [a1, a2, ..., a...] t-1 Dynamically adjust the current set of available atomic IDs A t :

[0108]

[0109] Among them, a t-1 Represents the atoms generated in step t-1, Constraint(K) i () represents the set of keywords that meet the conditions under the current prefix constraint, corresponding to the currently available set of atomic IDs A. t ;

[0110] The optimal solution for the current step is:

[0111]

[0112] in, It refers to the probability of predicting the decoding result of the current step based on the query and the decoding results of the previous step.

[0113] Step 4.4: Perform m iterations using steps Step 4.1-Step 4.3 above to generate the complete final document identifier:

[0114] DocID = [a1,a2,…,a m ].

[0115] Table 1 shows the pseudocode for dynamic multi-step constraint decoding.

[0116]

[0117]

[0118] This invention also provides a multilingual generative retrieval system based on cross-language semantic compression, the system comprising:

[0119] The first building module is used to construct a multilingual document retrieval dataset;

[0120] The second building module is used to extract keywords from multilingual documents from multiple perspectives through a keyword extraction model, and to calculate the extracted keywords using semantic similarity to build a similarity matrix;

[0121] The allocation module is used to perform semantic clustering based on the similarity matrix, and to represent the clusters using atomic IDs. Then, the clusters containing the keywords are used to assign document identifiers to each multilingual document.

[0122] The module is used in the inference phase to dynamically narrow the decoding range of the document identifier in the current step based on the decoding results of the previous steps after inputting the query, thereby obtaining the final document identifier.

[0123] Experiments were conducted on a multilingual document retrieval dataset to demonstrate the method proposed in this invention. The number of documents in each language was divided as follows:

[0124] Table 2 shows the language distribution of the multilingual document retrieval dataset.

[0125]

[0126] In this invention, all methods are based on the Transformer architecture and reproduced on the mT5-base model. The training data employs a strategy of using multilingual query representations of documents. Specifically, the model used in all pseudo-query generation tasks is Llama 3.1-8B, with a generation temperature set to 0.7. Each document sample generates 10 multilingual pseudo-queries using this model to construct the training data. In the keyword generation task, the Llama 3.1-8B model is still used, but the temperature is set to 0 to ensure the determinism and consistency of the generated content. For semantic similarity calculation, this invention uses the paraphrase-multilingual-MiniLM-L12-v2 model, which can effectively handle multilingual semantic matching tasks.

[0127] For training setup, this invention utilizes the PyTorch and Transformers frameworks. For the multilingual document retrieval dataset, the learning rate is set to 5×10⁻⁶. -4 The batch size was 128, the training epochs were 50, and m = 3 keywords were selected. The cross-entropy loss function was used as the training objective function. All experiments were conducted on eight NVIDIA A40 graphics cards, each with 46GB of video memory, ensuring efficient execution of large-scale training tasks.

[0128] Regarding evaluation metrics, this invention follows the established framework of existing research, evaluating model performance on the validation set of a multilingual document retrieval dataset. The primary evaluation metrics are Recall@1 and Recall@10, representing the proportion of relevant documents included in the top 1 and top 10 search results, respectively. Considering that the candidate document corpus comprises multiple languages, we reported the retrieval performance for each query language separately during the evaluation process to more comprehensively reflect the model's performance in a multilingual retrieval environment.

[0129] To verify the effectiveness of the method proposed in this invention, multilingual retrieval methods related to this invention were selected as baseline models, namely BM25, mDPR, DSI, DSI-QG, and SE-DSI, as follows:

[0130] BM25 is a classic sparse retrieval method based on an inverted index structure. It relies on exact keyword matching for retrieval operations and is widely used in traditional information retrieval systems.

[0131] mDPR is a dense retrieval method that employs a dual-encoder architecture, enabling the query and document to be encoded into vector representations respectively, and then performing similarity matching within the vector space. In this invention, LaBSE is used as its encoder to enhance multilingual representation capabilities.

[0132] DSI is a generative retrieval method that uses the document itself as training input and constructs document IDs (DocIDs) based on hierarchical clustering, thereby enabling a retrieval process without explicit indexing.

[0133] DSI-QG is an improved generative retrieval method that trains a query generation model to transform original documents into short pseudo-queries for modeling, while using random numbers to represent the corresponding document IDs, thereby enhancing the model's generalization ability.

[0134] SE-DSI also belongs to the generative retrieval paradigm, characterized by using strings containing semantic information as document IDs. In this invention, the title of multilingual documents is used as the DocID to improve the semantic richness of document representation and cross-language consistency.

[0135] Tables 3 and 4 show that on the validation set, BM25 and DSI have relatively limited retrieval capabilities; while mDPR maintains a moderate level of performance, it is quite sensitive to language variations. DSI-QG and SE-DSI show improvements in cross-linguistic recall balance compared to previous methods, but their performance is still affected by the richness of language resources. In contrast, this invention achieves stable and high recall rates across all languages ​​in the multilingual document retrieval dataset, showing a significant improvement, especially in languages ​​with low to medium resource availability. Recall@1 is improved by 2.79% compared to other methods, and recall@10 is improved by 5.01%. This demonstrates that the invention exhibits good generalization ability across language families with similar and significantly different language types.

[0136] Table 3 shows the Recall@1 performance comparison between the present invention and the baseline model.

[0137]

[0138] Table 4 shows the Recall@10 performance comparison between the present invention and the baseline model.

[0139]

[0140] To further validate the multilingual generative retrieval method proposed in this invention, hyperparameter experiments were designed to investigate the impact of semantic similarity threshold θ and the number of keywords m per document on model performance, as shown below:

[0141] To investigate the impact of the number of keywords *m* on retrieval performance and decoding time, we conducted experiments on a multilingual document retrieval dataset. For example... Figure 2 As shown, increasing the number of keywords can improve semantic representation capabilities, but it also increases the length of the DocID, leading to slower decoding speed and a decrease in overall performance. The optimal Recall@10 is 48.06%, corresponding to the use of three keywords. Fewer keywords (such as two) limit semantic coverage; while more keywords (such as five or six) reduce performance due to increased decoding complexity. Decoding time is positively correlated with DocID length. When the number of keywords increases from two to six, the decoding time approximately doubles. In comparison, SE-DSI has a Recall@10 of 29.96, and its latency is comparable to our method when using six keywords, further illustrating the strong correlation between sequence length and inference time.

[0142] Furthermore, to evaluate the sensitivity of keyword clustering to the semantic similarity threshold θ, we assessed retrieval performance on a multilingual document retrieval dataset with a threshold range of 0.5 to 0.9. Figure 3 As shown, increasing the threshold from 0.5 to 0.8 resulted in a steady increase in both Recall@1 and Recall@10, indicating that finer clustering helps improve retrieval accuracy. The best performance was achieved at a threshold of 0.8, with Recall@1 at 24.48% and Recall@10 at 47.89%. However, performance declined when the threshold was further increased to 0.9, with Recall@1 dropping to 22.02% (a decrease of 2.46%) and Recall@10 dropping to 44.71% (a decrease of 3.18%). This suggests that excessively high thresholds lead to overly granular clustering, grouping semantically related keywords into different groups, thereby reducing semantic generalization ability and limiting the model's retrieval coverage.

[0143] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multilingual generative retrieval method based on cross-lingual semantic compression, characterized in that: The method includes: Step 1: Construct a multilingual document retrieval dataset; Step 2: Extract keywords from multilingual documents from multiple perspectives using a keyword extraction model, and use semantic similarity to calculate the extracted keywords and construct a similarity matrix; Step 3: Perform semantic clustering based on the similarity matrix, and represent the clusters using atomic IDs. Then, assign a document identifier to each multilingual document based on the cluster where the keyword belongs. Step 4: In the reasoning stage, after inputting the query, a dynamic multi-complement constraint decoding method is adopted. Based on the decoding results of the previous steps, the decoding range of the document identifier in the current step is gradually narrowed to obtain the final document identifier. Step 2 includes: Using a prompt-based large language model (LLM) as a keyword extraction model, a fixed number of keywords are extracted from each multilingual document in the multilingual document collection as a semantically compact representation of the multilingual document content. The extracted keyword set has a unified format across all languages; Then, the keyword sets of all multilingual documents are merged into a global keyword set; Each keyword is encoded into a dense vector using a pre-trained text encoder, and cosine similarity is calculated to construct a similarity matrix between keywords for subsequent semantic modeling. Step 4 includes the following steps: Step 4.1: In the model inference stage, decode the document identifier of fixed length m, represent the document as an atomic ID sequence, and compress the decoding space into... By selecting from individual atoms, the overall decoding space can be reduced in size; Step 4.2: During the decoding process, input the query q, and the model generates the target document. The process of document identifiers is described as follows: ; in, Indicates the first Atoms generated in one step, conditional probability This indicates that the generation of the current step depends on the previously generated atomic sequence. and inquiries , Indicates query Generate target document from input. The document identifier DocID; Step 4.3: In each decoding step, a dynamic multi-step constraint decoding strategy is used, that is, the decoding range of the current step is limited according to the decoding result of the previous step. Step 4.4: Proceed through the above steps. The next iteration generates the complete final document identifier.

2. The multilingual generative retrieval method based on cross-lingual semantic compression according to claim 1, characterized in that: In Step 1, the multilingual document retrieval dataset is derived from Wikipedia data and related web page data. By extracting titles and multiple parts of content from the web pages and filtering out invalid information, title-content pairs are formed, thereby obtaining the multilingual document retrieval dataset.

3. The multilingual generative retrieval method based on cross-lingual semantic compression according to claim 1, characterized in that: Step 1 includes: Step 1.1: Automatically obtain Wikipedia webpage content using web crawling technology; obtain documents in different languages ​​by modifying the language field of the Wikipedia URL. Step 1.2: Use code to construct and send a fake web page request, injecting key information including user agent and access key into the request header, thereby successfully establishing a secure connection with the target website server; Step 1.3: After obtaining the webpage source code, use the lxml.etree module in Python to parse the page structure; then, use XPath path extraction technology to locate the anchor points of the required information for accurate collection of target text content in the page; Step 1.4: Finally, construct title-content information pairs from the obtained target text content to obtain a multilingual document retrieval dataset.

4. The multilingual generative retrieval method based on cross-lingual semantic compression according to claim 1, characterized in that: The specific steps of Step 2 include: Step 2.1: Use a prompt-based large language model (LLM) to process multilingual document collections. Extract each multilingual document Keywords; these keywords serve as a semantically compact representation of the content of multilingual documents; Step 2.2: Extract a fixed number of samples from each multilingual document. 10 keywords, to obtain the keyword set of the multilingual document. The extraction process is represented as follows: ； in, Multilingual documents The m-th keyword in; The keyword extraction process is performed in a standardized manner across all multilingual documents; Keyword set for all multilingual documents Merge into a global keyword set K; ； The global keyword set K contains unique keywords, totaling [number missing]. One, denoted as N represents the number of multilingual documents; Step 2.4: Encode each keyword into a dense vector using a pre-trained text encoder. Calculate the cosine similarity between any two keywords, and construct a similarity matrix between keywords based on the cosine similarity scores. Dense vectors , cosine similarity The calculation formula is as follows: ； Where d is the dimension of the dense vector.

5. The multilingual generative retrieval method based on cross-lingual semantic compression according to claim 1, characterized in that: Step 3 includes: setting a similarity threshold based on the similarity matrix; clustering pairs of keywords with similarity scores greater than the similarity threshold; grouping keywords with similarity scores less than the similarity threshold into a single cluster; using atomic IDs to represent each cluster; and assigning document identifiers to multilingual documents based on the cluster to which the keywords belong.

6. The multilingual generative retrieval method based on cross-lingual semantic compression according to claim 1, characterized in that: The specific steps in Step 3 include: Step 3.1: Based on the keyword similarity matrix constructed in Step 2 When there is a path between any two keywords, the similarity of all adjacent keywords on that path satisfies At that time, these keywords are clustered into a cluster, and the similarity threshold is used. If the similarity of a keyword to any other keyword is lower than the similarity threshold, then an independent cluster is formed. Step 3.2: For the clustering results of all keywords, assign a unique atom identifier, i.e., atom ID, to each cluster to obtain the atom set. One atom represents a group of semantically similar keywords, and the number of clusters... satisfy That is, the number of clusters Less than the number of documents N; Step 3.3: Since each keyword belongs to a cluster, each keyword... Replace it with the atomic ID of its cluster; For multilingual documents Keyword set Each of its keywords In the atomic ID set The mapping is represented by the corresponding atom ID; Step 3.4: Since each multilingual document is represented by its keywords, the document identifier DocID represents the set of atomic IDs to which the keywords of the multilingual document belong, i.e. ； For the i-th multilingual document, the first... The atom corresponding to each keyword ; Each multilingual document receives a sequence of atomic IDs of equal length, forming a consistent document representation in a shared semantic space.

7. A multilingual generative retrieval system based on cross-lingual semantic compression, characterized in that, The system includes a module for executing the multilingual generative retrieval method based on cross-lingual semantic compression as described in any one of claims 1 to 6.

Images