Chinese-vietnamese cross-language query expansion method based on retrieval enhancement and knowledge distillation

Abstract

The present application relates to a Chinese-Vietnamese cross-language query expansion method based on retrieval enhancement and knowledge distillation, and belongs to the technical field of natural language processing. The present application injects the thinking chain generation ability of a large-scale language model and the retrieved external knowledge into a multilingual pre-training model with fewer parameters through knowledge distillation and retrieval enhancement, thereby improving the thinking chain generation ability of the multilingual pre-training model. Compared with query expansion, cross-language query expansion can improve the reasoning and generation ability of a multilingual pre-training model in a low-resource language scenario. The present application plays an important role in Chinese-Vietnamese cross-language question answering, Chinese-Vietnamese cross-language information retrieval and other downstream tasks. The experimental results on the MLQA, XQuAD public data sets and the constructed Chinese-Vietnamese cross-language query expansion data set show that the performance indicators of the present application are better than those of the baseline model, and the MAP, Recall, NDCG and MRR are increased by 3.4%, 1.6%, 2.9% and 3.4%, respectively.

Description

Technical Field

[0001] This invention relates to a Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation, belonging to the field of natural language processing technology. Background Technology

[0002] The Chinese-Vietnamese cross-language query expansion aims to add terms and concepts semantically identical to Chinese queries and then translate the expanded Chinese queries into Vietnamese queries. This expansion is one of the core technologies for resolving issues such as query topic drift and word mismatch. Furthermore, effective Chinese-Vietnamese cross-language query expansion provides a solid foundation for downstream tasks such as Chinese-Vietnamese cross-language question answering and information retrieval.

[0003] This invention relates to a specific cross-language query expansion technique, namely, Chinese-Vietnamese cross-language query expansion. In recent years, significant research progress has been made in this field, with mainstream methods encompassing machine translation-based techniques, methods based on multilingual pre-trained language models, and methods utilizing large language models. In machine translation-based methods, the original query statement is converted into the target language using neural machine translation technology, and then query expansion is performed based on the translated content. However, the effectiveness of this method is heavily influenced by the accuracy of machine translation. For resource-rich languages, the results are ideal due to relatively small translation errors; however, for low-resource languages ​​such as Vietnamese, inaccurate translation may affect the expanded retrieval results. To address these issues, researchers have proposed methods based on multilingual pre-trained language models. The core idea of ​​this method is to expand the original query in the source language and then use multilingual pre-trained language models such as mBERT and mT5 to map these expanded queries to a unified semantic space, thereby achieving cross-language query expansion. However, in low-resource cross-language environments, existing multilingual pre-trained language models may encounter problems such as the curse of multilingualism and language bias. With the rise and application of large language models such as ChatGPT, these models have demonstrated significant effectiveness in query expansion tasks. Currently, methods based on large language models mainly cover strategies such as generating relevant query variations, generating pseudo-documents and keywords using pre-set templates, and performing thought chain reasoning to improve the quality of query results. Although large language models perform well in understanding and generating text context, they still face several challenges in cross-language query expansion tasks. First, large language models may suffer from illusions or fail to update in a timely manner, resulting in the generation of text containing irrelevant or misleading information. Second, due to the large number of parameters in large language models, the demand for computing resources is enormous, increasing the difficulty of direct application in cross-language query expansion tasks. In addition, when dealing with low-resource languages, the reasoning and text generation capabilities of multilingual pre-trained models often fail to achieve the expected results. This invention proposes a Chinese-Vietnamese cross-language query expansion method based on retrieval enhancement and knowledge distillation. Through knowledge distillation and retrieval enhancement, the thought chain generation capability of large language models and retrieved external knowledge are injected into multilingual pre-trained models with fewer parameters, thereby improving their thought chain generation capability. Experimental results on a self-built Chinese-Vietnamese query extension dataset and two public datasets show that the proposed method effectively improves the retrieval performance of cross-language query extension tasks. Summary of the Invention

[0004] This invention provides a Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation to address the problems of large-scale language models having a large number of parameters and high computational resource consumption, making it difficult to directly apply them to cross-lingual query expansion tasks; and the poor performance of multilingual pre-trained models in reasoning and generation capabilities in low-resource language scenarios.

[0005] The technical solution of this invention is: a Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation. The specific steps of the Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation are as follows:

[0006] Step 1: Collect query data from Chinese-Vietnamese Wikipedia and Vietnam News Network, and collect a Chinese-Vietnamese cross-language question and answer public dataset. Then, preprocess the data to form the experimental dataset of this invention.

[0007] Step 2: Building on Step 1, knowledge distillation and retrieval enhancement are used to inject the thought chain generation capability of the large-scale language model and the retrieved external knowledge into the multilingual pre-trained model with fewer parameters, thereby improving its thought chain generation capability.

[0008] Step 3: Based on Step 2, fine-tune the multilingual pre-trained model on the Chinese-Vietnamese retrieval dataset to improve the model's performance in Chinese-Vietnamese cross-language retrieval and enhance the retrieval effect of the Chinese-Vietnamese cross-language query extension task.

[0009] Furthermore, the specific steps of Step 1 are as follows:

[0010] Step 1.1: This invention collected relevant data from Wikipedia and Vietnam News Network, and collected a public dataset of Chinese-Vietnamese cross-language question and answer. In order to complete the expansion of Chinese-Vietnamese cross-language queries, 106,266 aligned Chinese-Vietnamese cross-language query document pairs were finally constructed.

[0011] Step 1.2: For Wikipedia and news website data, this invention extracts document data Dc and Dv from Chinese and Vietnamese Wikipedia pages. This invention obtains the Chinese query Qc(i) from the Chinese data document Dc, and calculates the similarity between the Chinese query Qc(i) and the Vietnamese document Dv(j) using cosine similarity through cross-linguistic word embedding. If the embedding result is greater than a preset threshold β, the Chinese query and the Vietnamese document are considered a similar language pair. This method achieves the alignment of related languages ​​between the Chinese query and the Vietnamese document.

[0012] Step 1.3: For the Chinese-Vietnamese cross-language question-answering public dataset, this invention extracts Chinese to Vietnamese (zh-vi) and Vietnamese to Chinese (vi-zh) data from the Chinese-Vietnamese cross-language question-answering dataset, obtains the Chinese and Vietnamese queries Qc(i) and Qv(i) and the related Vietnamese and Chinese texts Dv(j) and Dc(j), and constructs the Chinese-Vietnamese query extension dataset through this method.

[0013] Furthermore, Step 2 includes three new query expansion task modules: a thought chain generation module, a retrieval enhancement module, and a cross-language query expansion module; the specific steps are as follows:

[0014] Step 2.1 presents a Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation, comprising three novel query expansion task modules: a thought chain generation module, a retrieval enhancement module, and a cross-lingual query expansion module. The thought chain generation module constructs zero-shot hint templates for the original Chinese query and inputs them into a large-scale language model to generate zero-shot thought chains. The large-scale language model is then used as the teacher model, and its knowledge is distilled into the student model.

[0015] Step 2.2: The retrieval enhancement module retrieves the k most relevant Vietnamese documents using a cross-language retrieval model. Then, using retrieval enhancement suggestion templates, the Chinese query and relevant Vietnamese documents are input into the mT5 model for training. The training goal is to generate zero-shot thought chains and improve the model's inference and retrieval performance by iteratively refining and perfecting the chained thinking after retrieval enhancement.

[0016] Step 2.3: Using the query-mind chain data pairs constructed in Step 2.1 and Step 2.2, and integrating them with the top k Vietnamese-related documents retrieved through the original query, the original Chinese query is concatenated with the enhanced mind chain. Through vectorization processing of the query and documents, the most relevant Vietnamese documents are retrieved.

[0017] Furthermore, the specific steps of Step 2.1 are as follows:

[0018] Step 2.1.1: Input Q for the extended Chinese-Vietnamese cross-language query task C This invention utilizes the generation of corresponding thought chains based on query statements, and constructs data pairs between queries and thought chains accordingly. A specially designed prompt template called PromptCoT effectively guides large language models to generate the required thought chains.

[0019] CoT V =M(PromptCoT(Q) C ))

[0020] Step 2.1.2: Query-thought chain data pairs were constructed on the self-built dataset and two publicly available Han-Yue datasets. The Chinese query statement Q was selected from the datasets. C This information is then embedded into the PromptCoT prompt template to construct a complete prompt input. The prompt input is then passed to the large language model M to generate a Vietnamese thought chain CoT corresponding to the Chinese query. VThe generated Vietnamese thought chains were matched with the original Chinese query statements to form query-thought chain data pairs needed for training and testing.

[0021] Furthermore, the specific steps of Step 2.2 are as follows:

[0022] Step 2.2.1: Based on Step 2.1, for the constructed query-mind chain data CoT V Integrate the top k Vietnamese-related documents D retrieved through the original query. V1 D V2 …D Vk This allows for the training of a student model for cross-language query expansion. By combining retrieved Vietnamese documents with the original Chinese query, this invention designs a retrieval enhancement prompt template, PromptRAG.

[0023]

[0024]

[0025] As a search enhancement suggestion template designed specifically for generation, its working mechanism involves prioritizing the reading and incorporation of relevant fragments selected from a set of retrieved Vietnamese documents. i This represents the encoding of the search enhancement suggestion corresponding to the i-th document among the first k retrieved documents. All encoded search enhancement suggestions T... i They are concatenated in sequence to form the overall contextual input for the cross-language model decoder.

[0026]

[0027] Step 2.2.2: During the training process, this invention utilizes the retrieved relevant context document set T to enhance the model's capabilities. Through these documents, a cross-language model is trained to generate pseudo-target inference CoT. V .

[0028] Loss = -logmT5_Dec prop (CoT V |[PAD],T)

[0029] Among them mT5_Dec prop (x|y) represents the generation probability of x under condition y. The thought chain generation capability of the large language model is injected into the model through knowledge distillation, thereby optimizing its performance during inference. The cross-language model combines T and [PAD] to generate the output CoT. QE .

[0030] CoT QE=mT5_Dec([PAD],T)

[0031] The output CoT QE Appended to the original query Q C This allows for the expansion of the query.

[0032]

[0033] Furthermore, the specific steps of Step 2.3 are as follows:

[0034] Step 2.3.1, For the original Chinese query Q C This query reflects the user's initial information needs. To further enrich the query context, this invention employs a search-enhanced CoT (Coordinated Thinking Chain) mechanism. QE This expands the query content. Subsequently, the CoT enhances both the original query and the retrieval. QE The content is concatenated and vectorized, converting it into a high-dimensional vector to facilitate comparison from a semantic similarity perspective. This invention uses cosine similarity to calculate semantic similarity.

[0035]

[0036] Furthermore, the specific steps of Step 3 are as follows:

[0037] Step 3.1: The Chinese question portion of the self-built dataset is used as the query portion in the cross-language retrieval task, and the corresponding Vietnamese documents are used as the relevant documents for the retrieval task. Next, the question portion of the Chinese-Vietnamese cross-language question-and-answer dataset is used as the Chinese query, and the correct Vietnamese answers and related text segments are used as the relevant documents for the retrieval. Finally, an evaluation dataset of cross-language queries and relevant documents is obtained.

[0038] Step 3.2: Based on Step 3.1, Normalized Discount Cumulative Gain (NDCG), Mean Precision (MAP), Mean Reciprocal Rank (MRR), and Recall are used as evaluation metrics for the cross-language query expansion task.

[0039]

[0040]

[0041]

[0042]

[0043] Step 3.3: Building upon Step 3.1, this invention optimizes the Chinese-Vietnamese cross-language query expansion task. During fine-tuning, the Dropout size is 0.1, the batch size is 4, the maximum input length is 1024, and the maximum output length is 256. The model is trained for a total of 10 epochs. When the model obtains the best retrieval score on the validation set, the program evaluates it on the test set and records the corresponding score, finally selecting the best score as the result.

[0044] The beneficial effects of this invention are:

[0045] 1. For the cross-language query expansion task of low-resource Chinese and Vietnamese languages, a cross-language query expansion method based on retrieval enhancement and knowledge distillation is proposed to improve the retrieval performance of multilingual pre-trained models in the cross-language query expansion task of Chinese and Vietnamese.

[0046] 2. The model was trained and tested using a self-built Chinese-Vietnamese cross-language query extension dataset, and further validated on two publicly available cross-language question-answering datasets, MLQA and XQuAD. Experimental results show that the method of this invention can effectively improve the retrieval performance of the Chinese-Vietnamese cross-language query extension task. Attached Figure Description

[0047] Figure 1 This is a flowchart of the overall model in this invention;

[0048] Figure 2 This is an example of the present invention;

[0049] Figure 3 This is a model diagram of the present invention. Detailed Implementation

[0050] Example 1: As Figure 1-3 As shown, the specific steps of the Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation described in the invention are as follows:

[0051] Step 1: Collect query data from Chinese-Vietnamese Wikipedia and Vietnam News Network, and collect a Chinese-Vietnamese cross-language question and answer public dataset. Then, preprocess the data to form the experimental dataset of this invention.

[0052] As a preferred embodiment of the present invention, the specific steps of Step 1 are as follows:

[0053] Step 1.1: This invention collected relevant data from Wikipedia and Vietnam News Network, and collected a public dataset of Chinese-Vietnamese cross-language question and answer. In order to complete the expansion of Chinese-Vietnamese cross-language queries, 106,266 aligned Chinese-Vietnamese cross-language query document pairs were finally constructed.

[0054] Step 1.2: For Wikipedia and news website data, this invention extracts document data Dc and Dv from Chinese and Vietnamese Wikipedia pages. This invention obtains the Chinese query Qc(i) from the Chinese data document Dc, and calculates the similarity between the Chinese query Qc(i) and the Vietnamese document Dv(j) using cosine similarity through cross-linguistic word embedding. If the embedding result is greater than a preset threshold β, the Chinese query and the Vietnamese document are considered a similar language pair. This method achieves the alignment of related languages ​​between the Chinese query and the Vietnamese document.

[0055] Step 1.3: For the Sino-Vietnamese cross-language question-answering public dataset, this invention extracts data from Chinese to Vietnamese (zh-vi) and Vietnamese to Chinese (vi-zh) from the dataset, obtaining the Chinese and Vietnamese queries Qc(i) and Qv(i), as well as the related Vietnamese and Chinese texts Dv(j) and Dc(j). This method is used to construct the extended Sino-Vietnamese query dataset. The final dataset statistics are shown in Table 1.

[0056] Table 1. Retrieval Training and Evaluation Dataset Settings

[0057]

[0058] Step 2: Building on Step 1, knowledge distillation and retrieval enhancement are used to inject the thought chain generation capability of the large-scale language model and the retrieved external knowledge into the multilingual pre-trained model with fewer parameters, thereby improving its thought chain generation capability.

[0059] As a preferred embodiment of the present invention, Step 2 includes three new query expansion task modules: a thought chain generation module, a retrieval enhancement module, and a cross-language query expansion module; the specific steps of Step 2 are as follows:

[0060] Step 2.1: The thought chain generation module constructs a zero-shot suggestion template for the original Chinese query and inputs it into a large language model to generate a zero-shot thought chain. The large language model is then used as the teacher model, and its knowledge is distilled into the student model.

[0061] Step 2.2: The retrieval enhancement module retrieves the k most relevant Vietnamese documents using a cross-language retrieval model. Then, using retrieval enhancement suggestion templates, the Chinese query and relevant Vietnamese documents are input into the mT5 model for training. The training goal is to generate zero-shot thought chains and improve the model's inference and retrieval performance by iteratively refining and perfecting the chained thinking after retrieval enhancement.

[0062] Step 2.3: Using the query-mind chain data pairs constructed in Step 2.1 and Step 2.2, and integrating them with the top k Vietnamese-related documents retrieved through the original query, the original Chinese query is concatenated with the enhanced mind chain. Through vectorization processing of the query and documents, the most relevant Vietnamese documents are retrieved.

[0063] As a preferred embodiment of the present invention, the specific steps of step 2.1 are as follows:

[0064] Step 2.1.1: Input Q for the extended Chinese-Vietnamese cross-language query task C This invention utilizes the generation of corresponding thought chains based on query statements, and constructs data pairs between queries and thought chains accordingly. A specially designed prompt template called PromptCoT effectively guides large language models to generate the required thought chains.

[0065] COT V =M(PromptCoT(Q) C ))

[0066] Step 2.1.2: Query-thought chain data pairs were constructed on the self-built dataset and two publicly available Han-Yue datasets. The Chinese query statement Q was selected from the datasets. C This information is then embedded into the PromptCoT prompt template to construct a complete prompt input. The prompt input is then passed to the large language model M to generate a Vietnamese thought chain CoT corresponding to the Chinese query. V The generated Vietnamese thought chains were matched with the original Chinese query statements to form query-thought chain data pairs needed for training and testing.

[0067] As a preferred embodiment of the present invention, the specific steps of Step 2.2 are as follows:

[0068] Step 2.2.1: Based on Step 2.1, for the constructed query-mind chain data CoT V Integrate the first k Vietnamese-related documents D retrieved through the original query. V1 D V2 …D Vk This allows for the training of a student model for cross-language query expansion. By combining retrieved Vietnamese documents with the original Chinese query, this invention designs a retrieval enhancement prompt template, PromptRAG.

[0069]

[0070]

[0071] As a search enhancement suggestion template designed specifically for generation, its working mechanism involves prioritizing the reading and incorporation of relevant fragments selected from a set of retrieved Vietnamese documents. i This represents the encoding of the search enhancement suggestion corresponding to the i-th document among the first k retrieved documents. All encoded search enhancement suggestions T... i They are concatenated in sequence to form the overall contextual input for the cross-language model decoder.

[0072]

[0073] Step 2.2.2: During the training process, this invention utilizes the retrieved relevant context document set T to enhance the model's capabilities. Through these documents, a cross-language model is trained to generate pseudo-target inference CoT. V .

[0074] Loss = -logmT5_Dec prop (CoT V |[PAD],T)

[0075] Among them mT5_Dec prop (x|y) represents the generation probability of x under condition y. The thought chain generation capability of the large language model is injected into the model through knowledge distillation, thereby optimizing its performance during inference. The cross-language model combines T and [PAD] to generate the output CoT. QE .

[0076] CoT QE =mT5_Dec([PAD],T)

[0077] The output CoT QE Appended to the original query Q C This allows for the expansion of the query.

[0078]

[0079] As a preferred embodiment of the present invention, the specific steps of step 2.3 are as follows:

[0080] Step 2.3.1, For the original Chinese query Q C This query reflects the user's initial information needs. To further enrich the query context, this invention employs a search-enhanced CoT (Coordinated Thinking Chain) mechanism. QE This expands the query content. Subsequently, the CoT enhances both the original query and the retrieval. QE The content is concatenated and vectorized, converting it into a high-dimensional vector to facilitate comparison from a semantic similarity perspective. This invention uses cosine similarity to calculate semantic similarity.

[0081]

[0082] Step 3: Based on Step 2, fine-tune the model on the Chinese-Vietnamese retrieval dataset to improve the model's performance in Chinese-Vietnamese cross-language retrieval and enhance the retrieval effect of the Chinese-Vietnamese cross-language query extension task.

[0083] As a preferred embodiment of the present invention, the specific steps of step 3 are as follows:

[0084] Step 3.1: The Chinese question portion of the self-built dataset is used as the query portion in the cross-language retrieval task, and the corresponding Vietnamese documents are used as the relevant documents for the retrieval task. Next, the question portion of the Chinese-Vietnamese cross-language question-and-answer dataset is used as the Chinese query, and the correct Vietnamese answers and related text segments are used as the relevant documents for the retrieval. Finally, an evaluation dataset of cross-language queries and relevant documents is obtained.

[0085] Step 3.2: Based on Step 3.1, Normalized Discount Cumulative Gain (NDCG), Mean Precision (MAP), Mean Reciprocal Rank (MRR), and Recall are used as evaluation metrics for the cross-language query expansion task.

[0086]

[0087]

[0088]

[0089]

[0090] Step 3.3: Building upon Step 3.1, this invention optimizes the Chinese-Vietnamese cross-language query expansion task. During fine-tuning, the Dropout size is 0.1, the batch size is 4, the maximum input length is 1024, and the maximum output length is 256. The model is trained for a total of 10 epochs. When the model obtains the best retrieval score on the validation set, the program evaluates it on the test set and records the corresponding score, finally selecting the best score as the result.

[0091] To illustrate the detection effect of the present invention, a baseline method is used to compare the results with those of the present invention, specifically with the core query expansion method described below.

[0092] As a multilingual pre-trained model, mBERT is unique in that it can learn and extract general language representations using a rich multilingual corpus.

[0093] Q2D is a query expansion method based on a large-scale language model. It enhances the efficiency of information retrieval by leveraging a large language model to improve queries.

[0094] •CoT is a method that optimizes search queries by generating thought chains. Through an alternating process of retrieving and extracting information, it ultimately arrives at an accurate answer.

[0095] •CLCoT: This method is developed based on xCoT. It applies mind chain generation technology to cross-language environments, providing strong support for cross-language information retrieval tasks.

[0096] The experimental results of the Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation are shown in Table 2:

[0097] Table 2. Experimental Results of Cross-Language Query Expansion between Chinese and Vietnamese

[0098]

[0099] Furthermore, this invention also verifies the effectiveness of the cross-language query expansion method under different multilingual models, specifically by comparing it with the baseline of the multilingual model described below and the query expansion method of this invention.

[0100] XLM-R is a multilingual pre-trained model based on the RoBERTa architecture. Through self-supervised learning on large-scale cross-lingual datasets, it effectively learns the semantic relationships and commonalities between various languages, enabling it to perform understanding and generation tasks in multilingual environments.

[0101] SBERT is a sentence embedding model based on the BERT architecture. By training on sentence pairs under supervision, it can efficiently generate semantically similar sentence embeddings for various natural language processing tasks, such as similarity calculation, clustering, and retrieval.

[0102] MPNet is an improved pre-trained model that combines masked language models and sequence rearrangement strategies to better capture local and global contextual information, thus performing well in downstream tasks.

[0103] MiniLM is a lightweight pre-trained model designed to deliver performance close to that of large models like BERT while maintaining a small model size, making it suitable for resource-constrained scenarios.

[0104] Table 3 shows the experimental results analyzing the effects of adding cross-language query expansion methods under different multilingual models:

[0105] Table 3. Experimental results of cross-language query expansion between Chinese and Vietnamese languages ​​under different multilingual models.

[0106]

[0107] Tables 2 and 3 show the performance of the method of this invention on the Test2000 test set. Table 2 compares the performance of the method of this invention with other query expansion methods based on the same cross-language model, and Table 3 shows the experimental results under different cross-language pre-trained models. The method of this invention shows a certain improvement over other query expansion methods under the same multilingual model, and also shows different degrees of improvement under different models.

[0108] (1) Comparison of different query expansion methods under the same multilingual model: As shown in Table 2, the experimental results of Chinese-Vietnamese cross-language query expansion show that the method of this invention has improved performance compared with the baseline model query expansion methods. For cross-language query expansion, the monolingual query expansion method has a weak understanding of the target language, which makes Q2D and CoT methods perform poorly on the Chinese-Vietnamese cross-language query expansion dataset. Although the cross-language method CLCoT can improve performance to a certain extent, its effect is still not ideal due to the difficulty in capturing contextual relationships. The method of this invention injects the thought chain generation ability of the large-scale language model and the retrieved external knowledge into the multilingual pre-trained model with fewer parameters through knowledge distillation and retrieval enhancement. This not only improves the model's understanding of the target language, but also effectively utilizes contextual relationships, making it perform better in the Chinese-Vietnamese cross-language query expansion task.

[0109] (2) Effect Analysis of Adding Cross-Language Query Expansion Method under Different Multilingual Models: As shown in Table 3, the experimental results of Chinese-Vietnamese cross-language query expansion under different multilingual models 4-6 demonstrate that the method of this invention exhibits significant performance improvements across all multilingual models. When using different baseline models, adding the Chinese-Vietnamese cross-language query expansion method proposed in this invention improves performance to varying degrees. This indicates that the method of this invention has good adaptability and wide applicability, and can effectively improve the performance of Chinese-Vietnamese cross-language query expansion under various multilingual models.

[0110] To verify the effectiveness of the proposed method, an ablation experiment was designed, introducing different query expansion methods for comparative analysis. Here, w / CLCoT indicates the use of a cross-language thought chain on the baseline model, and w / Ours indicates the use of the method of this invention. Specific experimental results are shown in Table 4.

[0111] Table 4 Results of the Sino-Vietnamese cross-language query expansion and ablation experiment

[0112]

[0113] Analysis of the experimental results in Table 4 reveals that, firstly, introducing cross-linguistic thought chains as query expansion content significantly improves the performance compared to the baseline model. This indicates that in the Chinese-Vietnamese cross-linguistic query expansion task, utilizing the additional reasoning information generated for the original query by a large language model can significantly improve the accuracy of the retrieval results.

[0114] Secondly, when the method of this invention is introduced, the experimental results are significantly better than the cross-language thought chain method. This indicates that by injecting the thought chain generation capability of the large language model and the retrieved external knowledge into a multilingual pre-trained model with fewer parameters through knowledge distillation and retrieval enhancement, it helps to improve the performance of the Chinese-Vietnamese cross-language query expansion task. In summary, through comparative analysis of the ablation experiment results, this experiment verifies the effectiveness of different modules in the model.

[0115] Public dataset performance evaluation

[0116] To further verify the effectiveness and scalability of the proposed method, this experiment evaluated its performance on the MLQA and XQuAD datasets for cross-lingual question answering tasks from Chinese to Vietnamese (Zh-Vi) and from Vietnamese to Chinese (Vi-Zh). Tables 5 and 6 present the experimental results of the baseline model, the cross-lingual thought chain, and the proposed method on the MLQA and XQuAD datasets, respectively.

[0117] As shown in Tables 5 and 6, the experimental results demonstrate that the proposed method improves performance compared to the baseline model and the cross-lingual thought chain method. Specifically, on the MLQA dataset, the average NDCG and MRR for Vietnamese to Chinese queries are improved by 11.7% and 11.4%, respectively; on the XQuAD dataset, the average NDCG and MRR are improved by 7.4% and 8.3%, respectively. This indicates that the proposed method not only improves performance in the Chinese-Vietnamese cross-lingual query expansion task but also exhibits scalability in the Vietnamese-Chinese task. This fully demonstrates that when handling various cross-lingual query expansion tasks, the proposed method can more efficiently find cross-lingual answers and related documents.

[0118] Table 5 shows the experimental results of NDCG and MRR on the MLQA public dataset.

[0119]

[0120] Table 6 shows the experimental results of NDCG and MRR on the XQuAD public dataset.

[0121]

[0122] Furthermore, as shown in Tables 7 and 8, the method of the present invention significantly improves the MAP and Recall metrics for Vietnamese to Chinese on the MLQA and XQ-uAD datasets at all K values, indicating that it is more accurate in cross-language answer and related document retrieval.

[0123] Table 7 shows the experimental results of MAP and Recall on the MLQA public dataset.

[0124]

[0125] Table 8 shows the experimental results of NDCG and MRR on the XQuAD public dataset.

[0126]

[0127] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation, characterized by: The specific steps of the Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation are as follows: Step 1: Collect query document data and gather a Chinese-Vietnamese cross-language question-and-answer public dataset. Then, preprocess the data to form the experimental dataset. Step 2: Building on Step 1, knowledge distillation and retrieval enhancement are used to inject the thought chain generation capability of the large-scale language model and the retrieved external knowledge into the multilingual pre-trained model with fewer parameters, thereby improving its thought chain generation capability. Step 3: Based on Step 2, fine-tune the multilingual pre-trained model on the Chinese-Vietnamese retrieval dataset to improve the model's performance in Chinese-Vietnamese cross-language retrieval and enhance the retrieval effect of the Chinese-Vietnamese cross-language query extension task. Step 2 includes three new query expansion task modules: a thought chain generation module, a retrieval enhancement module, and a cross-language query expansion module; the specific steps are as follows: Step 2.1: The Mind Chain Generation Module constructs a zero-shot hint template for the original Chinese query and inputs it into a large language model to generate a zero-shot mind chain; it then uses the large language model as a teacher model to distill its knowledge into the student model. Step 2.2: The retrieval enhancement module obtains the k most relevant Vietnamese documents through a cross-language retrieval model; then, it inputs the Chinese query and relevant Vietnamese documents into the mT5 model for training using retrieval enhancement suggestion templates. The goal of training is to generate zero-shot thought chains and improve the inference and retrieval performance of the model by continuously iterating, correcting and improving the chain thinking after retrieval enhancement. Step 2.3: Using the query-mind chain data pairs constructed in Step 2.1 and Step 2.2, and integrating them with the top k Vietnamese-related documents retrieved through the original query, the original Chinese query is concatenated with the enhanced mind chain. Through vectorization processing of the query and documents, the most relevant Vietnamese documents are retrieved.

2. The Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.1: Data was collected from Wikipedia and Vietnam News Network, and a public dataset of cross-language questions and answers between Chinese and Vietnamese was collected. In order to complete the expansion of cross-language queries between Chinese and Vietnamese, several pairs of aligned cross-language query documents were finally constructed. Step 1.2: For Wikipedia and news website data, extract document data Dc and Dv from the Chinese and Vietnamese Wikipedia pages; obtain the Chinese query Qc(i) through the Chinese data document Dc, and calculate the similarity between the Chinese query Qc(i) and the Vietnamese document Dv(j) using cosine similarity through cross-language word embedding. If the embedding result is greater than the preset threshold β, then the Chinese query and the Vietnamese document are regarded as a similar language pair. This method realizes the alignment of related languages ​​between the Chinese query and the Vietnamese document. Step 1.3: For the Chinese-Vietnamese cross-language question-answering public dataset, extract the data from Chinese to Vietnamese (zh-vi) and Vietnamese to Chinese (vi-zh) from the Chinese-Vietnamese cross-language question-answering dataset, obtain the Chinese and Vietnamese queries Qc(i) and Qv(i) and the related Vietnamese and Chinese texts Dv(j) and Dc(j) through this method to construct the Chinese-Vietnamese query extension dataset.

3. The Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation according to claim 1, characterized in that: The specific steps of Step 2.1 are as follows: Step 2.1.1: Input for the Chinese-Vietnamese cross-language query expansion task It uses the generation of corresponding thought chains based on query statements and constructs data pairs between queries and thought chains accordingly; it designs a prompt template called PromptCoT, which can effectively guide large language models to generate the required thought chains. ； Step 2.1.2: Query-thought chain data pairs were constructed on the self-built dataset and two publicly available Han-Yue datasets; Chinese query statements were selected from the datasets. This information is then embedded into the PromptCoT prompt template to construct a complete prompt input. The prompt input is then passed to the large language model M to generate a Vietnamese thought chain corresponding to the Chinese query. The generated Vietnamese thought chains were matched with the original Chinese query statements to form query-thought chain data pairs needed for training and testing.

4. The Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation according to claim 1, characterized in that: The specific steps of Step 2.2 are as follows: Step 2.2.1: Based on Step 2.1, for the constructed query-mind chain data... It incorporates the top k Vietnamese-related documents retrieved through the original query. This was used to train a student model for cross-language query expansion; by combining retrieved Vietnamese documents with the original Chinese query, a search enhancement prompt template, PromptRAG, was designed. ； ； As a search enhancement suggestion template designed specifically for generation, it works by prioritizing the reading and incorporation of relevant fragments selected from a set of retrieved Vietnamese documents; This represents the encoding of the search enhancement suggestion corresponding to the i-th document among the first k retrieved documents; all encoded search enhancement suggestions... They are sequentially concatenated to form the overall contextual input for the cross-language model decoder; ； Step 2.2.2: During the training process, utilize the retrieved relevant context document set. To enhance the model's capabilities, these documents are used to train a cross-language model to generate pseudo-target inference. ; ； in This represents the probability of generating x under condition y. The ability of the large language model to generate thought chains is injected into the model through knowledge distillation, thereby optimizing its performance during the reasoning process. Cross-language models combine T and [PAD] to generate output. ; ； Output Appended to the original query This allows for the expansion of the query; 。 5. The Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation according to claim 1, characterized in that: The specific steps of Step 2.3 are as follows: Step 2.3.1, For the original Chinese query This query reflects the user's initial information needs; to further enrich the context of the query, a search-enhanced thought process was employed. To expand the query content; Subsequently, the original query and retrieval were enhanced. The content is concatenated and vectorized to convert it into a high-dimensional vector, which facilitates comparison from the perspective of semantic similarity; the cosine similarity method is used to calculate semantic similarity. 。 6. The Chinese-Vietnamese cross-lingual query expansion method based on retrieval enhancement and knowledge distillation according to claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1: Use the Chinese question portion of the self-built dataset as the query portion in the cross-language retrieval task, and use the corresponding Vietnamese documents as the relevant documents in the retrieval task; then use the question portion of the Chinese-Vietnamese cross-language question-and-answer dataset as the Chinese query, and use the correct Vietnamese answers and related text segments as the relevant documents in the retrieval; finally, obtain the evaluation dataset of cross-language queries and relevant documents. Step 3.2: Based on Step 3.1, Normalized Discount Cumulative Gain (NDCG), Mean Precision (MAP), Mean Reciprocal Rank (MRR), and Recall are used as evaluation metrics for the cross-language query expansion task. Step 3.3: Based on Step 3.1, optimizations were made for the Chinese-Vietnamese cross-language query expansion task. During the fine-tuning process, the Dropout size was 0.1, the batch size was 4, the maximum input length was 1024, the maximum output length was 256, and the model was trained for a total of 10 epochs. When the model obtained the best retrieval score on the validation set, the program evaluated and recorded the corresponding score on the test set, and finally selected the best score as the result and saved it.

Images