A cross-language information retrieval method and apparatus based on a vectorization model

Abstract

This disclosure provides a cross-language information retrieval method and apparatus based on a vectorization model. The method includes: receiving a retrieval request input by a user in the target language; converting the retrieval request into a semantic vector to be retrieved based on a multilingual embedding model; determining the M retrieval result vectors with the highest similarity to the semantic vector to be retrieved in a vector knowledge base, and feeding back the original text of the information entries corresponding to the retrieval result vectors to the user. This disclosure completes the learning and fine-tuning of the multilingual embedding model by constructing a set of positive and negative sample pairs in the target language, enabling the multilingual embedding model to construct a language-independent semantic space. This ensures that regardless of the target language used by the user, the converted semantic vector can be compared and matched with entries in the vector knowledge base within the same semantic space, ensuring semantic consistency of cross-language queries and significantly improving the accuracy of cross-language information retrieval.

Description

Technical Field

[0001] This disclosure relates to the fields of natural language processing and information retrieval technology, and in particular to a cross-language information retrieval method, apparatus, storage medium and electronic device based on a vectorization model. Background Technology

[0002] With the acceleration of globalization and the widespread adoption of the internet, information is experiencing explosive growth, and the languages ​​on which it is carried are becoming increasingly diverse. Users' demand for cross-language information retrieval is growing, with the expectation of being able to search in their familiar language and obtain documents or knowledge in different languages. Traditional monolingual information retrieval systems can no longer meet this increasing demand for cross-language information exchange.

[0003] Currently, there are several main technical approaches to achieving multilingual information retrieval (MIR) or cross-lingual information retrieval (CLIR):

[0004] (1) Query translation: Translate the language entered by the user from the source language into the target language, and then use the translated query to search in the document set of the target language;

[0005] (2) Document translation: Translate documents in the target language into the source language, and then use the source language query for retrieval;

[0006] (3) Intermediate representation: Convert both the query and the document into a language-independent intermediate representation, and then perform matching in this intermediate representation space.

[0007] While the methods described above have addressed the problem of cross-language information retrieval to some extent, they generally suffer from the following limitations:

[0008] (1) High cost and limited scalability of multilingual knowledge base construction: Traditional methods require building a high-quality knowledge base for each language separately, relying on a large amount of manually labeled data, which is costly and difficult to cover low-resource languages ​​and code-switching (mixed language) scenarios. For example, if it is necessary to support Spanish and Hindi retrieval, two separate knowledge bases need to be labeled, resulting in duplicate investment of resources.

[0009] (2) Over-reliance on translation quality: Existing methods rely on high-quality translation data to generate target language queries, but translation errors may introduce noise and affect the accuracy of retrieval.

[0010] (3) Limitations of negative sample sampling in contrastive learning: Traditional methods rely on a single sampling method of the most difficult negative example or random negative examples. The most difficult negative example only selects the negative examples with the most similar labels, which can easily lead to local overfitting and ignore global semantic consistency. Randomly sampled negative examples lack semantic discrimination ability and cannot strengthen the model's learning of subtle differences.

[0011] Therefore, existing technologies lack a universal method for efficient, low-cost cross-language information retrieval that can fully utilize existing knowledge base resources. Summary of the Invention

[0012] The purpose of this disclosure is to provide a cross-language information retrieval method, apparatus, storage medium, and electronic device based on a vectorization model, so as to solve the problems existing in the prior art.

[0013] The embodiments of this disclosure adopt the following technical solution: a cross-language information retrieval method based on a vectorization model, comprising: receiving a retrieval request input by a user in any target language; converting the retrieval request into a semantic vector to be retrieved based on a multilingual embedding model, wherein the multilingual embedding model is fine-tuned by comparative learning on a base model based on a set of positive and negative example samples; determining the M retrieval result vectors with the highest similarity to the semantic vector to be retrieved in a vector knowledge base, and feeding back the original text of the information entries corresponding to the retrieval result vectors to the user.

[0014] This disclosure also provides a cross-language information retrieval device based on a vectorization model, comprising: an interaction module for receiving a retrieval request input by a user in any target language; a retrieval embedding module for converting the retrieval request into a semantic vector to be retrieved based on a multilingual embedding model, wherein the multilingual embedding model is fine-tuned by comparative learning on a base model based on a set of positive and negative example samples; and a retrieval module for determining, in a vector knowledge base, the M retrieval result vectors with the highest similarity to the semantic vector to be retrieved, and providing the user with the original text of the information entries corresponding to the retrieval result vectors.

[0015] This disclosure also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described cross-language information retrieval method based on a vectorization model.

[0016] This disclosure also provides an electronic device, including at least a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of the above-described cross-language information retrieval method based on a vectorization model.

[0017] The beneficial effects of this disclosure are as follows: by constructing a set of positive and negative sample pairs of the target language to complete the learning and fine-tuning of the multilingual embedding model, the multilingual embedding model constructs a language-independent semantic space, so that no matter what language the user uses as the target language, its converted semantic vector can be compared and matched with the entries in the vector knowledge base in the same semantic space, ensuring the semantic consistency of cross-language queries and significantly improving the accuracy of cross-language information retrieval. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the cross-language information retrieval method based on a vectorization model in the first embodiment of this disclosure;

[0020] Figure 2 This is a schematic diagram illustrating the training steps of the multilingual embedding model in the first embodiment of this disclosure;

[0021] Figure 3 This is a schematic diagram of the cross-language information retrieval device based on a vectorization model in the first embodiment of this disclosure. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0023] To address the problems existing in the prior art, the first embodiment of this disclosure provides a cross-language information retrieval method based on a vectorization model, the flowchart of which is shown below. Figure 1 As shown, it mainly includes steps S10 to S30:

[0024] S10 receives a search request input by the user in any target language.

[0025] The cross-language information retrieval method provided in this embodiment can provide retrieval services to users through retrieval platforms, retrieval systems, and other implementing entities. The platform or system can receive user retrieval requests by providing an input-enabled retrieval interface as an interactive entry point. User retrieval requests can be input in any target language, specifically any retrieval language supported by the business scenario definition, such as Chinese, English, Russian, Japanese, Spanish, Korean, Arabic, etc., and can also include various less commonly spoken languages. In actual implementation, the interactive entry point can handle various forms of query input, such as text input and voice input.

[0026] S20, based on a multilingual embedding model, transforms the retrieval request into a semantic vector to be retrieved.

[0027] The multilingual embedding model in this embodiment is fine-tuned by performing comparative learning on the base model based on a set of positive and negative example samples. It is mainly used to transform retrieval requests in any language into semantic vectors in the semantic space, and to accurately express the content of the retrieval request through these semantic vectors.

[0028] Specifically, multilingual embedding models are based on, for example, Figure 2 The steps shown are from training:

[0029] S210, Obtain manually annotated monolingual datasets.

[0030] To train and fine-tune the multilingual embedding model, high-quality contrastive learning training data, i.e., pairs of positive and negative samples, is needed. This data primarily originates from a knowledge base and its manually annotated high-quality data, typically using high-resource languages ​​(e.g., Chinese, English, etc.). The monolingual dataset in this embodiment includes multiple data pairs, and its data structure can be as follows:

[0031]

[0032] in, This represents a sample query written in Chinese. The label represents the core semantic intent of the example query statement. These labels are manually defined to ensure they are highly relevant to the business scenario.

[0033] For example:

[0034] How can I reset my password?

[0035] Password reset

[0036] ("How can I reset my password?", "Password Reset")

[0037] S220 divides the monolingual dataset into a first subset and a second subset.

[0038] Based on a manually annotated monolingual dataset, it is divided into a first subset N1 and a second subset N2. N1 is used to build the index; therefore, N1 needs to cover a sufficient number of labels to ensure the diversity of the retrieval index. It includes multiple first data pairs, which are obtained by stratified sampling from the monolingual dataset according to the distribution ratio of the labels, ensuring that the distribution ratio of each label in the first subset N1 is consistent with the distribution ratio of the corresponding label in the monolingual dataset. The second subset N2 includes multiple second data pairs, mainly referring to other data pairs in the monolingual dataset besides N1. Because high-quality annotated data is usually scarce and expensive, it is necessary to maximize the use of limited annotated data; therefore, N2 is used to construct data for production-level comparative learning.

[0039] It is important to note that the distribution of N1 and N2 can be adjusted based on factors such as the amount of labeled data, the complexity of the target language, and computational resource constraints. The aim is to maximize the use of labeled data to generate diverse comparative learning samples while ensuring the quality of the retrieval index.

[0040] S230, Generate a set of positive sample pairs, a set of negative sample pairs, and a set of synthetic data pairs for each target language based on the second subset.

[0041] In this embodiment, the dataset used for comparative learning mainly includes a set of positive sample pairs, a set of negative sample pairs, and a set of synthetic data pairs. By learning from the above sample sets, the model enables different languages ​​to achieve effective alignment in the same semantic space.

[0042] The purpose of constructing the set of positive sample pairs is to force the model to map similar queries from different languages ​​to a shared vector space, achieving cross-lingual semantic alignment. This is primarily generated based on the following steps: For each example query statement in the second data pair... Example query statement Translate into the target language T to obtain the target language query statement. Then, randomly select an example query statement from the first subset N1 that corresponds to the second data pair. Example query statements with the first data pair having the same label are used as positive example query statements. and associate with the target language query statement. and positive example query statement To form a set of positive sample pairs .For example:

[0043] Example query (in Chinese): "How can I reset my password?"

[0044] Tag: Password Reset

[0045] Translated (in Spanish): “¿Cómo puedo restablecer mi contraseña”

[0046] N1 search terms with the same tag (in Chinese): "I need to change my password".

[0047] Correct example:

[0048] (“¿Cómo puedo restablecer mi contraseña”, “I need to change my password”)

[0049] For the set of negative sample pairs, the quality of the samples is crucial to the effectiveness of contrastive learning. In this embodiment, the set of negative sample pairs includes a set of random negative sample pairs and a set of difficult negative sample pairs. The set of random negative sample pairs mainly represents samples with completely unrelated labels, which are used for global semantic learning. The set of difficult negative sample pairs is for negative samples that are semantically similar to positive samples but are not completely matched. They are more difficult for the model to distinguish, forcing the model to learn more fine-grained semantic features.

[0050] Specifically, for each target language query statement Randomly select an example query statement from the first subset N1 that matches the second data pair. Example query statements of the first data pair with different labels as negative example query statements Related target language query statement Negative instance query statement This forms a set of random negative sample pairs.

[0051] For the set of difficult negative sample pairs, firstly, for each target language query statement... Calculate its tags With each label in the first subset N1 Similarity score Based on the similarity score, the first data pair in the first subset N1 is weighted and sampled. From the sampling results, k sample result data pairs are selected, and the example query statements in the k sample result data pairs are determined. Related target language query statement Example query statements for k sample result data pairs Forming a set of difficult negative samples .

[0052] Finally, a negative sample pair set is formed based on the constructed set of random negative sample pairs and the set of difficult negative sample pairs. By combining random negative samples and weighted sampling of difficult negative samples, it can be ensured that the model can effectively distinguish subtle semantic differences while learning a wide range of semantic differences, thereby improving the robustness and accuracy of the model.

[0053] For example:

[0054] Query (Spanish): “¿Cómo puedo restablecer mi contraseña”

[0055] Label Password reset

[0056] Random negative examples "How's the weather today?" (Tag: "Weather Inquiry")

[0057] Difficult negative examples "I forgot my login information" (tagged "account recovery");

[0058] "Can you help me recover my account information?" (Tag: "Account Recovery")

[0059] Negative example pairs:

[0060] (“¿Cómo puedo restablecer mi contraseña”, “How’s the weather today?”)

[0061] (“¿Cómo puedo restablecer mi contraseña”, “I forgot my login information?”)

[0062] (“¿Cómo puedo restablecer mi contraseña”, “Can you help me retrieve my account information?”)

[0063] Because unlabeled data is abundant while high-quality labeled examples in the knowledge base are scarce, synthetic data is needed to construct additional positive and negative pairs to enrich the training data for contrastive learning. Specifically, any large language model can be invoked to generate query statements in the target language. At least one composite positive query statement with the same semantics And generate a query statement in the target language. At least k semantically distinct composite negative query statements Based on the composite positive example query statement Synthetic negative instance query statement Form a set of synthetic data pairs; where the synthetic positive query statement is... Synthetic negative instance query statement The expression methods are all the same as those of the target language query statement. Different approaches are used to enhance the robustness of the model.

[0064] For example:

[0065] Query (Spanish): “¿Cómo puedo restablecer mi contraseña”

[0066] A positive example of compounding (in Spanish): “¿Cómo puedo recuperar mi contraseña”

[0067] Synthetic negative example (Spanish):

[0068] “¿Cuál es el clima hoy” (Tag: "Weather Inquiry")

[0069] “¿Cómo contacto al soporte técnico” (Tag: “Technical Support Inquiry”)

[0070] Synthetic data pairs:

[0071] ("¿Cómo puedo restablecer mi contraseña", "¿Cómo puedo recuperar micontraseña")

[0072] (“¿Cómo puedo restablecer mi contraseña”, “¿Cuál es el clima hoy”)

[0073] ("¿Cómo puedo restablecer mi contraseña", "¿Cómo contacto al soporte técnico")

[0074] By synthesizing datasets, the problem of scarce labeled data is solved, making it particularly suitable for low-resource languages ​​and code-switching scenarios. At the same time, it enables the model to generate diverse expressions, avoiding model dependence on specific translation models and enhancing model robustness.

[0075] It is important to note that the retrieval system can support multiple target languages. Therefore, when constructing the positive sample pair set, negative sample pair set, and synthetic data pair set, a corresponding data set should be generated for each target language to ensure that the model can learn the semantic understanding of all target languages.

[0076] S240. Select the multilingual Transformer embedding model as the base model, and fine-tune the base model by contrastive learning based on all positive sample pairs, negative sample pairs and synthetic data pairs until the contrastive loss is minimized, thus obtaining the multilingual embedding model.

[0077] This embodiment can use currently mainstream pre-trained multilingual Transformer embedding models as the base model, such as Qwen3-Embedding. This series of models is based on the dense base model of the Qwen3 series, inheriting its excellent multilingual capabilities and long text understanding capabilities, and performs well in text retrieval, code retrieval, text classification, text clustering, and bilingual text mining; or multilingual-e5: this series is a multilingual embedding model optimized for semantic search tasks, and performs well in multiple cross-language information retrieval benchmark tests. In actual implementation, the selection of a suitable model needs to consider the language types of the knowledge base, computational resources, and expected performance.

[0078] After selecting the base model, the base model is trained using the prepared sets of positive sample pairs, negative sample pairs, and synthetic data pairs. The training objective is to minimize the contrastive loss function. In this embodiment, InfoNCE Loss is used, and its formula is as follows:

[0079]

[0080] in, Indicates the query anchor point. Indicates a pair of positive samples. Indicates negative sample pairs, This represents the temperature parameter. The loss function aims to maximize the similarity between positive sample pairs while minimizing the similarity between positive and negative samples. In this way, the model can learn to map semantically similar cross-lingual queries to nearby locations in the embedding space.

[0081] exist When the minimum value is reached, the model training is considered complete, resulting in a multilingual embedding model that accurately converts the user's input natural language retrieval requirements into the semantic vector to be retrieved.

[0082] S30. In the vector knowledge base, determine the M search result vectors with the highest similarity to the semantic vector to be searched, and provide the user with the original text of the information entries corresponding to the search result vectors.

[0083] When a user performs an information retrieval, the multilingual embedding model first transforms the retrieval request into a semantic vector to be retrieved. Then, the retrieval system compares the similarity of the semantic vector to be retrieved with all knowledge semantic vectors in the currently constructed vector knowledge base. The system then uses the M knowledge semantic vectors with the highest similarity as the retrieval result vectors and returns the original text of the information entries corresponding to the retrieval result vectors to the user. The value of M can be set according to actual retrieval needs, including but not limited to setting it according to a user-defined number, using the number of similarity calculation results higher than the preset similarity threshold as the value of M, or having the system set a default positive integer value as the value of M.

[0084] In some embodiments, the vector knowledge base is also constructed based on a trained multilingual embedding model. For the original text of each information entry in the preset knowledge base (e.g., medical knowledge base, computer knowledge base, patent database, etc.), the multilingual embedding model is used to vectorize the original text, resulting in a fixed-dimensional semantic vector. To achieve efficient similarity retrieval, these vectors need to be stored in a preset vector index structure to form a vector knowledge base. The preset vector index library can be selected according to the business scenario. Commonly used vector index libraries include: Faiss (Facebook AI Similarity Search), a library for efficient similarity search and clustering, supporting multiple index types, including exact search and Approximate Nearest Neighbor (ANN) search, suitable for large-scale datasets; Milvus, an open-source vector database designed for vector data management, storage, and retrieval in AI applications. It can be used not only as open-source software but also as a cloud service. Milvus can store, index, and manage massive amounts of vector embeddings and supports efficient similarity search, mainly used in knowledge base, semantic search, and retrieval augmentation generation (RAG) scenarios.

[0085] After receiving the semantic vector to be retrieved output by the multilingual embedding model, the retrieval system can determine the M knowledge semantic vectors with the highest similarity to the semantic vector to be retrieved from the vector knowledge base as the retrieval result vector through similarity calculation. The similarity measurement method used in this embodiment is cosine similarity, which has a good performance in capturing semantic similarity. Its calculation formula is as follows:

[0086]

[0087] in, For two vectors that need to be similar (e.g., the semantic vector to be retrieved and one of the knowledge semantic vectors in the knowledge base). It is their dot product. and It refers to their L2 norm. In actual implementation, other similarity measurement methods, such as Euclidean distance and Manhattan distance, can be selected according to different business scenarios. This embodiment does not impose any restrictions.

[0088] It should be noted that the result obtained after the above similarity calculation is still a semantic vector. Therefore, before providing the search results to the user, the original text of the information entry corresponding to the search result vector should be determined, and the original text of the information entry should be sorted in descending order of similarity before being provided to the user for viewing.

[0089] In some embodiments, after obtaining the original text of the information entries corresponding to the search result vector, it can be detected whether the language of the original text of each information entry is the same as the target language when the user inputs the search request. If the language of the original text of the information entry is the same as the target language, it means that the user can directly understand the search result content, and the original text of the information entry can be returned directly. If the language of the original text of the information entry is different from the target language, it means that the user may not be able to directly understand the search result content. In this case, it can be translated by machine translation service or large language model, and the translated original text of the information entry in the target language is returned to the user for easy viewing. For example, if the user uses English to search, but the search result is in Chinese, the Chinese content can be translated into English before being presented to the customer.

[0090] This embodiment fine-tunes the multilingual embedding model by constructing a set of positive and negative sample pairs in the target language. This allows the multilingual embedding model to build a language-independent semantic space, ensuring that regardless of the user's target language, the converted semantic vectors can be compared and matched with entries in the vector knowledge base within the same semantic space. This guarantees semantic consistency in cross-language queries and significantly improves the accuracy of cross-language information retrieval. Furthermore, this embodiment supports low-resource language and code-switching (such as mixed language) scenarios without requiring labeled data in the target language. By treating code-switching as an independent language design, it directly generates training data using a monolingual knowledge base, eliminating the need for additional annotation and saving annotation costs. It also avoids translation dependence; even if translation errors exist, the model can still learn robust semantic representations through comparative learning, making it suitable for resource-constrained business scenarios.

[0091] Based on the same inventive concept, the second embodiment of this disclosure provides a cross-language information retrieval device based on a vectorization model. This device can be installed in any retrieval platform or system, serving as the core function of the retrieval platform or system. Its structural schematic diagram is shown below. Figure 3 As shown, it mainly includes: an interaction module 10, used to receive retrieval requests input by users in any target language; a retrieval embedding module 20, used to convert the retrieval request into a semantic vector to be retrieved based on a multilingual embedding model, the multilingual embedding model being fine-tuned and trained by comparative learning of a base model based on a set of positive and negative example samples; and a retrieval module 30, used to determine the M retrieval result vectors with the highest similarity to the semantic vector to be retrieved in the vector knowledge base, and to provide the user with the original text of the information entries corresponding to the retrieval result vectors.

[0092] In some embodiments, the multilingual embedding model is trained based on the following steps: Obtaining a manually annotated monolingual dataset, which includes multiple data pairs, each including an example query and a label, the label representing the core semantic intent of the example query; dividing the monolingual dataset into a first subset and a second subset; wherein the first subset includes multiple first data pairs, which are obtained by stratified sampling from the monolingual dataset according to the distribution ratio of the labels, so that the distribution ratio of each label in the first subset is consistent with the distribution ratio of the corresponding label in the monolingual dataset; the second subset includes multiple second data pairs, which are other data pairs in the monolingual dataset excluding the first subset; generating a set of positive sample pairs, a set of negative sample pairs, and a set of synthetic data pairs for each target language based on the second subset; selecting a multilingual Transformer embedding model as the base model, and fine-tuning the base model through contrastive learning based on all positive sample pairs, negative sample pairs, and synthetic data pairs until the contrastive loss is minimized, thus obtaining the multilingual embedding model.

[0093] In some embodiments, the set of positive sample pairs is generated based on the following steps: for each second data pair, the sample query statement... Example query statement Translate into the target language T to obtain the target language query statement. Randomly select an example query statement from the first subset that matches the second data pair. Example query statements with the first data pair having the same label are used as positive example query statements. and associate with the target language query statement. and positive example query statement To form a set of positive sample pairs .

[0094] In some embodiments, the negative sample set is generated based on the following steps: for each target language query statement Randomly select an example query statement from the first subset that corresponds to the second data pair. Example query statements of the first data pair with different labels as negative example query statements Related target language query statement Negative instance query statement This forms a set of random negative sample pairs; for each target language query statement Calculate its tags With each tag in the first subset Similarity score Based on the similarity score, the first data pair in the first subset is weighted and sampled. From the sampling results, k sampled data pairs are selected, and the example query statements in the k sampled data pairs are determined. Related target language query statement Example query statements for k sample result data pairs Forming a set of difficult negative samples A negative sample pair set is formed based on the set of random negative sample pairs and the set of difficult negative sample pairs.

[0095] In some embodiments, the synthetic data pair set is generated based on the following steps: generating a query statement in the target language. At least one composite positive query statement with the same semantics And generate a query statement in the target language. At least k semantically distinct composite negative query statements Based on the composite positive example query statement Synthetic negative instance query statement Form a set of synthetic data pairs; where the synthetic positive query statement is... Synthetic negative instance query statement The expression methods are all the same as those of the target language query statement. different.

[0096] In some embodiments, the vector knowledge base is constructed based on the following steps: vectorizing the original text of each information entry in the preset knowledge base through a multilingual embedding model to obtain a fixed-dimensional knowledge semantic vector; storing the knowledge semantic vector based on a preset vector index structure to form a vector knowledge base.

[0097] In some embodiments, the retrieval module 30 is specifically used to detect whether the language of the original text of each information entry is the same as the target language input by the user; if the language of the original text of the information entry is the same as the target language, the original text of the information entry is directly returned; if the language of the original text of the information entry is different from the target language, the original text of the information entry is translated into the target language and the translated original text of the information entry is returned to the user.

[0098] It should be noted that after the multilingual embedding model and vector knowledge base are built, the retrieval embedding module 20 and the retrieval module 30 can be directly called without repeated training for each retrieval.

[0099] This embodiment fine-tunes the multilingual embedding model by constructing a set of positive and negative sample pairs in the target language. This allows the multilingual embedding model to build a language-independent semantic space, ensuring that regardless of the user's target language, the converted semantic vectors can be compared and matched with entries in the vector knowledge base within the same semantic space. This guarantees semantic consistency in cross-language queries and significantly improves the accuracy of cross-language information retrieval. Furthermore, this embodiment supports low-resource language and code-switching (such as mixed language) scenarios without requiring labeled data in the target language. By treating code-switching as an independent language design, it directly generates training data using a monolingual knowledge base, eliminating the need for additional annotation and saving annotation costs. It also avoids translation dependence; even if translation errors exist, the model can still learn robust semantic representations through comparative learning, making it suitable for resource-constrained business scenarios.

[0100] Based on the same inventive concept, the third embodiment of this disclosure provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the cross-language information retrieval method based on a vectorization model described in the first embodiment of this disclosure.

[0101] Based on the same inventive concept, the fourth embodiment of this disclosure provides an electronic device, including at least a memory and a processor. The memory stores a computer program, and when the processor executes the computer program in the memory, it implements the steps of the cross-language information retrieval method based on a vectorization model described in the first embodiment of this disclosure.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A cross-language information retrieval method based on a vectorization model, characterized in that, include: It can receive search requests from users in any target language; The retrieval request is transformed into a semantic vector to be retrieved based on a multilingual embedding model, which is fine-tuned by performing comparative learning on a base model based on a set of positive and negative example samples. In the vector knowledge base, the M search result vectors with the highest similarity to the semantic vector to be searched are determined, and the original text of the information entries corresponding to the search result vectors is fed back to the user. The multilingual embedding model is trained based on the following steps: Obtain manually annotated monolingual datasets, which include multiple data pairs. Each data pair includes an example query statement and a label, where the label is used to characterize the core semantic intent of the example query statement. The monolingual dataset is divided into a first subset and a second subset; wherein, the first subset includes multiple first data pairs, which are obtained by stratified sampling from the monolingual dataset according to the distribution ratio of the labels, so that the distribution ratio of each label in the first subset is consistent with the distribution ratio of the corresponding label in the monolingual dataset; the second subset includes multiple second data pairs, which are other data pairs in the monolingual dataset excluding the first subset; Generate a set of positive sample pairs, a set of negative sample pairs, and a set of synthetic data pairs for each of the target languages ​​based on the second subset; A multilingual Transformer embedding model is selected as the base model, and the base model is fine-tuned through contrastive learning based on all the positive sample pairs, the negative sample pairs, and the synthetic data pairs until the contrastive loss is minimized, thus obtaining the multilingual embedding model. ； The set of positive sample pairs is generated based on the following steps: For each of the example query statements in the second data pair The example query statement Translate into the target language T to obtain the target language query statement. ; Randomly select an example query statement from the first subset that matches the second data pair. Example query statements with the first data pair having the same label are used as positive example query statements. and associate it with the target language query statement. and the positive example query statement To form a set of positive sample pairs ; The negative example sample set is generated based on the following steps: For each target language query statement Randomly select an example query statement from the first subset that corresponds to the second data pair. Example query statements of the first data pair with different labels as negative example query statements Associated with the target language query statement and the negative example query statement This forms a set of random negative sample pairs; For each target language query statement Calculate its tags With each tag in the first subset Similarity score ; Based on the similarity score, a weighted sampling is performed on the first data pair in the first subset. From the sampling results, k sample result data pairs are selected, and example query statements are determined from the k sample result data pairs. Associated with the target language query statement Example query statements for k sample result data pairs Forming a set of difficult negative samples ; The negative sample pair set is formed based on the random negative sample pair set and the difficult negative sample pair set; The synthetic data set is generated based on the following steps: Generate a query statement in the target language. At least one composite positive query statement with the same semantics And generate a query statement in the target language. At least k semantically distinct composite negative query statements According to the synthesized positive example query statement and the synthesized negative example query statement Form the synthetic data pair set; wherein, the synthetic positive example query statement and the synthesized negative example query statement The expression methods are all the same as the target language query statement. different.

2. The cross-language information retrieval method according to claim 1, characterized in that, The vector knowledge base is constructed based on the following steps: The original text of each information entry in the preset knowledge base is vectorized using the multilingual embedding model to obtain a fixed-dimensional knowledge semantic vector. The knowledge semantic vectors are stored based on a preset vector index structure to form the vector knowledge base.

3. The cross-language information retrieval method according to claim 1 or 2, characterized in that, The system provides the user with the original text of the information entries corresponding to the search result vector, including: Detect whether the language of the original text of each information entry is the same as the target language input by the user; If the language of the original text of the information entry is the same as the target language, the original text of the information entry is returned directly. If the language of the original text of the information entry is different from the target language, the original text of the information entry is translated into the target language, and the translated original text of the information entry is returned to the user.

4. A cross-language information retrieval device based on a vectorization model, characterized in that, include: The interaction module is used to receive search requests from users in any target language; The retrieval embedding module is used to transform the retrieval request into a semantic vector to be retrieved based on a multilingual embedding model, wherein the multilingual embedding model is fine-tuned by performing comparative learning on a base model based on a set of positive and negative example samples. The retrieval module is used to determine the M retrieval result vectors with the highest similarity to the semantic vector to be retrieved in the vector knowledge base, and to return the original text of the information entries corresponding to the retrieval result vectors to the user; The multilingual embedding model is trained based on the following steps: Obtain manually annotated monolingual datasets, which include multiple data pairs. Each data pair includes an example query statement and a label, where the label is used to characterize the core semantic intent of the example query statement. The monolingual dataset is divided into a first subset and a second subset; wherein, the first subset includes multiple first data pairs, which are obtained by stratified sampling from the monolingual dataset according to the distribution ratio of the labels, so that the distribution ratio of each label in the first subset is consistent with the distribution ratio of the corresponding label in the monolingual dataset; the second subset includes multiple second data pairs, which are other data pairs in the monolingual dataset excluding the first subset; Generate a set of positive sample pairs, a set of negative sample pairs, and a set of synthetic data pairs for each of the target languages ​​based on the second subset; A multilingual Transformer embedding model is selected as the base model, and the base model is fine-tuned through contrastive learning based on all the positive sample pairs, the negative sample pairs, and the synthetic data pairs until the contrastive loss is minimized, thus obtaining the multilingual embedding model. ； The set of positive sample pairs is generated based on the following steps: For each of the example query statements in the second data pair The example query statement Translate into the target language T to obtain the target language query statement. ; Randomly select an example query statement from the first subset that matches the second data pair. Example query statements with the first data pair having the same label are used as positive example query statements. and associate it with the target language query statement. and the positive example query statement To form a set of positive sample pairs ; The negative example sample set is generated based on the following steps: For each target language query statement Randomly select an example query statement from the first subset that corresponds to the second data pair. Example query statements of the first data pair with different labels as negative example query statements Associated with the target language query statement and the negative example query statement This forms a set of random negative sample pairs; For each target language query statement Calculate its tags With each tag in the first subset Similarity score ; Based on the similarity score, a weighted sampling is performed on the first data pair in the first subset. From the sampling results, k sample result data pairs are selected, and example query statements are determined from the k sample result data pairs. Associated with the target language query statement Example query statements for k sample result data pairs Forming a set of difficult negative samples ; The negative sample pair set is formed based on the random negative sample pair set and the difficult negative sample pair set; The synthetic data set is generated based on the following steps: Generate a query statement in the target language. At least one composite positive query statement with the same semantics And generate a query statement in the target language. At least k semantically distinct composite negative query statements According to the synthesized positive example query statement and the synthesized negative example query statement Form the synthetic data pair set; wherein, the synthetic positive example query statement and the synthesized negative example query statement The expression methods are all the same as the target language query statement. different.

5. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the cross-language information retrieval method based on a vectorization model as described in any one of claims 1 to 3.

6. An electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program on the memory, it implements the steps of the cross-language information retrieval method based on a vectorization model as described in any one of claims 1 to 3.

Images