Retrieval enhancement generation method and system supporting multi-language knowledge base

Abstract

The present application discloses a retrieval enhancement generation method and system supporting a multilingual knowledge base, the method comprising: obtaining a document to be processed; identifying the language category of the document to be processed, the document to be processed contains text in a first language, and the text in the first language accounts for the largest proportion in the document to be processed; using a document splitting model for the first language to split the document to be processed to obtain a plurality of text segments; identifying the language category in the first text segment, the first text segment contains text in a second language, and the proportion of the text in the second language in the first text segment is greater than a preset threshold, and the second language is the same as or different from the first language; storing the first text segment in a language knowledge base corresponding to the second language. The present application can better preserve the semantic integrity of the text segment, improve the text understanding effect, and greatly improve the accuracy and efficiency of multilingual document processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation method and system supporting a multilingual knowledge base. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval (Retrieval) and natural language generation (Generation), and is often used to enhance the performance of language models, especially when dealing with tasks that require external knowledge or contextual information.

[0003] In the related art, in the scenario of multilingual knowledge base question and answer, there are mainly two ways of information retrieval: one is a retrieval method based on a probability model, and the other is a retrieval method based on a dense vector. However, the retrieval method based on a probability model is usually based on an independent dictionary and document matrix, and lacks a unified cross-language representation when establishing connections between different languages, which makes it difficult to accurately calculate the semantic similarity between multiple languages. In addition, since the probability model method is mainly based on word frequency and position, it is difficult to distinguish the polysemy of the same word in different contexts. For some low-resource languages, the retrieval method based on dense vectors will affect the performance of the dense vector model due to insufficient training data. At the same time, since the vocabulary and concepts in different languages ​​need to be aligned, the vocabulary of different languages ​​with the same meaning needs to be mapped to the same vector space, resulting in inconsistencies in the migration effect between different languages, especially between language pairs with large differences in structure and grammar. In addition, with the addition of new data and new languages, it may be necessary to retrain or fine-tune the dense vector model, resulting in relatively complex updates and maintenance. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the purpose of the present application is to provide a retrieval enhancement generation method supporting a multilingual knowledge base, which can improve the accuracy and efficiency of multilingual document processing.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a retrieval enhancement generation method supporting a multilingual knowledge base, the method comprising the following steps: obtaining a document to be processed; identifying a language category of the document to be processed, wherein the document to be processed contains text in a first language, and the text in the first language is the language text with the largest proportion in the document to be processed; splitting the document to be processed using a document splitting model for the first language to obtain a plurality of text segments; identifying a language category in a first text segment, wherein the first text segment contains text in a second language, and the proportion of the text in the second language in the first text segment is greater than a preset threshold, and the second language is the same as or different from the first language; and storing the first text segment in a language knowledge base corresponding to the second language.

[0007] Furthermore, the method also includes: in response to the first text segment including texts in at least two languages, and the proportions of the texts in the two languages ​​in the first text segment are greater than the preset threshold, storing the first text segment in at least two language knowledge bases, wherein any of the language knowledge bases corresponds to a language whose proportion is greater than the preset threshold.

[0008] Furthermore, the method further comprises: extracting keywords from the first text segment using a word segmenter for the second language, and associating the extracted keywords with the first text segment and storing them in the language knowledge base.

[0009] Furthermore, the method also includes: using a vectorization model for the second language to vectorize the first text segment to generate a dense vector array, and associating the dense vector array with the first text segment and storing it in the language knowledge base.

[0010] Furthermore, the method also includes: obtaining language types of multiple language knowledge bases; translating questions into several target languages, any one of the target languages ​​corresponding to a language type of the acquired language knowledge base; searching in the corresponding language knowledge base based on the questions translated into the target language to extract text fragments related to the questions, and generating answers to the questions based on the text fragments.

[0011] Furthermore, the method also includes: using a word segmenter for the target language to extract keywords from the question to obtain question keywords; based on the question keywords, using a probability model to search in a corresponding language knowledge base to extract text fragments related to the question.

[0012] Furthermore, the method also includes: vectorizing the question using a vectorization model for the target language to generate a dense vector array about the question; based on the dense vector array about the question, searching in the corresponding language knowledge base using a dense vector model to extract text fragments related to the question.

[0013] Furthermore, the method also includes: using a probability model to search in a language knowledge base corresponding to the target language, obtaining the probability similarity of each text fragment with respect to the question, and sorting the text fragments according to the probability similarity; using a dense vector model to search in a language knowledge base corresponding to the target language, obtaining the dense vector similarity of each text fragment with respect to the question, and sorting the text fragments according to the dense vector similarity; using a reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results and the probability similarity sorting results, and calculating the comprehensive score of each text fragment; screening a preset number of text fragments with high comprehensive scores, and generating an answer to the question based on the screened text fragments.

[0014] Furthermore, the method also includes: using a Rerank model to search in a language knowledge base corresponding to the target language, obtaining the Rerank similarity of each text fragment with respect to the question, and sorting the text fragments according to the Rerank similarity; using a reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results, the probability similarity sorting results, and the Rerank similarity sorting results, and calculating a comprehensive score for each text fragment; screening a preset number of text fragments with high comprehensive scores, and generating an answer to the question based on the screened text fragments.

[0015] In a second aspect, the present application further provides a retrieval enhancement generation system supporting a multilingual knowledge base, wherein the system adopts the retrieval enhancement generation method supporting a multilingual knowledge base as described in any one of the first aspects.

[0016] The above-mentioned retrieval enhancement generation method and system supporting multilingual knowledge bases obtains a document to be processed, and identifies the language category of the document to be processed, takes the text with the largest proportion in the document to be processed as the text of the first language, and uses the document segmentation model for the first language to split the document to be processed to obtain a plurality of text segments; further, the language category in the first text segment is identified, and the text with a proportion greater than a preset threshold in the first text segment is taken as the text of the second language, and finally the first text segment is stored in the language knowledge base corresponding to the second language, so as to achieve accurate segmentation of the document to be processed by using the grammatical rules and punctuation marks of various languages, and to aggregate documents of the same language into independent language knowledge bases, thereby avoiding the problem that a single language knowledge base cannot accurately split the text segments and affects the semantic integrity and coherence of the text segments, thereby achieving more accurate segmentation of the text segments, better retaining the semantic integrity of the text segments, improving the text understanding effect, and greatly improving the accuracy and efficiency of multilingual document processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A first flow chart of a retrieval enhancement generation method supporting a multilingual knowledge base provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of storing a document to be processed into a language knowledge base provided in an embodiment of the present application;

[0019] Figure 3 A second flow chart of the retrieval enhancement generation method supporting a multilingual knowledge base provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of generating an answer to a question based on a question provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram provided in an embodiment of the present application supports a third flow chart of a method for enhancing retrieval of a multilingual knowledge base;

[0022] Figure 6 A schematic diagram provided in an embodiment of the present application supports a fourth flow chart of a retrieval enhancement generation method for a multilingual knowledge base. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the specific implementation manner of the present application will be clearly and completely described below in conjunction with the drawings in the implementation manner of the present application.

[0024] In the related art, in the scenario of multilingual knowledge base question and answer, there are mainly two ways of information retrieval: one is the retrieval method based on the probability model, which measures the importance of a word in the document and the entire document set based on word frequency, document length and word distribution in the document set. However, the retrieval method based on the probability model is usually based on independent dictionaries and document matrices, and lacks a unified cross-language representation when establishing connections between different languages, which makes it difficult to accurately calculate the semantic similarity between multiple languages. In addition, since the probability model method is mainly based on word frequency and position, it is difficult to distinguish the polysemy of the same word in different contexts.

[0025] The other is a dense vector-based retrieval method, which uses a pre-trained dual-tower model (dual encoder) to encode queries and documents into dense vectors, and then retrieves relevant documents through vector similarity (such as cosine similarity). However, for some low-resource languages, the performance of the model may be affected by insufficient training data for dense vector-based retrieval methods. At the same time, since vocabulary and concepts in different languages ​​need to be aligned, different language vocabulary with the same meaning needs to be mapped to the same vector space, resulting in inconsistencies in the migration effect between different languages, especially between language pairs with large differences in structure and grammar. In addition, with the addition of new data and new languages, the model may need to be retrained or fine-tuned, making updates and maintenance relatively complex.

[0026] The retrieval enhancement generation method supporting multilingual knowledge base provided by the present application includes: obtaining a document to be processed; identifying the language category of the document to be processed, the document to be processed contains text in a first language, and the proportion of the text in the first language in the document to be processed is the largest; using a document splitting model for the first language to split the document to be processed to obtain multiple text segments; identifying the language category in the first text segment, the first text segment contains text in a second language, and the proportion of the text in the second language in the first text segment is greater than a preset threshold, and the second language is the same as or different from the first language; storing the first text segment in a language knowledge base corresponding to the second language. This method adopts a variety of document splitting models, each of which accurately splits the document to be processed by using the grammatical rules and punctuation marks of the corresponding language, and collects documents of the same language into an independent language knowledge base, avoiding the single language knowledge base from being unable to accurately split the text segment, affecting the semantic integrity and coherence of the text segment, thereby achieving more accurate text segmentation, better retaining the semantic integrity of the text segment, improving the text understanding effect, and greatly improving the accuracy and efficiency of multilingual document processing.

[0027] The retrieval enhancement generation method supporting a multilingual knowledge base of the present application is described in detail below with reference to the accompanying drawings.

[0028] The present application embodiment provides a retrieval enhancement generation method supporting a multilingual knowledge base, referring to Figure 1 and Figure 2 , the method specifically comprises the following steps:

[0029] Step S101: Obtain documents to be processed.

[0030] Among them, the documents to be processed include product specifications, product manuals or various documents that may be used in various question-and-answer scenarios. The documents to be processed can be in the form of but not limited to e-books, web pages, reports, articles, etc., and can be obtained from databases, file systems, and network resources.

[0031] Step S102: identifying the language category of the document to be processed, wherein the document to be processed includes text in a first language, and the text in the first language accounts for the largest proportion in the document to be processed.

[0032] The language category indicates the language used by characters, words, etc. in the document to be processed, such as Chinese, English, French, etc. The document to be processed can be scanned by a language recognition tool to determine the language category of the document to be processed.

[0033] After identifying that the document to be processed contains texts in multiple languages, the frequency of occurrence of each language and the proportion of each language in the document to be processed are counted. In the embodiment of the present application, the language of the text with the largest proportion in the document to be processed is defined as the first language.

[0034] It should be noted that for longer documents to be processed, local document sampling can be used to first divide the documents to be processed into fixed lengths, and then a part of each document is intercepted as a sample document, and the frequency of occurrence of various languages ​​in the sample document is counted to determine the first language of the document to be processed. Furthermore, a large language model can be used to determine the first language of the sample document to improve the recognition accuracy of multilingual documents.

[0035] Step S103: Split the document to be processed using the document splitting model for the first language to obtain a plurality of text segments.

[0036] Specifically, multiple document splitting models are provided, each of which can adapt to document splitting in at least one language. Using the document splitting model for the first language, the document can be split into multiple text fragments according to the grammatical rules and punctuation marks of the first language. The text fragments can be words, sentences, paragraphs or text blocks divided according to specific rules.

[0037] Step S104: identifying the language category in the first text segment, the first text segment contains text in a second language, and the proportion of the text in the second language in the first text segment is greater than a preset threshold, and the second language is the same as or different from the first language.

[0038] Specifically, in addition to the first language category, the first text segment may also contain other languages, so the language category in the first text segment is identified. For ease of explanation, in the embodiment of the present application, the text whose proportion in the first text segment is greater than the preset threshold is defined as text in the second language, and the second language can be the same as or different from the first language. The preset threshold is used to determine whether a certain language in the first text segment is the main language of the first text segment. For example, the preset threshold can be set according to the maximum value of the proportion in the first text segment, or according to an empirical value.

[0039] For example, in the first text segment {“Xiao Ming’s favorite movie is Inception”}, it can be seen that in this case, the second language in the first text segment is Chinese, which is the same as the first language.

[0040] For example, in the first text segment {“Xiao Ming’s favorite movie is inception”}, it can be seen that in this case, the second language in the first text segment is English, which is different from the first language.

[0041] Step S105: store the first text segment in a language knowledge base corresponding to the second language.

[0042] Specifically, for each text segment, according to the identified second language, the text segment is stored in the corresponding language knowledge base. The knowledge base can be a database, a document storage system or other form of data storage facility, which is used to store and process data in a specific language and ensure that the text segment is correctly classified and stored in the language knowledge base.

[0043] According to the above description, the retrieval enhancement generation method of this embodiment obtains a document to be processed, and identifies the language category of the document to be processed, takes the text with the largest proportion in the document to be processed as the text of the first language, and uses the document segmentation model for the first language to split the document to be processed to obtain a plurality of text segments; further, the language category in the first text segment is identified, and the text with a proportion greater than a preset threshold in the first text segment is taken as the text of the second language, and finally the first text segment is stored in the language knowledge base corresponding to the second language, so as to realize the use of grammatical rules and punctuation marks of various languages ​​to accurately split the document to be processed, and to aggregate documents of the same language into independent language knowledge bases, so as to avoid the single language knowledge base being unable to accurately split the text segments and affecting the semantic integrity and coherence of the text segments, thereby achieving more accurate segmentation of the text segments, better retaining the semantic integrity of the text segments, improving the text understanding effect, and greatly improving the accuracy and efficiency of multilingual document processing.

[0044] In one embodiment, the retrieval enhancement generation method supporting multilingual knowledge bases also includes: in response to the first text segment including texts in at least two languages, and the proportions of the texts in the two languages ​​in the first text segment are both greater than a preset threshold, storing the first text segment in at least two language knowledge bases, wherein any language knowledge base corresponds to a language whose proportion is greater than the preset threshold.

[0045] Exemplarily, assuming that the first text segment contains Chinese, English and French, and the proportions of texts in the three languages ​​in the first text segment are 40%, 40% and 30% respectively, if the preset threshold is 25%, then the proportions of Chinese, English and French are all greater than the preset threshold, and the first text segment is stored in the Chinese knowledge base, English knowledge base, and French knowledge base respectively.

[0046] In this way, it can ensure that multilingual text fragments are stored accurately and completely, and can provide more reasonable applications in language analysis and information retrieval scenarios.

[0047] In one embodiment, the retrieval enhancement generation method supporting a multilingual knowledge base further includes: extracting keywords from the first text segment using a word segmenter for the second language, and associating the extracted keywords with the first text segment and storing them in the language knowledge base.

[0048] Among them, keywords are words that can represent the main content or intention of a text segment. Keyword extraction can be but is not limited to using TF-IDF keyword extraction algorithm, LDA topic model keyword extraction algorithm, mutual information keyword extraction algorithm, etc. The keyword extraction process refers to relevant documents and will not be repeated here.

[0049] A tokenizer is a language-specific tokenizer that breaks text into words or other language units according to the grammatical rules of a particular language.

[0050] Exemplarily, the first text segment is "Today's weather is sunny, the temperature is suitable, and outings, mountain climbing, and fishing are recommended", and the second language is Chinese. The Chinese word segmenter is used to segment the first text segment into a series of words or phrases {"Today's weather is sunny", "The temperature is suitable", "I recommend outings", "Mountain climbing", "Fishing"} according to the keywords. Then, the keyword extraction algorithm is combined to extract the keywords {"weather", "temperature", "recommendation", "outing", "mountain climbing", "fishing"}. Then, the extracted keywords are associated with the first text segment. This association can be a simple metadata tag, or a more complex key-value pair or graph structure. The associated keywords are stored in the language knowledge base corresponding to the second language to ensure that the text segments and keywords are effectively classified and stored, so that the related information can be quickly obtained during subsequent retrieval and analysis.

[0051] Similarly, if the second language is English, an English word segmenter is used to avoid the problem of being unable to accurately extract keywords due to the use of a single extractor. This embodiment improves the accuracy of keyword extraction by utilizing the language of the acquired text fragment and then combining the word segmenter corresponding to the language to extract keywords.

[0052] In one embodiment, the retrieval enhancement generation method supporting a multilingual knowledge base also includes: vectorizing the first text segment using a vectorization model for a second language to generate a dense vector array, and associating the dense vector array with the first text segment and storing it in the language knowledge base.

[0053] Among them, the vectorization model uses a semantic vector model (Embedding Model), such as the BCEmbedding model, the BGEEmbedding model, and the M3E model.

[0054] Exemplarily, assume that the first text segment is {"Mia's favorite movie is inception"}, and the second language is English. Select a vectorization model for English, such as the BCEmbedding model, and build a dictionary based on the words in the first text segment: {"Mia": 1, "favorite": 2, "movie": 3, "is": 4, "inception": 5}. The dictionary contains 5 words. According to the dictionary, the first text segment is expressed as a dense vector array [[1, 0, 0, 0, 0], [0, 2, 0, 0, 0], [0, 0, 3, 0, 0], [0, 0, 0, 4, 0], [0, 0, 0, 0, 5]]. Then, an association is established between the dense vector array and the first text segment. This association can be a simple metadata tag, or a more complex key-value pair or graph structure. The associated dense vector array is stored in the corresponding English knowledge base.

[0055] In this way, the semantic information of text fragments can be stored in the form of dense vector arrays, which can improve the efficiency of subsequent complex text analysis, such as text similarity calculation, topic modeling or sentiment analysis. Moreover, this method can avoid the complex and arduous alignment work required to map language words with the same meaning to the same vector space. At the same time, with the addition of new data and new languages, only the vectorization model of the specific language needs to be added, without the need to retrain or fine-tune the existing vectorization model.

[0056] In one embodiment, reference Figure 3 , the retrieval enhancement generation method supporting multilingual knowledge base also includes:

[0057] Step S301: Obtain the language types of multiple language knowledge bases.

[0058] Exemplarily, the language of the Chinese knowledge base is Chinese, the language of the English knowledge base is English, and the language of the French knowledge base is French.

[0059] Step S302: Translate the question into a plurality of target languages, wherein any target language corresponds to a language type of an acquired language knowledge base.

[0060] Among them, the translation can use a natural language translation model. Preferably, a large language model can also be used to translate questions. The large language model can recognize the specific semantics of the questions. When recognizing questions with rich meanings, it can retain the original semantics during the translation process for free translation. Especially for scenarios where questions are asked in multiple languages, the large language model can improve the accuracy of translation.

[0061] For example, an English question is “What's the weather like today?”. Assuming there are two language types, Chinese and French, the target languages ​​are Chinese and French. Therefore, the English question is translated into Chinese “What's the weather like today?” and French “Quel temps fait-il aujourd'hui?” respectively, so as to be subsequently retrieved from the language knowledge base corresponding to the target language.

[0062] Step S303: Based on the question translated into the target language, a search is performed in the corresponding language knowledge base to extract text segments related to the question, and an answer to the question is generated based on the text segments.

[0063] For example, according to the question in English, a search is performed in the English knowledge base to extract the English text fragments related to the question; according to the question in Chinese, a search is performed in the Chinese knowledge base to extract the Chinese text fragments related to the question; according to the question in French, a search is performed in the French knowledge base to extract the French text fragments related to the question. Finally, based on the text fragments of each language, assuming that no English text fragments and French text fragments are extracted, and the Chinese text fragment "Today's weather is sunny", then according to the Chinese text fragment, it is translated into the English answer to the question "It's sunny today".

[0064] In this embodiment, by translating the original question into the language type of the acquired language knowledge base, it is ensured that the question can be searched in multiple language knowledge bases to obtain valid answers, and information can be obtained from multiple angles, thereby improving the comprehensiveness and accuracy of the answer.

[0065] In one embodiment, the retrieval enhancement generation method supporting multilingual knowledge bases also includes: using a word segmenter for the target language to extract keywords from the question to obtain question keywords; based on the question keywords, using a probability model to search in the corresponding language knowledge base to extract text fragments related to the question.

[0066] The probability model may be, but is not limited to, a BM25 (Best Matching 25) algorithm, a BM25F (Best Matching 25 Field) algorithm, etc. The probability model is used to determine the correlation between the question keywords and the text segments in the corresponding language knowledge base.

[0067] For example, the Chinese question is "Is today suitable for outdoor activities?". The target language is English, and the question is translated into English "Is it suitable for outdoor activities today?". The question is segmented using an English word segmenter to obtain {"Is", "it", "suitable", "for", "outdoor", "activities", "today"}, and the keywords "suitable", "outdoor", and "activities" are extracted. Then, based on these question keywords, a probability model is used to retrieve text fragments containing these keywords in the English knowledge base, and text fragments related to the question are extracted, such as "The weather is suitable for outdoor activities today".

[0068] In this embodiment, by extracting keywords from the question through a language-specific word segmenter, more accurate question keywords can be obtained, and a probability model is used to retrieve text fragments related to the question keywords in the language-specific knowledge base, thereby improving the efficiency and robustness of the retrieval.

[0069] In one embodiment, the retrieval enhancement generation method supporting multilingual knowledge bases also includes: vectorizing questions using a vectorization model for the target language to generate a dense vector array about the question; based on the dense vector array about the question, using a dense vector model to search in the corresponding language knowledge base to extract text fragments related to the question.

[0070] Among them, the dense vector model can use but is not limited to deep learning models such as BERT (Bidirectional Encoder Representation from Transformers) and GPT (Generative Pre-trained Transformer).

[0071] For example, the Chinese question is "What movies does Mia like?". The target language is English, and the question is translated into English "What movies does Mia like?". The English vectorization model is used to vectorize the scores and generate a dense vector array about the question. Then, the dense vector model is used to search in the English knowledge base to extract text fragments related to the question, such as "Mia's favorite movie is inception".

[0072] In this embodiment, the question is vectorized by a vectorization model of a specific language, and then a dense vector model is used to capture the semantic relationship between the question and the text fragments in the specific language knowledge base, which can filter out irrelevant text fragments and improve the retrieval accuracy for complex language scenarios such as context understanding, synonyms, polysemous words, etc.

[0073] In order to improve the accuracy of the search, in one embodiment, reference Figure 4 and Figure 5 , the retrieval enhancement generation method supporting multilingual knowledge base also includes:

[0074] Step S501: Use a probability model to search in a language knowledge base corresponding to the target language, obtain the probability similarity of each text segment with respect to the question, and sort the text segments according to the probability similarity.

[0075] For example, the target language is English, the question in the target language is "What movies does Mia like?", and the language knowledge base of the target language includes three text segments {"Mia's favorite movie is inception", "She likes movies with complex plots", "Inception is a mind-bending thriller"}. The probability model is used to search in the English knowledge base, and the probability similarity of each text segment with respect to the question is calculated. Assuming that the probability similarities of the three text segments with the question are 90%, 80%, and 20% respectively, the text segments are sorted from large to small according to the probability similarity, and then "Mia's favorite movie is inception" is ranked 1, "She likes movies with complex plots" is ranked 2, and "Inception is a mind-bending thriller" is ranked 3.

[0076] Step S502: Use a dense vector model to search in the language knowledge base corresponding to the target language, obtain the dense vector similarity of each text segment with respect to the question, and sort the text segments according to the dense vector similarity.

[0077] For example, the target language is English, the question in the target language is "What movies does Mia like?", and the language knowledge base of the target language includes three text segments {"Mia's favorite movie is inception", "She likes movies with complex plots", "Inception is a mind-bending thriller"}. The dense vector model is used to search in the English knowledge base, and the dense vector similarity of each text segment with respect to the question is calculated. Assuming that the dense vector similarities of the three text segments with the question are 95%, 85%, and 20% respectively, the text segments are sorted from large to small according to the dense vector similarity, then "Mia's favorite movie is inception" is ranked 1, "She likes movies with complex plots" is ranked 2, and "Inception is a mind-bending thriller" is ranked 3.

[0078] Step S503: Use the reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results and the probability similarity sorting results, and calculate the comprehensive score of each text segment.

[0079] Among them, the reciprocal sorting fusion algorithm is used to combine multiple structure sets with different relevance indicators into a single result set. The reciprocal sorting fusion algorithm can calculate the weight of the sorting results, normalize the weight, merge the weight, and sort according to the merged weight to obtain the comprehensive score of each text segment.

[0080] Exemplarily, the sorting results in steps S01 and S02 are merged and sorted using a reciprocal sorting fusion algorithm, and the comprehensive scores of the three text segments are 5, 3, and 2, respectively.

[0081] Step S504: Filter a preset number of text segments with high comprehensive scores, and generate answers to the questions based on the filtered text segments.

[0082] The preset number may be obtained based on experience or model training, and the preset number is intended to screen out text segments that are highly similar to the question.

[0083] Exemplarily, two text segments with the highest comprehensive scores are selected to generate answers to the questions as “Mia’s favorite movie is inception” and “She likes movies with complex plots”.

[0084] It should be noted that if there are N target languages, the above steps are performed N times to ensure the accuracy of question-answer retrieval.

[0085] In this embodiment, the similarity ranking between the question and the text fragments in the target language knowledge base is obtained through the probability model and the dense vector model, and then the dense vector similarity ranking results and the probability similarity ranking results are merged and sorted through the reciprocal sorting fusion algorithm to obtain a comprehensive score, and the text fragments with high comprehensive scores are screened to generate answers to the questions, which greatly improves the efficiency, accuracy and robustness of question and answer retrieval, and further improves the accuracy and efficiency of multilingual question and answer retrieval.

[0086] In one embodiment, reference Figure 6 , the retrieval enhancement generation method supporting multilingual knowledge base also includes:

[0087] Step S601: Use the Rerank model to search in the language knowledge base corresponding to the target language, obtain the Rerank similarity of each text segment with respect to the question, and sort the text segments according to the Rerank similarity.

[0088] Step S602: Use the reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results, the probability similarity sorting results and the Rerank similarity sorting results, and calculate the comprehensive score of each text segment.

[0089] Step S603: Filter a preset number of text segments with high comprehensive scores, and generate answers to the questions based on the filtered text segments.

[0090] Among them, the Rerank model can understand the semantic relationship between questions and text fragments, and can perform fine-grained analysis on the preliminary retrieval results, considering more features and contextual information, thereby providing more accurate ranking.

[0091] Exemplarily, the target language is English, the question in the target language is "What movies does Mia like?", and the language knowledge base of the target language includes three text fragments {"Mia's favorite movie is inception", "She likes movies with complex plots", "Inception is a mind-bending thriller"}. The Rerank model is used to search in the English knowledge base, and the Rerank similarity of each text fragment to the question is calculated. Assuming that the Rerank similarities of the three text fragments to the question are 100%, 90%, and 10% respectively, the text fragments are sorted from large to small according to the Rerank similarity. Then, the dense vector similarity sorting results, the probability similarity sorting results, and the Rerank similarity sorting results are merged and sorted using the reciprocal sorting fusion algorithm. Assuming that the comprehensive scores of each text fragment are 5, 4, and 2. A text fragment with a high comprehensive score is selected to generate an answer to the question as "Mia's favorite movie is inception".

[0092] In this embodiment, the similarity ranking between the question and the text fragments in the target language knowledge base is obtained through the Rerank model, probability model and dense vector model, and then the three ranking results are merged and sorted through the inverse sorting fusion algorithm to obtain a comprehensive score, and the text fragments with high comprehensive scores are filtered to generate answers to the questions, which greatly improves the efficiency, accuracy and robustness of question and answer retrieval, and further improves the accuracy and efficiency of multilingual question and answer retrieval.

[0093] Based on the same inventive concept, the present application also provides a search enhancement generation system supporting a multilingual knowledge base, which adopts a search enhancement generation method supporting a multilingual knowledge base as in any of the above embodiments. The implementation scheme for solving the problem provided by the system is similar to the implementation method recorded in the above method, so the specific definition of the search enhancement generation system supporting a multilingual knowledge base can be found in the definition of the search enhancement generation method supporting a multilingual knowledge base above, which will not be repeated here.

[0094] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the claims attached to this application.

Claims

1. A retrieval enhancement generation method supporting a multilingual knowledge base, characterized in that: The method comprises the following steps: Get the documents to be processed; Identifying a language category of the document to be processed, wherein the document to be processed includes text in a first language, and the text in the first language accounts for the largest proportion of text in the document to be processed; Splitting the document to be processed using a document segmentation model for the first language to obtain a plurality of text segments; Identifying a language category in a first text segment, where the first text segment contains text in a second language, and the proportion of the text in the second language in the first text segment is greater than a preset threshold, and the second language is the same as or different from the first language; Storing the first text segment in a language knowledge base corresponding to the second language; A word segmenter for the second language is used to extract keywords from the first text segment, and the extracted keywords are associated with the first text segment and stored in the language knowledge base.

2. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 1, characterized in that: In response to the first text segment including texts in at least two languages, and the proportions of the texts in the two languages ​​in the first text segment are both greater than the preset threshold, The first text segment is stored in at least two language knowledge bases, wherein any of the language knowledge bases corresponds to a language whose proportion is greater than the preset threshold.

3. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 1 or 2, characterized in that: The method further comprises: The first text segment is vectorized using a vectorization model for the second language to generate a dense vector array, and the dense vector array is associated with the first text segment and stored in the language knowledge base.

4. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 1, characterized in that: The method further comprises: Obtain the language types of multiple language knowledge bases; Translating the question into a plurality of target languages, any of the target languages ​​corresponding to a language type of the acquired language knowledge base; According to the question translated into the target language, a search is performed in the corresponding language knowledge base to extract text segments related to the question, and an answer to the question is generated based on the text segments.

5. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 4, characterized in that: The method further comprises: Extracting keywords from the question using a word segmenter for the target language to obtain question keywords; According to the question keywords, a probability model is used to search in the corresponding language knowledge base to extract text fragments related to the question.

6. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 4 or 5, characterized in that: The method further comprises: Vectorizing the question using a vectorization model for the target language to generate a dense vector array for the question; According to the dense vector array about the question, a dense vector model is used to search in the corresponding language knowledge base to extract text fragments related to the question.

7. The retrieval enhancement generation method supporting a multilingual knowledge base according to claim 6, characterized in that: The method further comprises: Using a probability model to search in a language knowledge base corresponding to the target language, obtaining the probability similarity of each text segment with respect to the question, and sorting the text segments according to the probability similarity; Using a dense vector model to search in a language knowledge base corresponding to the target language, obtaining a dense vector similarity of each text segment with respect to the question, and sorting the text segments according to the dense vector similarity; Use the reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results and the probability similarity sorting results, and calculate the comprehensive score of each text segment; A preset number of text segments with high comprehensive scores are screened, and an answer to the question is generated based on the screened text segments.

8. The search enhancement generation method supporting multilingual knowledge base according to claim 7, characterized in that: The method further comprises: Using the Rerank model, searching in the language knowledge base corresponding to the target language, obtaining the Rerank similarity of each text segment with respect to the question, and sorting the text segments according to the Rerank similarity; Use the reciprocal sorting fusion algorithm to merge and sort the dense vector similarity sorting results, probability similarity sorting results, and Rerank similarity sorting results, and calculate the comprehensive score of each text segment; A preset number of text segments with high comprehensive scores are screened, and an answer to the question is generated based on the screened text segments.

9. A retrieval enhancement generation system supporting multilingual knowledge base, characterized in that: The system adopts the retrieval enhancement generation method supporting a multilingual knowledge base as described in any one of claims 1 to 8.

Images