Multi-language machine translation method and device based on retrieval enhancement, electronic equipment and storage medium

Abstract

The invention discloses a multi-language machine translation method and device based on retrieval enhancement, electronic equipment and a storage medium. According to the method, a multi-language data set is collected, and systematic data preprocessing is carried out on the data set; performing language recognition and field division on the preprocessed data, and performing word segmentation and byte pair coding; a multi-language retrieval enhancement database including English, German, French, Italian, Chinese and Japanese is constructed; identifying a to-be-translated sentence through the domain theme identification model, taking the to-be-translated sentence as a query, and retrieving similar sentence pairs through a retrieval algorithm based on text embedding and a retrieval algorithm based on word matching; and guiding the large model to generate translation of the sentence to be translated through the prompt template, and selecting an optimal translation through large model translation quality evaluation. Through a retrieval enhancement mode, the adaptive capacity of the model to low-resource and cross-domain scenes is effectively improved, and the translation quality of multi-language machine translation is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing in the field of information technology, and particularly to technical fields such as deep learning, multilingual, and cross-lingual machine translation. Specifically, a retrieval-enhanced multilingual machine translation method, device, electronic device, and storage medium are provided. Background Art

[0002] Since the traditional neural machine translation was proposed in 2013, it has gone through multiple stages of development. From the initial recurrent neural network architecture based on encoder-decoder, to the improved version with the introduction of the attention mechanism, and then to the wide application of the Transformer model, it has made large-scale parallel data efficient parallel training possible. In addition, in recent years, large-scale pre-trained language models represented by GPT, BERT, and T5 have further expanded the capabilities of the translation system. By performing unsupervised pre-training on massive non-parallel corpora, these models not only learn rich language semantic and structural knowledge, but also can quickly adapt to specific translation tasks through fine-tuning.

[0003] With the continuous progress of technology, large-scale language models have gradually become a research hotspot in the field of machine translation due to their excellent fluency and relatively high translation accuracy. Representative models include GPT-4, LLaMA-2, and Gemma-2, etc. However, despite the remarkable progress, large models still face the following two major challenges in machine translation tasks: First, the translation performance on low-resource languages is still not ideal. Since most large models are mainly trained based on English-centered multilingual corpora, when facing language pairs with scarce resources, the model performance often drops significantly. Second, the ability of cross-domain translation still needs to be improved. The corpora relied on for current large model training are mostly general domain texts, lacking sufficient modeling of the language characteristics of professional fields. Therefore, when dealing with translation tasks in specific fields such as medicine, law, and IT, how to improve the domain adaptation ability and translation effect of the model is still an important problem that urgently needs to be solved. In response to the above challenges, the retrieval enhancement technology provides us with a new idea. By retrieving external knowledge to supplement the deficiencies of the pre-trained model in specific language pairs or domain knowledge, the retrieval enhancement method is expected to significantly improve the quality of low-resource language and cross-domain translation, providing a feasible path for achieving more accurate multilingual machine translation. Summary of the Invention

[0004] The object of the present invention is to provide a multi - language machine translation method, device, electronic device and storage medium based on retrieval enhancement. This method collects a multi - language dataset and conducts systematic data pre - processing on the dataset; performs language identification and domain division on the pre - processed data, and conducts word segmentation and byte - pair encoding; encodes the processed dataset through a pre - trained language model; obtains the encoded dense vectors, and constructs a multi - language retrieval - enhanced database including languages such as English, German, French, Italian, Chinese, Japanese, etc.; identifies the domain theme of the sentence to be translated through a domain - theme recognition model, and extracts the domain theme to which the sentence to be translated belongs; uses the sentence to be translated as a query through a retrieval algorithm based on text embedding and a retrieval algorithm based on word matching to retrieve similar sentence pairs; combines the two retrieval results with domain information and constructs two prompt templates, and guides the large model to generate the translation of the sentence to be translated through the prompt templates, and finally selects the best translation through translation quality evaluation. The present invention effectively improves the adaptability of the model to low - resource and cross - domain scenarios and improves the translation quality of multi - language machine translation through the method of retrieval enhancement.

[0005] The multi - language machine translation method based on retrieval enhancement according to the present invention is carried out according to the following steps:

[0006] a. Collect a multi - language dataset, conduct systematic pre - processing on the parallel corpus contained therein, and divide the processed corpus into a training set, a validation set and a test set. The systematic pre - processing includes standardizing the character format, parsing and replacing escape characters, cleaning special characters, filtering parallel sentence pairs with too large or too small lengths, and deleting duplicate sentence pairs;

[0007] b. Perform multi - language language identification and domain - theme division on the pre - processed training set, validation set and test set data obtained in step a, and conduct word - segmentation and byte - pair encoding processing to obtain the processed training set, validation set and test set;

[0008] c. Construct a retrieval - enhanced database, encode the training set obtained in step b through a pre - trained language model, and construct a retrieval - enhanced database by extracting the dense vectors generated by the model;

[0009] d. Construct a domain - theme recognition model, use the pre - trained language model to identify the domain information of the sentence to be translated, and obtain the corresponding domain information; for the identification of the domain information of the sentence to be translated, use the BERTopic pre - trained model, and the identified domain - theme information is provided to the large model through a prompt template;

[0010] e. Similar text retrieval: Use the sentence to be translated as a query condition through database query instructions. Respectively, use the retrieval method based on text embedding and the retrieval method based on word matching to retrieve similar parallel sentence pairs from the retrieval-enhanced database, and combine them with the domain information obtained in step d to form two prompt templates;

[0011] f. The large model performs translation inference with the help of the prompt templates, generates different translation results based on the two prompt templates, and uses the large model to evaluate the translation quality to select the best translation; The large model evaluates the translation quality of the two translations and selects the best translation as the final translation result. The evaluation method uses a prompt template for evaluating the translation quality, allowing the large model to quantitatively score the two translation results to select the best translation.

[0012] In step b, the multi-language dataset processed in step a is re-identified for language. The FastText and langdetect recognition methods are used, and the data is divided according to different domains. For Chinese texts, the open-source tool jieba is used for word segmentation, and for other languages, the moses tool is used for word segmentation. Finally, byte pair encoding processing is performed on the segmented text.

[0013] In step c, pre-trained language models such as SONAR and RoBERTa are used to encode the sentences to obtain corresponding dense vectors. The vectors and texts are combined into entries to build a retrieval-enhanced database. After building retrieval-enhanced databases for datasets in different languages, the entries in the databases are merged to form a multi-language retrieval-enhanced database.

[0014] The similarity calculation described in step e adopts two methods: based on text embedding calculation and based on keyword matching. For dense vectors, the IndexFlatL2 and IndexIVFFlat vector retrieval methods are used to calculate the L2 distance between the query and the entries in the database to obtain the k samples with the highest similarity. At the same time, to ensure keyword matching, a text similarity retrieval strategy based on word matching is also adopted, including but not limited to the BM25 similar text retrieval method, and it forms two prompt templates with the results obtained from the previous text embedding retrieval and provides them to the large model for translation inference.

[0015] An apparatus involved in a multi-language machine translation method based on retrieval enhancement. The apparatus is composed of a data preprocessing module, a retrieval-enhanced database module, a domain topic recognition module, a text similarity retrieval module, and a translation inference module based on prompt templates, where:

[0016] Data preprocessing module: Collect multi-language datasets and systematically preprocess the parallel sentence pairs therein, including a data cleaning unit, a data recognition unit, and a data segmentation unit;

[0017] Retrieval-Augmented Database Module: Encode parallel sentence pairs in a multilingual dataset to build a vector database;

[0018] Domain Topic Recognition Module: Use a pre-trained language model to identify domain information in the sentence to be translated and obtain the corresponding domain information;

[0019] Text Similarity Retrieval Module: Use the sentence to be translated as a query and perform retrieval in the retrieval-augmented database using two text similarity retrieval algorithms to retrieve the parallel sentence pair with the highest similarity, forming two prompt templates;

[0020] Translation Inference Module Based on Prompt Templates: Provide the retrieval results to a large model with the help of prompt templates. The large model performs multilingual machine translation under the prompts of similar texts and domain information, and conducts quality assessment on the generated translations, and selects the best translation as the final translation result.

[0021] The data preprocessing module includes:

[0022] Dataset Cleaning Unit: Standardize character formats, parse and replace escape characters, clean special characters, filter parallel sentence pairs with overly large or small lengths, and delete duplicate sentence pairs;

[0023] Data Recognition Unit: Re-identify the dataset in multiple languages and divide the data according to different domains;

[0024] Data Splitting Unit: Divide the dataset described by the data recognition unit into a training set, a validation set, and a test set, and perform word segmentation and byte pair encoding processing.

[0025] An electronic device, characterized in that the device includes at least one processor; at least one GPU computing card; and a memory communicatively connected to the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that at least one processor or at least one GPU computing card can execute the method described in claims 1-4.

[0026] A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method described in claims 1-4.

[0027] The instance scenario it faces is to implement the translation of large models for multilingual and cross-domain texts, that is, the translation effect of large models for low-resource languages or specific domains is not good, but the adaptability of the model can be enhanced through retrieval enhancement and domain topic recognition. For example, parallel corpora in medical, legal, IT and other fields between English and German are used to build a retrieval enhancement database with in-domain data. By identifying the domain to which the sentence to be translated belongs, the corresponding retrieval enhancement database in the domain is selected, and similar parallel sentence pairs are retrieved. The retrieval results are provided to the large model with the help of a prompt template to improve the translation performance of the model for specific domains and low-resource languages.

[0028] The present invention provides a method, device, electronic device and storage medium for multilingual machine translation of large models based on retrieval enhancement. Compared with the prior art, the embodiments of the present invention effectively solve the problem of poor translation effect for low-resource languages and cross-domain texts in traditional large model machine translation by integrating the retrieval enhancement of similar texts and the domain topic recognition strategy, improve the adaptability of the model, and can be widely applied in fields such as machine translation and text generation: BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of a multilingual machine translation method based on retrieval enhancement provided by the present invention;

[0030] Figure 2 It is a schematic diagram of the data cleaning unit provided by the present invention;

[0031] Figure 3 It is a schematic diagram of the data identification and segmentation unit provided by the present invention;

[0032] Figure 4 It is a schematic diagram of text similarity retrieval provided by the present invention;

[0033] Figure 5 It is a schematic diagram of the structure of a multilingual machine translation device based on retrieval enhancement provided by the present invention;

[0034] Figure 6 It is a block diagram of an electronic device for a multilingual machine translation method based on retrieval enhancement provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered merely exemplary. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application.

[0036] The method described in the present invention is applicable to the situation of mutual translation between multiple languages in a multilingual scenario; this method is executed by a large model multilingual machine translation device based on retrieval enhancement, and this device is implemented in software and / or hardware.

[0037] Embodiment

[0038] The following combines Figure 1 to describe the implementation method of the retrieval-enhanced multilingual machine translation model according to the embodiments of the present invention:

[0039] A retrieval-enhanced multilingual machine translation method described in the present invention is carried out according to the following steps:

[0040] a. Collect a multilingual dataset, perform systematic preprocessing on the parallel corpus contained therein, and divide the processed corpus into a training set, a validation set, and a test set. The systematic preprocessing includes standardizing the character format, parsing and replacing escape characters, cleaning special characters, filtering parallel sentence pairs with too large or too small lengths, and deleting duplicate sentence pairs;

[0041] b. Perform multilingual language identification and domain theme division on the preprocessed training set, validation set, and test set data obtained in step a, and perform word segmentation and byte pair encoding processing to obtain the processed training set, validation set, and test set;

[0042] c. Construct a retrieval-enhanced database, encode the training set obtained in step b through a pre-trained language model, extract the dense vectors generated by the model, and construct a retrieval-enhanced database;

[0043] d. Construct a domain theme recognition model, use a pre-trained language model to perform domain information recognition on the sentence to be translated, and obtain the corresponding domain information; the domain information recognition of the sentence to be translated uses a BERTopic pre-trained model, and the recognized domain theme information is provided to the large model through a prompt template;

[0044] e. Similar text retrieval, use the sentence to be translated as a query condition through a database query instruction, and respectively use a retrieval method based on text embedding and a retrieval method based on word matching to retrieve similar parallel sentence pairs from the retrieval-enhanced database, and combine them with the domain information obtained in step d to form two prompt templates;

[0045] f. The large model performs translation inference with the help of prompt templates, generates different translation results based on two prompt templates, and selects the best translation by evaluating the translation quality with the large model; the large model evaluates the translation quality of the two translations, selects the best translation among them as the final translation result, and the evaluation method uses a prompt template for evaluating translation quality to let the large model quantitatively score the two translation results to select the best translation.

[0046] In step b, language recognition is performed again on the multilingual dataset processed in step a. FastText and langdetect recognition methods are used, and the data is divided according to different fields. For Chinese texts, the open-source tool jieba is used for word segmentation, and for other languages, the moses tool is used for word segmentation. Finally, byte pair encoding processing is performed on the segmented text.

[0047] In step c, pre-trained language models such as SONAR and RoBERTa are used to encode sentences to obtain corresponding dense vectors. The vectors and texts are formed into entries to build a retrieval-enhanced database. After building retrieval-enhanced databases for datasets in different languages, the entries in the databases are merged to form a multilingual retrieval-enhanced database.

[0048] The similarity calculation described in step e adopts two methods: based on text embedding calculation and based on keyword matching. For dense vectors, IndexFlatL2 and IndexIVFFlat vector retrieval methods are used to calculate the L2 distance between the query and the entries in the database to obtain the k samples with the highest similarity. At the same time, to ensure keyword matching, a text similarity retrieval strategy based on word matching is also adopted, including but not limited to the BM25 similar text retrieval method, and it forms two prompt templates with the results obtained from the previous text embedding retrieval and provides them to the large model for translation inference.

[0049] A device involved in a multilingual machine translation method based on retrieval enhancement. This device is composed of a data preprocessing module, a retrieval-enhanced database module, a domain topic recognition module, a text similarity retrieval module, and a translation inference module based on prompt templates, where:

[0050] Data preprocessing module: Collect multilingual datasets and systematically preprocess the parallel sentence pairs therein, including a data cleaning unit, a data recognition unit, and a data segmentation unit;

[0051] Retrieval-enhanced database module: Encode the parallel sentence pairs in the multilingual dataset to build a vector database;

[0052] Domain topic recognition module: Use a pre-trained language model to identify the domain information of the sentence to be translated to obtain the corresponding domain information;

[0053] Text similarity retrieval module: Using the sentence to be translated as a query, two text similarity retrieval algorithms are used to retrieve in the retrieval-enhanced database, and the parallel sentence pairs with the highest similarity are retrieved to form two prompt templates;

[0054] Translation inference module based on prompt templates: With the help of the prompt templates, the retrieval results are provided to the large model. The large model realizes multilingual machine translation under the prompts of similar texts and domain information, and conducts quality evaluation on the generated translations, and selects the best translation as the final translation result.

[0055] The data preprocessing module includes:

[0056] Dataset cleaning unit: Standardize the character format, parse and replace escape characters, clean special characters, filter parallel sentence pairs with too large or too small lengths, and delete duplicate sentence pairs;

[0057] Data recognition unit: Re-recognize the dataset in multiple languages and divide the data according to different domains;

[0058] Data splitting unit: Divide the dataset described by the data recognition unit into a training set, a validation set, and a test set, and perform word segmentation and byte pair encoding processing.

[0059] An electronic device, characterized in that the device includes at least one processor; at least one GPU computing card; and a memory communicatively connected to the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that at least one processor can or at least one GPU computing card can execute the method described in claims 1-4.

[0060] A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method described in claims 1-4.

[0061] Figure 1 .S110. Collect a multilingual dataset and perform systematic preprocessing on the parallel corpus contained therein;

[0062] This operation aims to filter and clean the collected dataset and divide the dataset. For specific operations, see Figure 2 :

[0063] S210. Symbol format specification;

[0064] S220. Escape character replacement;

[0065] S230. Clean special characters;

[0066] S240. Filter parallel sentence pairs with excessive or insufficient lengths;

[0067] S250. Delete duplicate sentence pairs;

[0068] S260. Divide the dataset into a training set, a validation set, and a test set;

[0069] Figure 1 .S130. Perform multi - language language identification, domain - topic division on the dataset, and perform word segmentation and byte - pair encoding processing;

[0070] In the embodiments of the present invention, this operation aims to further improve the quality of the dataset. A new language identification is performed, and the identification algorithms adopt methods such as FastText and langdetect, and the data is divided according to different domains; for Chinese texts, the open - source tool jieba is used for word segmentation, and for other languages, the moses tool is used for word segmentation. Finally, byte - pair encoding processing is performed on the segmented text. For specific operations, see Figure 3 :

[0071] S310. Multi - language language identification;

[0072] S320. Domain - topic division;

[0073] S330. Word segmentation. For Chinese texts, the open - source tool jieba is used for word segmentation, and for other languages, the moses tool is used for word segmentation;

[0074] S340. Byte - pair encoding. Perform byte - pair encoding processing on the segmented text;

[0075] Figure 1 .S130. Build a retrieval - enhanced database. Encode the processed training set through a pre - trained language model, store the obtained dense vectors by extracting the dense vectors generated by the model, build a retrieval - enhanced database, and merge the retrieval - enhanced databases constructed in different languages, including but not limited to languages such as English, German, French, Italian, Chinese, Japanese, etc., to build a multi - language retrieval - enhanced database;

[0076] In the embodiments of the present invention, a pre - trained model, such as SONAR, RoBERTa, or Sentence - BERT, is used to convert the text into dense vectors. Define the pre - processed dataset as T = {t1, t2,..., t N}, where t i is a piece of data in the dataset. Assume the embedding function of the model is f θ , then the dense vector after encoding each piece of text can be expressed as:

[0077] h i = fθ (t i ),h i ∈R d

[0078] The obtained retrieval enhanced database is represented as:

[0079]

[0080] Exemplarily,

[0081] In the text translation task in the medical field, there are parallel corpora for translations from English to French, from English to Chinese, from English to German, and from Chinese to Japanese. Multiple retrieval enhanced databases are obtained by encoding parallel sentence pairs through a pre-trained language model. Merging the entries of the databases gives a multilingual retrieval enhanced database;

[0082] Figure 1 .S140. Construct a domain topic recognition model and train the model using the dataset obtained in step c. Use the domain topic recognition model to classify the sentence to be translated into the corresponding domain to obtain the domain information of the sentence to be translated;

[0083] In the embodiments of the present invention, for the sentence to be translated, accurately identifying the domain topic to which it belongs is the key to improving the cross-domain translation performance of the large model; the present invention uses a pre-trained language model for domain classification, such as classification models like BERTopic, etc.; after identifying the corresponding domain, select the retrieval enhanced database for the corresponding domain and add domain information to the prompt template to improve the domain adaptation ability of the large model;

[0084] Figure 1 .S150. Text similarity retrieval. Use the sentence to be translated as a query condition through a database query instruction, and retrieve similar parallel sentence pairs from the retrieval enhanced database according to text similarity;

[0085] In the embodiments of the present invention, this step aims to retrieve similar texts as examples to enhance the translation performance of the large model in low-resource and cross-domain scenarios, such as Figure 4 as shown, and its specific operation is:

[0086] S410. Adopt a retrieval algorithm based on text embedding, use dense vectors as retrieval indexes, and obtain retrieval results by calculating the L2 distance between the query vector and the vectors in the retrieval enhanced database;

[0087] Specifically, define the vector of the sentence q to be queried after being encoded by the pre-trained language model as v q , and by calculating the distance between v q and the vectors in the retrieval enhanced database , such as the Euclidean distance, etc. Assuming the vector dimension is n, the distance can be expressed as:

[0088]

[0089] As an evaluation index of text similarity:

[0090]

[0091] Finally, through sorting, the entry most similar to the sentence q to be queried is obtained, that is, the entry with the smallest Euclidean distance:

[0092]

[0093] S420. Adopt a retrieval algorithm based on word matching, which focuses on the matching degree of keywords, retrieves texts with high similarity, and obtains retrieval results;

[0094] When dealing with translation tasks in specific fields, the accuracy of keyword translation is often considered the key to evaluating the quality of translation. In order to improve the matching degree of words in the retrieval results and better guide the model to perform translation tasks, the embodiment of the present invention adds a retrieval algorithm based on word matching on the basis of the retrieval algorithm based on text embedding, including but not limited to algorithms such as BM25 and BM25F, which alleviates the problem of insufficient keyword matching degree of the retrieval algorithm based on text embedding.

[0095] Exemplarily:

[0096] When dealing with translation texts in the medical field, whether the medical term "daptomycin" is accurately translated has a great impact on the quality of the translation. Therefore, retrieving parallel sentence pairs containing "daptomycin" and providing them as examples to the large model can improve the domain adaptation ability of the large model.

[0097] Specifically, the BM25 algorithm is a method that provides term frequency and document length to measure the similarity between a query and database entries, and its calculation method is:

[0098]

[0099] Among them, t is an entry in the retrieval library, q is the sentence to be queried, and q i is the i-th word of the sentence to be queried. f(q i , i) is the frequency of the word appearing in the data entry t, |t| is the length (number of words) of the data entry t, and avg_len is the average value of the lengths of data entries in the retrieval enhanced database. k1 and b belong to hyperparameters, and IDF(q i ) is the inverse document frequency, and its calculation method is:

[0100]

[0101] where T is the total number of entries in the database, df(q i ) represents the number of entries containing the word q i .

[0102] This text similarity algorithm considers factors such as word frequency and inverse document frequency. Compared with the similarity calculation algorithm based on text embedding, the method based on word matching has greater advantages in short text and keyword retrieval. Therefore, in the embodiment, the present invention adopts two different retrieval algorithms to improve the retrieval accuracy.

[0103] S430. Integrate domain information;

[0104] In the embodiment of the present invention, the identified domain information is provided to the large model by writing it into the prompt template to guide the large model to perform domain adaptation.

[0105] S450. Generate two prompt templates based on the retrieval results obtained by the two retrieval algorithms;

[0106] Figure 1 .S160. The large model performs translation inference with the help of the prompt template to generate a translation in the target language. The prompt template contains parallel sentence pairs extracted from the constructed retrieval-enhanced database and the identified domain information, and a multilingual large model is used to achieve multilingual machine translation;

[0107] Exemplarily:

[0108] In the English-to-German translation task, the sentence to be translated is "NeoRecormon Multidose is provided as a lyophilisate vial and solvent ampoule for solution for injection." By the pre-trained language model, the domain to which the sentence to be translated belongs is identified as "medical". At the same time, the sentence to be translated is used as a query, and based on the text similarity algorithm, similar texts are retrieved from the retrieval-enhanced database as examples:

[0109] "The solvent ampoule contains water for injections." Corresponding translation: "Die Ampulle mit Wasser für Injektionszwecke."

[0110] "NeoRecormon Multidose is supplied as a lyophilisate for solution for injection in vials." Corresponding translation: "NeoRecormon Multidose liegt in der Durchstechflasche als Lyophilisat zur Herstellung einer vor."

[0111] "2ml solvent ampoule contains sodium chloride, hydrochloric acid, sodium hydroxide and water for injections." Corresponding translation: "1 Ampulle mit 2ml Natriumchlorid, Natriumhydroxid und Wasser für Injektionszwecke."

[0112] "Preparation of NeoRecormon Multidose solution (medical care staff) (1) Take the lyophilisate vial out of the package." Corresponding translation: "Herstellung der NeoRecormon Multidose (medizinisches Pflegepersonal) (1) Nehmen Sie die Durchstechflasche mit dem Lyophilisat aus der Packung."

[0113] After performing domain identification and text retrieval, the present invention provides the obtained domain information and retrieval results to the large model through a prompt template, generates different translation results based on two prompt templates, and selects the best translation with the help of the large model to obtain more accurate and high-quality translation results.

[0114] The instance scenario it faces is to implement the translation of large models for multilingual and cross - domain texts, that is, the translation effect of large models for low - resource languages or specific domains is not good, but the adaptability of the model can be enhanced through retrieval enhancement and domain - topic recognition. For example, using parallel corpora in fields such as medicine, law, and IT between English and German, a retrieval - enhanced database is constructed with in - domain data. By identifying the domain to which the sentence to be translated belongs, the corresponding retrieval - enhanced database is selected, and similar parallel sentence pairs are retrieved. With the help of a prompt template, the retrieval results are provided to the large model to improve the translation performance of the model for specific domains and low - resource languages;

[0115] As Figure 5 shown, the present invention provides a multilingual machine translation device based on retrieval enhancement, which is characterized in that the device is composed of a data pre - processing module, a retrieval - enhanced database module, a domain - topic recognition module, a text similarity retrieval module, and a translation inference module based on a prompt template, where:

[0116] Data pre - processing module: Collect multilingual datasets and systematically pre - process the parallel sentence pairs therein, including a data cleaning unit, a data recognition unit, and a data segmentation unit.

[0117] Retrieval - enhanced database module: Encode the parallel sentence pairs in the multilingual dataset to construct a vector database.

[0118] Domain - topic recognition module: Use a pre - trained language model to identify the domain information of the sentence to be translated to obtain the corresponding domain information.

[0119] Text similarity retrieval module: Use the sentence to be translated as a query and retrieve in the retrieval - enhanced database using two text similarity retrieval algorithms to retrieve the parallel sentence pair with the highest similarity, forming two prompt templates.

[0120] Translation inference module based on a prompt template: Provide the retrieval results to the large model with the help of the prompt template. The large model realizes multilingual machine translation under the prompt of similar texts and domain information, and conducts quality assessment on the generated translation, and selects the best translation as the final translation result;

[0121] The present invention provides an electronic device and a readable storage medium;

[0122] As Figure 6 shown, it is a block diagram of the electronic device of the present invention. The electronic device refers to various modern electronic digital computers, including, for example: personal computers, portable computers, and various server devices. The components, their interconnection relationships, and functions shown herein are only examples;

[0123] As Figure 6As shown, the electronic device includes: one or more multi-core processors, one or more GPU computing cards, and a memory. To enable the electronic device to interact, it should further include: an input device and an output device. Various devices are interconnected and communicate through a bus;

[0124] The memory is the non-transitory computer-readable storage medium provided by the present application. Among them, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor or the at least one GPU computing card, so that the at least one processor or the at least one GPU computing card can execute the method described in any one of the embodiments of the present application;

[0125] The input device provides and accepts control signals input by the user into the electronic device, including a keyboard that generates digital or character information and a mouse that controls the device to generate other key signals. The output device provides feedback information to the user of the electronic device, including a display that prints the execution result or process.

[0126] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include known common knowledge or conventional technical means in the technical field not disclosed by the present invention. It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A retrieval-enhanced multilingual machine translation method, characterized in that Proceed as follows: a. Collect multilingual datasets, systematically preprocess the parallel corpora contained therein, and divide the processed corpora into training sets, validation sets, and test sets. The systematic preprocessing includes standardizing character formats, parsing and replacing escape characters, cleaning special characters, filtering parallel sentence pairs with overly large or small lengths, and deleting duplicate sentence pairs; b. Perform multilingual language identification and domain theme division on the preprocessed training set, validation set, and test set data obtained in step a, and perform word segmentation and byte pair encoding processing to obtain the processed training set, validation set, and test set; c. Construct a retrieval-enhanced database. Encode the training set obtained in step b through a pre-trained language model, and construct a retrieval-enhanced database by extracting the dense vectors generated by the model; d. Construct a domain theme identification model, use a pre-trained language model to identify domain information for the sentence to be translated, and obtain the corresponding domain information; for identifying domain information for the sentence to be translated, use the BERTopic pre-trained model, and provide the identified domain theme information to the large model through a prompt template; e. Similar text retrieval. Use the sentence to be translated as a query condition through a database query instruction, and respectively use a text embedding-based retrieval method and a word matching-based retrieval method to retrieve similar parallel sentence pairs from the retrieval-enhanced database, and combine them with the domain information obtained in step d to form two prompt templates; f. The large model performs translation inference with the help of the prompt template, generates different translation results based on the two prompt templates, and selects the best translation through the translation quality evaluation of the large model; the large model evaluates the translation quality of the two translations, selects the best translation among them as the final translation result, and the evaluation method uses a prompt template for evaluating translation quality to allow the large model to quantitatively score the two translation results to select the best translation.

2. The multi - language machine translation method based on retrieval enhancement according to claim 1, characterized in that, In step b, a new language identification is performed on the multilingual dataset processed in step a. The FastText and langdetect identification methods are used, and the data is divided according to different domains. For Chinese texts, the open-source tool jieba is used for word segmentation, and for other languages, the moses tool is used for word segmentation. Finally, byte pair encoding processing is performed on the segmented text.

3. A multilingual machine translation method based on retrieval enhancement according to claim 1, wherein In step c, pre-trained language models such as SONAR and RoBERTa are used to encode the sentences to obtain the corresponding dense vectors, and the vectors and texts are combined into entries to construct a retrieval-enhanced database. After constructing retrieval-enhanced databases for datasets in different languages, the entries in the databases are merged to form a multilingual retrieval-enhanced database.

4. A multilingual machine translation method based on retrieval enhancement according to claim 1, characterized in that, In step e, the similarity calculation is performed in two ways: based on text embedding calculation and keyword matching. For dense vectors, the IndexFlatL2 and IndexIVFFlat vector retrieval methods are used to calculate the L2 distance between the query and the entries in the database, and the k samples with the highest similarity are obtained. At the same time, to ensure the keyword matching degree, a text similarity retrieval strategy based on word matching is also adopted, including the BM25 similar text retrieval method, and the results obtained from the previous text embedding retrieval and the results of the word matching-based text similarity retrieval constitute two prompt templates, which are provided to the large model for translation inference.

5. An apparatus involved in the retrieval-enhanced multi-lingual machine translation method as described in claim 1, characterized in that, The device consists of a data preprocessing module, a retrieval-enhanced database module, a domain topic recognition module, a text similarity retrieval module, and a translation inference module based on a prompt template, where: Data preprocessing module: Collect multilingual datasets and systematically preprocess the parallel sentence pairs therein, including a data cleaning unit, a data recognition unit, and a data segmentation unit; Retrieval-enhanced database module: Encode the parallel sentence pairs in the multilingual dataset to build a vector database; Domain topic recognition module: Use a pre-trained language model to identify the domain information of the sentence to be translated and obtain the corresponding domain information; Text similarity retrieval module: Use the sentence to be translated as a query and use two text similarity retrieval algorithms to retrieve in the retrieval-enhanced database to retrieve the parallel sentence pair with the highest similarity and form two prompt templates; Translation inference module based on a prompt template: Provide the retrieval results to the large model with the help of the prompt template. The large model realizes multilingual machine translation under the prompt of similar texts and domain information, and performs quality evaluation on the generated translations, and selects the best translation as the final translation result.

6. An apparatus as claimed in claim 6, wherein, The data preprocessing module includes: Dataset cleaning unit: Standardize the character format, parse and replace escape characters, clean special characters, filter parallel sentence pairs with too large or too small lengths, and delete duplicate sentence pairs; Data recognition unit: Re-recognize the dataset in multiple languages and divide the data according to different domains; Data segmentation unit: Divide the dataset described by the data recognition unit into a training set, a validation set, and a test set, and perform word segmentation and byte pair encoding processing.

7. An electronic device, characterized in that, The device includes at least one processor; at least one GPU computing card; and a memory communicatively connected to the processor; where: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that at least one processor or at least one GPU computing card can execute the method described in claims 1-4.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method described in claims 1-4.

Images