A vertical domain translation model training method and storage medium

Abstract

This invention relates to the field of machine translation technology, and particularly to a vertical domain translation model training method and storage medium. The translation model employs an encoder-decoder architecture. The training method involves the following steps: first, inputting bilingual text into the translation model and training sentence vectors; then, fine-tuning the trained translation model using high-resource vertical domain data, which consists of high-resource sentences; next, retrieving high-resource sentences using low-resource sentences, and inputting both the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the decoder's output state; finally, performing contextual nearest neighbor retrieval based on the decoder's output state to obtain the translation output probability distribution. In essence, this translation model training method utilizes high-resource vertical domain data to enhance low-resource vertical domain data, thereby achieving better translation results.

Description

Technical Field

[0001] This invention relates to the field of machine translation technology, and in particular to a method for training and storing a translation model in a vertical domain. Background Technology

[0002] With the development of the internet, the continuous expansion of available data, and the improvement of computer hardware capabilities, the performance of machine translation models, represented by neural machine translation models, has been significantly improved. They have moved from the laboratory to industry and are gradually being implemented to serve various sectors. Machine translation models can be divided into general-purpose machine translation models and domain-specific machine translation models based on the domain of the corpus used. Due to the massive scale of news data, the performance of general-purpose machine translation models has reached a relatively satisfactory level. However, their performance in domain-specific translation is limited by the scale of the data.

[0003] In previous machine translation based on translation memory, the methods of retrieval instance fusion were relatively simple, which could introduce significant noise into the original model. For example, the target sentence obtained through similarity retrieval was directly concatenated to the sentence to be translated, or further, parallel data alignment information was used to filter out some words to reduce the concatenation length. These methods can, to some extent, achieve domain transfer from high-resource languages ​​to low-resource languages, guiding low-resource languages ​​to achieve better translation performance in vertical domains. However, these methods significantly increase the length of the source sentence, noticeably impacting encoding speed. Second, these methods cannot concatenate too many sentences; too many retrieved sentences can "overwhelm" the sentence to be translated, making the translation less faithful to the original text. Third, these methods often retrieve similar sentences through word coverage, which may not help improve translation performance. Fourth, these methods only combine sentence-level information; for vertical domains, the corpus is sparse, and even for high-resource languages, the data scale is not large enough to obtain effective information in coarse-grained retrieval. Summary of the Invention

[0004] To address the issue of low translation performance of existing machine translation models in vertical domains, this invention provides a method for training and storing translation models in vertical domains.

[0005] The solution to the technical problem of this invention is to provide a method for training a translation model in a vertical domain, wherein the translation model is an encoder-decoder architecture, and includes the following steps:

[0006] Obtain the translation model and bilingual text, input the bilingual text into the translation model, and train the translation model with sentence vectors;

[0007] Acquire high-resource vertical domain data, which consists of high-resource sentences, and use the high-resource vertical domain data to fine-tune the trained translation model;

[0008] Acquire low-resource vertical domain data, which consists of low-resource sentences. Use the low-resource sentences to retrieve high-resource sentences. Input the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder.

[0009] Based on the output state of the decoder, a contextual nearest neighbor search is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution is then used to adjust the current output state of the decoder to generate a translation output probability distribution.

[0010] The fine-tuning of the trained translation model involves freezing all encoder parameters and fine-tuning all decoder parameters based on high-resource vertical domain data. The specific steps for inputting low-resource sentences and retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the decoder's output state include: creating a memory bank; encoding each high-resource sentence using the encoder and obtaining its sentence vector through an average pooling layer; storing the high-resource sentences and their sentence vectors in the memory bank; acquiring low-resource vertical domain data; inputting low-resource sentences into the memory bank for retrieval to obtain at least one high-resource sentence; and inputting the low-resource sentences and high-resource sentences into the encoder for cross-language multi-instance fusion, calculating based on each decoding time step and each layer of the cross-attention module using a multi-head attention mechanism.

[0011] Preferably, the encoder adopts the Transformers model.

[0012] Preferably, the process of acquiring a translation model and bilingual text, inputting the bilingual text into the translation model, and training sentence vectors in the translation model specifically includes the following steps:

[0013] The bilingual text is input into the translation model and encoded by an encoder to obtain sample sentence vectors;

[0014] One pair is extracted from the sample sentence vectors as positive samples, and the positive samples are combined with an interval-based optimization method for training.

[0015] Preferably, the loss function for training the positive samples using an interval-based optimization method is:

[0016]

[0017] in, denoted as positive sample; N represents the number of samples, each sample containing parallel sentence pairs of bilingual text; cos represents cosine similarity; m represents hyperparameter.

[0018] Preferably, low-resource vertical domain data is acquired, and the low-resource sentences are input into a memory for retrieval to obtain at least one high-resource sentence. Specifically, the low-resource sentences are input into the memory, and vector distance retrieval is performed using the Faiss tool to obtain at least one high-resource sentence.

[0019] Preferably, the low-resource sentence and the high-resource sentence are input into the encoder for cross-language multi-instance fusion. Based on each decoding time step and each layer of the cross-attention module in the decoder, a multi-head attention mechanism is used for computation to achieve knowledge transfer from the high-resource sentence to the low-resource sentence and obtain the output state at the decoder. Specifically, this includes the following steps:

[0020] Low-resource sentences and high-resource sentences are input into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and at the same time receives the state sequence of the high-resource sentence at the top layer of the encoder.

[0021] The output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence. The corresponding number of state representations are obtained by performing calculations on the low-resource sentence and the high-resource sentence.

[0022] A gating mechanism is set up to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder.

[0023] Preferably, the translation probability distribution of relevant high-resource sentences is obtained by performing contextual nearest neighbor retrieval based on the output state of the decoder, and the translation output probability distribution is adjusted using this translation probability distribution to generate the translation output probability distribution. Specifically:

[0024] Based on the output state of the decoder, contextual nearest neighbor retrieval is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution absorbs fine-grained information from high-resource vertical domain data to adjust the current output state of the decoder and generate a translation output probability distribution, thereby guiding the adaptation from low-resource domains to high-resource domains.

[0025] To address the aforementioned technical problems, this invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the machine translation model construction method as described above.

[0026] Compared with existing technologies, the vertical domain translation model training method and storage medium of the present invention have the following advantages:

[0027] 1. The present invention provides a vertical domain translation model training method, wherein the translation model is an encoder-decoder architecture. The training method first acquires the translation model and bilingual text, inputs the bilingual text into the translation model, and trains sentence vectors on the translation model; then, it acquires high-resource vertical domain data, which consists of high-resource sentences, and fine-tunes the trained translation model using the high-resource vertical domain data; next, it acquires low-resource vertical domain data, which consists of low-resource sentences, and retrieves high-resource sentences using the low-resource sentences, inputting the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder; finally, it performs contextual nearest neighbor retrieval based on the output state of the decoder to obtain the translation probability distribution of related high-resource sentences, and uses this translation probability distribution to adjust the current output state of the decoder to generate a translation output probability distribution. That is, the translation model training method uses high-resource vertical domain data to further improve low-resource vertical domain data to achieve better translation results.

[0028] 2. In the steps of this invention, the encoder adopts the Transformers model. First, the bilingual text is input into the translation model and encoded by the encoder to obtain sample sentence vectors. Then, one pair is extracted from the sample sentence vectors as positive samples. The positive samples are combined with the interval-based optimization method for training, so that the original translation model can be trained to have the ability to generate sentence vectors, thereby improving the quality of sentence vectors.

[0029] 3. In the steps of this invention, when fine-tuning the trained translation model, all parameters of the encoder are frozen, and all parameters of the decoder are fine-tuned based on high-resource vertical domain data. This step aims to better reflect the translation capabilities of high-resource vertical domain data, using better high-resource vertical domain data to help improve the translation capabilities of low-resource vertical domain data. Furthermore, fine-tuning enables high-resource sentences and low-resource sentences to have a unified semantic representation, thereby facilitating better knowledge transfer and effectively improving translation quality.

[0030] 4. The steps of this invention for obtaining the output state of the decoder specifically include the following steps: First, a memory bank is created. Each high-resource sentence is encoded by an encoder and its sentence vector is obtained through an average pooling layer. The high-resource sentences and their sentence vectors are stored in the memory bank. Then, low-resource vertical domain data is obtained. The low-resource sentences are input into the memory bank for retrieval to obtain at least one high-resource sentence. Finally, the low-resource sentences and high-resource sentences are input into the encoder for cross-language multi-instance fusion. Based on each decoding time step and each layer of the cross-attention module of the decoder, a multi-head attention mechanism is used for calculation to achieve knowledge transfer from high-resource sentences to low-resource sentences and obtain the output state of the decoder. In actual translation, the scale of low-resource vertical domain data is relatively limited and lacks training, while high-resource vertical domain data is richer in domain knowledge than low-resource vertical domain data. Creating a memory bank can enrich the scale of high-resource vertical domain data, which is conducive to expanding the retrieval scale of translation. Through cross-language multi-instance fusion, users can achieve stronger knowledge transfer, thereby improving the domain translation performance of low-resource sentences.

[0031] 5. In the steps of this invention, low-resource vertical domain data is obtained, and low-resource sentences are input into a memory for retrieval to obtain at least one high-resource sentence. Specifically, low-resource sentences are input into the memory, and vector distance retrieval is performed using the Faiss tool to obtain at least one high-resource sentence. Using vector distance retrieval can better discover multiple high-resource sentences with high similarity, which is beneficial to improving the translation performance of low-resource sentences.

[0032] 6. The steps of obtaining the output state at the decoder end in this invention specifically include the following steps: First, low-resource sentences and high-resource sentences are input into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and simultaneously receives the state sequence of the high-resource sentence at the top layer of the encoder. Then, the output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence. The corresponding number of state representations are obtained by performing calculations according to the low-resource sentences and high-resource sentences. Finally, a gating mechanism is set to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder end. The calculation using the attention mechanism makes the expression of word meaning more diversified, which can improve the translation effect. The weighted calculation using the gating mechanism enables the low-resource sentence to adaptively absorb knowledge, so as to achieve knowledge transfer and obtain the final output state at the decoder end.

[0033] 7. In the steps of this invention, generating a translation output probability distribution based on the output state of the decoder specifically involves: performing contextual nearest neighbor retrieval based on the output state of the decoder to obtain a relevant high-resource sentence translation probability distribution; absorbing fine-grained information from the high-resource vertical domain data to adjust the current output state of the decoder to generate a translation output probability distribution; and guiding the adaptation from the low-resource domain to the high-resource domain, further enhancing the knowledge transfer from high-resource sentences to low-resource sentences, and improving the domain translation performance of low-resource vertical domain data.

[0034] 8. The present invention also provides a computer-readable storage medium that has the same beneficial effects as the above-mentioned translation model training method in the vertical field, which will not be elaborated here. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the steps of a vertical domain translation model training method provided in the first embodiment of the present invention.

[0037] Figure 2 This is an example of a vertical domain translation model training method provided in the first embodiment of the present invention. Figure 1 .

[0038] Figure 3 This is a flowchart of step S1 of a vertical domain translation model training method provided in the first embodiment of the present invention.

[0039] Figure 4 This is an example of a vertical domain translation model training method provided in the first embodiment of the present invention. Figure 2 .

[0040] Figure 5 This is a flowchart of step S3 of a vertical domain translation model training method provided in the first embodiment of the present invention.

[0041] Figure 6 This is a flowchart of step S33 of a vertical domain translation model training method provided in the first embodiment of the present invention.

[0042] Figure 7 This is a flowchart of step S4 of a vertical domain translation model training method provided in the first embodiment of the present invention.

[0043] Figure 8 This is an example of a vertical domain translation model training method provided in the first embodiment of the present invention. Figure 3 . Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] Please combine Figure 1 and Figure 2 The first embodiment of the present invention provides a method for training a translation model in a vertical domain, wherein the translation model is an encoder-decoder architecture, and includes the following steps:

[0046] S1: Obtain the translation model and bilingual text, input the bilingual text into the translation model, and train the translation model with sentence vectors;

[0047] S2: Obtain high-resource vertical domain data, which consists of high-resource sentences, and use the high-resource vertical domain data to fine-tune the trained translation model;

[0048] S3: Obtain low-resource vertical domain data, which consists of low-resource sentences. Use the low-resource sentences to retrieve high-resource sentences. Input the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder.

[0049] S4: Based on the output state of the decoder, perform contextual nearest neighbor retrieval to obtain the translation probability distribution of relevant high-resource sentences, and use this translation probability distribution to adjust the current output state of the decoder to generate the translation output probability distribution.

[0050] Understandably, the steps of this invention first involve acquiring a translation model and bilingual text, inputting the bilingual text into the translation model, and training the translation model with sentence vectors. Then, high-resource vertical domain data is acquired, consisting of high-resource sentences, and the trained translation model is fine-tuned using this high-resource vertical domain data. Next, low-resource vertical domain data is acquired, consisting of low-resource sentences, and high-resource sentences are retrieved using these low-resource sentences. The low-resource sentences and the retrieved high-resource sentences are input into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder. Finally, based on the output state of the decoder, contextual nearest neighbor retrieval is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution is used to adjust the current output state of the decoder to generate a translation output probability distribution. In other words, this translation model training method utilizes high-resource vertical domain data to further improve low-resource vertical domain data, thereby achieving better translation results.

[0051] Please combine Figure 3 and Figure 4 Step S1 specifically includes the following steps:

[0052] S11: Input the bilingual text into the translation model and encode the bilingual text using an encoder to obtain sample sentence vectors;

[0053] S12: Extract one pair from the sample sentence vectors as positive samples, and combine the positive samples with an interval-based optimization method for training.

[0054] Understandably, the bilingual text is first input into the translation model and encoded by an encoder to obtain sample sentence vectors. Then, a pair of positive samples is extracted from the sample sentence vectors and trained using an interval-based optimization method. This trains the original translation model to have the ability to generate sentence vectors, thereby improving the quality of the sentence vectors.

[0055] As an optional implementation, the encoder employs the Transformers model.

[0056] Specifically, the translation model is a model based on general domain data. Its model architecture is an encoder-decoder model, using the Transformers model as the backbone model and adopting a joint training method. It uses the best sentence vector training method at present. During translation training, the encoder encodes the input bilingual text separately and combines it with the interval-based optimization method to obtain a translation training model with sentence vector generation capability, thereby improving the quality of sentence vectors.

[0057] Understandably, the bilingual text is described as "Chinese / English sentences and Chinese / Japanese sentences". During training, the Chinese-English sentence pairs and the Chinese-Japanese sentence pairs are encoded using encoders to encode the Chinese / English sentences and the Chinese / Japanese sentences respectively. Optionally, the encoder uses a shared translation model encoder, and the encoder model uses Transformers.

[0058] Furthermore, assuming the encoder is E, the input sequence of the bilingual text is... The encoder's output state is Where n represents the sequence length, the extracted sentence vector is represented as:

[0059]

[0060] Furthermore, using Translation Ranking loss as the objective function, combined with an interval-based optimization method, it is assumed that... For the extracted Chinese-English sentence vector pairs, As positive samples, other English sentences are then drawn from the same batch of samples as negative samples for that batch, and denoted as... This completes the construction of a training sample.

[0061] As an optional implementation, the loss function for training positive samples using a margin-based optimization method is:

[0062]

[0063] in, denoted as positive sample; N represents the number of samples, each sample containing parallel sentence pairs of bilingual text; cos represents cosine similarity; m represents hyperparameter.

[0064] Understandably, cosine similarity is used to calculate the similarity between the sentence vectors of Chinese and English sentences. The first part of the loss function aims to make the sentence vectors of parallel sentence pairs in the sample as similar as possible, while making the vectors of other sentences as dissimilar as possible; the second part is the translation loss, usually cross-entropy loss. Furthermore, when calculating cosine similarity, a hyperparameter *m* is added to control the margin between the similarity of parallel sentence pairs and other sentences. Increasing *m* makes the difference in similarity between parallel sentence pairs and other sentences larger, which is beneficial for subsequent data modification.

[0065] Furthermore, step S2 specifically involves freezing all parameters of the encoder and fine-tuning all parameters of the decoder based on high-resource vertical domain data when fine-tuning the trained translation model. This step aims to better reflect the translation capabilities of high-resource vertical domain data, using better high-resource vertical domain data to help improve the translation capabilities of low-resource vertical domain data. Moreover, fine-tuning enables high-resource sentences and low-resource sentences to have a unified semantic representation, thereby facilitating better knowledge transfer and effectively improving translation quality.

[0066] Understandably, high-resource vertical domain data consists of Chinese-English sentence pairs, while low-resource vertical domain data consists of Chinese-Japanese sentence pairs. When fine-tuning the trained translation model, all parameters of the encoder are frozen, and all parameters of the decoder are fine-tuned based on the high-resource vertical domain data. That is, the training data is changed. It should be noted that the output of the trained translation model is the same as the output of the untrained translation model. The fine-tuning is simply to enable the high-resource vertical domain data to help improve the translation performance of low-resource sentences, so that high-resource sentences and low-resource sentences have a unified semantic representation, so as to better transfer knowledge and effectively improve translation quality.

[0067] Please see Figure 5 Step S3 specifically includes the following steps:

[0068] S31: Create a memory bank, encode each high-resource sentence through an encoder and obtain the sentence vector of the high-resource sentence through an average pooling layer, and store the high-resource sentence and sentence vector in the memory bank;

[0069] S32: Obtain low-resource vertical domain data, input the low-resource sentences into the memory for retrieval to obtain at least one high-resource sentence;

[0070] S33: Input low-resource sentences and high-resource sentences into the encoder and perform cross-language multi-instance fusion. Based on each decoding time step of the decoder and each layer of cross-attention module, a multi-head attention mechanism is used to calculate the knowledge transfer from high-resource sentences to low-resource sentences to obtain the output state of the decoder.

[0071] Understandably, the steps of this invention first involve creating a memory bank, encoding each high-resource sentence using an encoder, obtaining the sentence vector of the high-resource sentence through an average pooling layer, and storing the high-resource sentence and sentence vector in the memory bank. Then, low-resource vertical domain data is acquired, and the low-resource sentences are input into the memory bank for retrieval to obtain at least one high-resource sentence. Finally, the low-resource and high-resource sentences are input into the encoder for cross-language multi-instance fusion. Multi-head attention is used to calculate based on each decoding time step and each layer of the cross-attention module in the decoder, achieving knowledge transfer from low-resource sentences to high-resource sentences to obtain the output state at the decoder end. In actual translation, the scale of low-resource vertical domain data is limited and lacks training, while high-resource vertical domain data is richer in domain knowledge. Creating a memory bank can enrich the scale of high-resource vertical domain data, facilitating the expansion of translation retrieval scale. Through cross-language multi-instance fusion, users can achieve stronger knowledge transfer, thereby improving the domain translation performance of low-resource sentences.

[0072] Furthermore, step S32 specifically involves inputting the low-resource sentence into the memory and using the Faiss tool to perform vector distance retrieval to obtain at least one high-resource sentence. Using vector distance retrieval can effectively discover multiple high-resource sentences with high similarity, which is beneficial for improving the translation performance of low-resource sentences.

[0073] Specifically, low-resource sentences are retrieved from the memory using the Faiss tool to obtain at least one high-resource sentence. Vector distance retrieval uses the inner product as the distance metric; vectors with larger inner products have higher similarity. This method is used to find similar high-resource sentences for low-resource sentences.

[0074] Please combine Figure 6 and Figure 8 Step S33 specifically includes the following steps:

[0075] S331: Input the low-resource sentence and the high-resource sentence into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and at the same time receives the state sequence of the high-resource sentence at the top layer of the encoder.

[0076] S332: The output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using the attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence, and the corresponding number of state representations are obtained by performing the corresponding number of calculations based on the low-resource sentence and the high-resource sentence.

[0077] S333: Set up a gating mechanism to perform weighted calculations on the obtained state representations to obtain the final output state of the decoder.

[0078] Understandably, in the steps of this invention, low-resource sentences and high-resource sentences are first input into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and simultaneously receives the state sequence of the high-resource sentence at the top layer of the encoder. Then, the output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the decoder's state representation based on the context of the high-resource sentence. A corresponding number of state representations are obtained by performing calculations according to the low-resource sentences and high-resource sentences. Finally, a gating mechanism is set to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder. The calculation using the attention mechanism makes the expression of word meaning more diversified, which can improve the translation effect. The weighted calculation using the gating mechanism enables the low-resource sentence to adaptively absorb knowledge, thereby achieving knowledge transfer to obtain the final output state at the decoder.

[0079] Specifically, in step S331, the low-resource sentence and the high-resource sentence are input into the encoder. Each layer of the decoder's cross-attention module receives the output of the decoder's self-attention module, and simultaneously receives the state sequence of the high-resource sentence at the top layer of the encoder. Specifically, the low-resource sentence and the high-resource sentence act on each layer of the decoder's cross-attention module, and the cross-attention module receives the output of the self-attention module. ,in This represents the number of layers in the decoder, and also receives the state sequence of the top layer of the encoder. The logical expression is obtained as follows:

[0080]

[0081] Optionally, a multi-head attention mechanism allows each attention mechanism to optimize different feature parts of each sentence, thereby balancing the bias that may be produced by the same attention mechanism, giving sentences more diverse expressions, and thus improving the expressive performance of the trained model.

[0082] Further, in step S332, the output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence. Additionally, a corresponding number of state representations are obtained by performing calculations based on the low-resource and high-resource sentences. Specifically, this involves the decoder's self-attention output state. Both need and To enable interaction, knowledge also needs to be extracted from high-resource sentences in the search.

[0083] Specifically, assuming that it was obtained through retrieval A set of similar sentences derived from high-resource vertical domain data, whose top-level state representation is denoted as... ,make An attention mechanism is used to compute the decoder state representation based on the high-resource sentence context by performing attention mechanism computation on the K top-level state sequences. for:

[0084]

[0085] Where k represents the k-th retrieval instance, and for each decoding layer, K+1 cross-attention calculations are performed simultaneously to obtain K+1 state representations. .

[0086] Furthermore, in step S333, a gating mechanism is set to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder. Specifically, the gating mechanism is set to enable the representation of low-resource language to be weighted. Capable of adaptively absorbing knowledge, the gating mechanism provides a set of normalized weights for adaptive knowledge transfer, resulting in the weighted output of the cross-attention module:

[0087]

[0088] Where W, U, and V represent matrices; This indicates the output status at the decoder end.

[0089] Optionally, based on the output state at the decoder end, the low-resource vertical domain data is fine-tuned so that the low-resource sentences can learn similar high-resource sentences during decoding, and adaptively form the decoder's state output that meets the domain translation requirements.

[0090] Please see Figure 7 Step S4 specifically involves performing contextual nearest neighbor retrieval based on the output state of the decoder to obtain the relevant high-resource sentence translation probability distribution. This translation probability distribution is then used to absorb fine-grained information from the high-resource vertical domain data to adjust the current output state of the decoder and generate a translation output probability distribution, thereby guiding the adaptation from the low-resource domain to the high-resource domain.

[0091] Understandably, the specific steps of generating a translation output probability distribution based on the output state of the decoder in this invention are as follows: based on the output state of the decoder, a contextual nearest neighbor search is performed to obtain a relevant high-resource sentence translation probability distribution; this translation probability distribution is then used to absorb fine-grained information from the high-resource vertical domain data to adjust the current output state of the decoder and generate a translation output probability distribution, thereby guiding the adaptation from the low-resource domain to the high-resource domain, further enhancing the knowledge transfer from high-resource sentences to low-resource sentences, and improving the domain translation performance of low-resource vertical domain data.

[0092] Specifically, KNN-MT (K-nearest-neighbor Machine Translation) is used to assist in the domain translation of low-resource vertical domain data. Then, the translation probability output distribution of the original low-resource sentences is corrected by the translation probability output distribution of similar high-resource sentences to obtain domain adaptability. That is, it guides low-resource vertical domain data to adapt from the current domain to other domains, further enhances the knowledge transfer from low-resource sentences to high-resource sentences, and improves the domain translation performance of low-resource vertical domain data.

[0093] Specifically, the difference in the output state of the decoder obtained from high-resource vertical domain data and low-resource vertical domain data after cross-language multi-instance fusion is reduced, making KNN-MT, which was originally only applicable to translation of the same language but different domains, feasible.

[0094] Understandably, contextual nearest neighbor retrieval uses the decoder's output state and the target vocabulary to be generated as key-value pairs to form a database. The logical expression is as follows:

[0095]

[0096] in, Let be the decoder output state at step t; x represents the source sentence. This represents the translated sentence generated by the decoder before time t. To generate the target vocabulary, that is, to record the output state at each step of the decoder for a pair of parallel sentences through this logical expression, as well as the vocabulary that should be output.

[0097] Furthermore, in the translation of low-resource sentences, the logical operation for each time step of the decoder is as follows:

[0098]

[0099] This is represented as follows: In the translation and decoding process of low-resource sentences, at step t, KNN-MT uses the output state of the decoder to perform retrieval, and generates a kNN translation probability distribution based on the output state of similar high-resource sentences retrieved; where... This is represented by the decoder output state of a high-resource sentence in the database, compared to the decoder output state of the current low-resource sentence. Similarity calculation is performed; d is the distance metric; T is the temperature coefficient; I is the indicator function, when... Corresponding to the output state of the decoder of high-resource sentences in the database When they are the same, they will be determined based on their similarity. This leads to an increase in the probability of successful translation.

[0100] Furthermore, the final translation result is obtained through calculation:

[0101]

[0102] Understandably, the decoder output state after fusing high-resource and low-resource vertical domain data across multiple instances in a language is further improved by using contextual nearest neighbor retrieval and weighted fusion of multiple translation probability distributions to enhance the domain translation performance of low-resource vertical domain data.

[0103] In summary, by introducing sentence vector training, the encoder output can directly form sentence vectors for memory retrieval without requiring additional training of other modules for sentence vector encoding. This also promotes unified multilingual representation within the encoder. Fine-tuning with high-resource vertical domain data enhances the decoder's domain generation capability without affecting the encoder's unified representation. Cross-lingual multi-instance fusion allows low-resource sentences to reference similar high-resource sentences during decoding, achieving knowledge transfer from high-resource to low-resource sentences. Finally, contextual nearest neighbor retrieval, based on the decoder's output state with unified semantics after cross-lingual instance fusion, further improves the domain translation performance of low-resource sentences by weighted fusion of multiple translation output probability distributions after cross-lingual KNN database retrieval.

[0104] The second embodiment of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a machine translation model construction method provided in the first embodiment of the present invention.

[0105] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0106] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to the invention.

[0107] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It is particularly important to note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] Compared with existing technologies, the vertical domain translation model training method and storage medium of the present invention have the following advantages:

[0110] 1. The present invention provides a vertical domain translation model training method, wherein the translation model is an encoder-decoder architecture. The training method first acquires the translation model and bilingual text, inputs the bilingual text into the translation model, and trains sentence vectors on the translation model; then, it acquires high-resource vertical domain data, which consists of high-resource sentences, and fine-tunes the trained translation model using the high-resource vertical domain data; next, it acquires low-resource vertical domain data, which consists of low-resource sentences, and retrieves high-resource sentences using the low-resource sentences, inputting the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder; finally, it performs contextual nearest neighbor retrieval based on the output state of the decoder to obtain the translation probability distribution of related high-resource sentences, and uses this translation probability distribution to adjust the current output state of the decoder to generate a translation output probability distribution. That is, the translation model training method uses high-resource vertical domain data to further improve low-resource vertical domain data to achieve better translation results.

[0111] 2. In the steps of this invention, the encoder adopts the Transformers model. First, the bilingual text is input into the translation model and encoded by the encoder to obtain sample sentence vectors. Then, one pair is extracted from the sample sentence vectors as positive samples. The positive samples are combined with the interval-based optimization method for training, so that the original translation model can be trained to have the ability to generate sentence vectors, thereby improving the quality of sentence vectors.

[0112] 3. In the steps of this invention, when fine-tuning the trained translation model, all parameters of the encoder are frozen, and all parameters of the decoder are fine-tuned based on high-resource vertical domain data. This step aims to better reflect the translation capabilities of high-resource vertical domain data, using better high-resource vertical domain data to help improve the translation capabilities of low-resource vertical domain data. Furthermore, fine-tuning enables high-resource sentences and low-resource sentences to have a unified semantic representation, thereby facilitating better knowledge transfer and effectively improving translation quality.

[0113] 4. The steps of this invention for obtaining the output state of the decoder specifically include the following steps: First, a memory bank is created. Each high-resource sentence is encoded by an encoder and its sentence vector is obtained through an average pooling layer. The high-resource sentences and their sentence vectors are stored in the memory bank. Then, low-resource vertical domain data is obtained. The low-resource sentences are input into the memory bank for retrieval to obtain at least one high-resource sentence. Finally, the low-resource sentences and high-resource sentences are input into the encoder for cross-language multi-instance fusion. Based on each decoding time step and each layer of the cross-attention module of the decoder, a multi-head attention mechanism is used for calculation to achieve knowledge transfer from high-resource sentences to low-resource sentences and obtain the output state of the decoder. In actual translation, the scale of low-resource vertical domain data is relatively limited and lacks training, while high-resource vertical domain data is richer in domain knowledge than low-resource vertical domain data. Creating a memory bank can enrich the scale of high-resource vertical domain data, which is conducive to expanding the retrieval scale of translation. Through cross-language multi-instance fusion, users can achieve stronger knowledge transfer, thereby improving the domain translation performance of low-resource sentences.

[0114] 5. In the steps of this invention, low-resource vertical domain data is obtained, and low-resource sentences are input into a memory for retrieval to obtain at least one high-resource sentence. Specifically, low-resource sentences are input into the memory, and vector distance retrieval is performed using the Faiss tool to obtain at least one high-resource sentence. Using vector distance retrieval can better discover multiple high-resource sentences with high similarity, which is beneficial to improving the translation performance of low-resource sentences.

[0115] 6. The steps of obtaining the output state at the decoder end in this invention specifically include the following steps: First, low-resource sentences and high-resource sentences are input into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and simultaneously receives the state sequence of the high-resource sentence at the top layer of the encoder. Then, the output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence. The corresponding number of state representations are obtained by performing calculations according to the low-resource sentences and high-resource sentences. Finally, a gating mechanism is set to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder end. The calculation using the attention mechanism makes the expression of word meaning more diversified, which can improve the translation effect. The weighted calculation using the gating mechanism enables the low-resource sentence to adaptively absorb knowledge, so as to achieve knowledge transfer and obtain the final output state at the decoder end.

[0116] 7. In the steps of this invention, generating a translation output probability distribution based on the output state of the decoder specifically involves: performing contextual nearest neighbor retrieval based on the output state of the decoder to obtain a relevant high-resource sentence translation probability distribution; absorbing fine-grained information from the high-resource vertical domain data to adjust the current output state of the decoder to generate a translation output probability distribution; and guiding the adaptation from the low-resource domain to the high-resource domain, further enhancing the knowledge transfer from high-resource sentences to low-resource sentences, and improving the domain translation performance of low-resource vertical domain data.

[0117] 8. The present invention also provides a computer-readable storage medium that has the same beneficial effects as the above-mentioned translation model training method in the vertical field, which will not be elaborated here.

[0118] The foregoing has provided a detailed description of a vertical domain translation model training method and storage medium disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a translation model in a vertical domain, wherein the translation model is an encoder-decoder architecture, characterized in that: Includes the following steps: Obtain the translation model and bilingual text, input the bilingual text into the translation model, and train the translation model with sentence vectors; Acquire high-resource vertical domain data, which consists of high-resource sentences, and use the high-resource vertical domain data to fine-tune the trained translation model; Acquire low-resource vertical domain data, which consists of low-resource sentences. Use the low-resource sentences to retrieve high-resource sentences. Input the low-resource sentences and the retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder. Based on the output state of the decoder, a contextual nearest neighbor search is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution is then used to adjust the current output state of the decoder to generate a translation output probability distribution. Among them, when fine-tuning the trained translation model, all parameters of the encoder are frozen, and all parameters of the decoder are fine-tuned based on high-resource vertical domain data; The specific steps for inputting low-resource sentences and retrieved high-resource sentences into the encoder for cross-language multi-instance fusion to obtain the output state of the decoder include the following: creating a memory bank, encoding each high-resource sentence through the encoder and obtaining the sentence vector of the high-resource sentence through an average pooling layer, and storing the high-resource sentences and sentence vectors in the memory bank; obtaining low-resource vertical domain data, inputting low-resource sentences into the memory bank for retrieval to obtain at least one high-resource sentence; Low-resource sentences and high-resource sentences are input into the encoder for cross-language multi-instance fusion. The calculation is performed through a multi-head attention mechanism based on each decoding time step and each layer of cross-attention module of the decoder.

2. The vertical domain translation model training method as described in claim 1, characterized in that: The encoder uses the Transformers model.

3. The vertical domain translation model training method as described in claim 1, characterized in that: The process of obtaining a translation model and bilingual text, inputting the bilingual text into the translation model, and training sentence vectors in the translation model includes the following steps: The bilingual text is input into the translation model and encoded by an encoder to obtain sample sentence vectors; One pair is extracted from the sample sentence vectors as positive samples, and the positive samples are combined with an interval-based optimization method for training.

4. The vertical domain translation model training method as described in claim 3, characterized in that: The loss function for training the positive samples using an interval-based optimization method is: in, denoted as positive sample; N represents the number of samples, each sample containing parallel sentence pairs of bilingual text; cos represents cosine similarity; m represents hyperparameter.

5. The vertical domain translation model training method as described in claim 1, characterized in that: To obtain low-resource vertical domain data, the low-resource sentences are input into a database memory for retrieval to obtain at least one high-resource sentence. Specifically, the low-resource sentences are input into the database memory, and vector distance retrieval is performed using the Faiss tool to obtain at least one high-resource sentence.

6. The vertical domain translation model training method as described in claim 1, characterized in that: Low-resource sentences and high-resource sentences are input into the encoder for cross-language multi-instance fusion. Based on each decoding time step and each layer of cross-attention module in the decoder, a multi-head attention mechanism is used for computation to achieve knowledge transfer from high-resource sentences to low-resource sentences and obtain the output state at the decoder end. The specific steps include the following: Low-resource sentences and high-resource sentences are input into the encoder. The cross-attention module of each layer of the decoder receives the output of the self-attention module of the decoder, and at the same time receives the state sequence of the high-resource sentence at the top layer of the encoder. The output of the self-attention module and the state sequence of the high-resource sentence at the top layer of the encoder are calculated using an attention mechanism to obtain the state representation of the decoder based on the context of the high-resource sentence. The corresponding number of state representations are obtained by performing calculations on the low-resource sentence and the high-resource sentence. A gating mechanism is set up to perform weighted calculations on the obtained state representations to obtain the final output state at the decoder.

7. The vertical domain translation model training method as described in claim 1, characterized in that: Based on the output state of the decoder, a contextual nearest neighbor search is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution is then used to adjust the current output state of the decoder to generate the translation output probability distribution. Specifically: Based on the output state of the decoder, contextual nearest neighbor retrieval is performed to obtain the translation probability distribution of relevant high-resource sentences. This translation probability distribution absorbs fine-grained information from high-resource vertical domain data to adjust the current output state of the decoder and generate a translation output probability distribution, thereby guiding the adaptation from low-resource domains to high-resource domains.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, it implements the vertical domain translation model training method as described in any one of claims 1-7.

Images