Domain machine translation method and system for dynamically fusing k neighbor knowledge

Abstract

The invention relates to a field machine translation method and system for dynamically fusing k neighbor knowledge, and belongs to the field of machine translation. The method comprises the following steps: training a basic teacher translation model by using general field data; inputting the target domain data into the basic teacher translation model, and constructing a translation knowledge base of the target domain; performing fine tuning on the basic teacher translation model by using the training corpus of the target domain to obtain a target domain translation model, dynamically extracting related knowledge from the translation knowledge base in the fine tuning process, and gradually training the remaining corpus until the fine tuning of the model is completed; inputting the target domain training corpus into the finely-adjusted target domain translation model, updating the translation knowledge base, and deleting the knowledge base constructed in the previous round; and dynamically adjusting the learning rate until a preset maximum number of iterations is reached, thereby obtaining a final target field machine translation model. The method disclosed by the invention has excellent advantages in the field adaptation of limited data.

Description

Technical Field

[0001] This invention relates to a domain machine translation method and system that dynamically incorporates k-nearest neighbor knowledge, and belongs to the field of machine translation. Background Technology

[0002] Domain-specific translation is an important research direction in natural language processing, aiming to eliminate information exchange barriers between different domains. Traditional machine translation methods, such as rule-based translation models and statistical machine translation, while achieving good results in general translation tasks, have limitations when handling domain-specific texts. The main challenges include the large amount of specialized terminology and unique expressions in different domains, as well as the need for accurate translation in specific contexts. Traditional methods often struggle to achieve high-quality translations for these domain-specific vocabulary and sentence structures, especially in low-resource domains.

[0003] To address these issues, researchers have proposed various methods to enhance domain-specific translation, improving model performance in particular domains. Effective techniques include knowledge graphs, transfer learning, meta-learning, and k-nearest neighbor machine translation. Knowledge graphs integrate domain expert knowledge, enabling models to identify and accurately use domain-specific terminology during translation. Transfer learning transfers general or related domain models to the target domain, helping models leverage source domain knowledge and patterns to solve domain-specific language problems, thus improving translation accuracy and naturalness. Meta-learning allows models to quickly adapt to new tasks with limited data, making it particularly suitable for translation scenarios in low-resource domains. k-nearest neighbor machine translation provides immediate domain knowledge by dynamically retrieving instances similar to the current content, significantly improving translation quality; however, it still faces challenges in handling noise and knowledge selection under low-resource conditions. While these methods improve domain-specific translation to varying degrees, optimizing the full utilization of general domain data knowledge in low-resource domains remains crucial.

[0004] To further improve the performance of domain adaptation, this invention proposes a domain machine translation method that dynamically incorporates k-nearest neighbor knowledge. By combining techniques such as reinforcement learning and k-nearest neighbor search, it dynamically selects and incorporates k-nearest neighbor knowledge related to the target domain to enhance the model's multi-domain adaptability, thereby further optimizing the performance of domain translation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a domain machine translation method and system that dynamically incorporates k-nearest neighbor knowledge. This method effectively utilizes shared knowledge between the source and target domains when domain data is insufficient, thereby improving adaptation performance. Furthermore, this method employs techniques such as reinforcement learning and k-nearest neighbor search to effectively enhance translation performance.

[0006] The technical solution of this invention is: a domain machine translation method that dynamically incorporates k-nearest neighbor knowledge, the method comprising:

[0007] Step 1: Collect parallel data from the general domain and the target domain, preprocess the collected data, and then use the general domain data to train a basic teacher translation model;

[0008] Step 2: Input the target domain data into the basic teacher translation model, extract the hidden layer representation of the model decoder, and build a translation knowledge base for the target domain;

[0009] Step 3: Fine-tune the basic teacher translation model using training corpus in the target domain to obtain the target domain translation model. During the fine-tuning process, relevant knowledge is dynamically extracted from the translation knowledge base, and the remaining corpus is gradually trained until the model fine-tuning is completed.

[0010] Step 4: Input the target domain training corpus into the fine-tuned target domain translation model, update the translation knowledge base, and delete the knowledge base built in the previous round.

[0011] Step 5: Repeat Step 3 to Step 4, dynamically adjusting the learning rate until the preset maximum number of iterations is reached, thereby obtaining the final machine translation model for the target domain, which is then used for data translation.

[0012] Furthermore, Step 1 includes:

[0013] Step 1.1: Collect parallel data from the general domain and the target domain, and preprocess the collected data. Preprocessing includes deduplication and filtering. The purpose of preprocessing is to filter out low-quality data, thereby improving the overall reliability of the data. Based on the attributes of the target domain, organize the parallel datasets of the target domain into D = (S, D1, D2, ..., D...). n (1≤i≤n), where D i Let S be the corpus set for the i-th target domain; let S be the general domain data, and let (s,t) be the parallel pair of general domain data s and target domain data t in each domain dataset.

[0014] Step 1.2: Input the preprocessed general domain data S into the basic neural machine translation model in batches for training until the convergence condition is met, thereby constructing a basic teacher translation model Θ. base .

[0015] Furthermore, step 2 includes:

[0016] The training data from the target domain is fed into the basic teacher translation model. baseThe last hidden vector of the decoder is selected as the key K, and the distribution of the next corresponding target word is taken as the value V. The entire target domain dataset is iterated through to construct a translation knowledge base for the target domain, which is represented as follows:

[0017]

[0018] Where (x,y)∈(X,Y) represents a parallel two-statement, (X,Y) represents the parallel data of the entire target domain, and f(x,y) represents the parallel two-statement. <t y represents the hidden vector representation of the translation context at time t. <t This indicates that the model has already predicted and generated the target sentence at prediction time t, y t To predict the truth value of the next target word, in (x,y), x represents the complete source language sentence, and y represents the complete target language sentence. For example, if the source sentence is "I love China," and time t is set to 2, the generated target sentence is... <sos> l( <sos>(Indicates the starting prediction symbol), the next prediction word y t For love; the building tool used is the FAISS toolkit, a library developed by Facebook in 2017. It can pre-train pre-built data storage vectors, cluster and quantize the vectors, thereby improving search speed and space efficiency.

[0019] Furthermore, step 3 includes:

[0020] Step 3.1: Freeze all parameters of the basic teacher translation model to retain its general domain knowledge, and add a domain-specific adapter layer at the top of the encoder and decoder to learn domain-specific knowledge; after the forward propagation of the decoder, introduce an adaptive k-value (AD-K) module and a confidence module (CONF);

[0021] Step 3.2: Fine-tune the target domain translation model after adding the adapter layer using parallel data from the target domain; at each decoding time t, dynamically introduce knowledge from the translation knowledge base to minimize the distance between the model's generated distribution, the final fusion distribution, and the real distribution, thereby optimizing model performance; at the same time, introduce reinforcement learning strategies to further improve model performance by adjusting the relative weights of the loss terms.

[0022] Further, in Step 3.1, the confidence module is used to calculate the confidence score of the target domain translation model for the current translated word at time t, thereby determining the weights in the fusion process. When the confidence score λ is higher than the preset threshold τ, it means that the model's translation accuracy has reached a sufficiently high level, so the translation knowledge base retrieval step is skipped. At this time, a small perturbation value ε is added to the distribution generated by the target domain translation model to replace the final fusion distribution at that time. When the confidence score λ is lower than the set threshold τ, the translation knowledge base retrieval is performed according to the normal process. The formulas for dynamic fusion and confidence are:

[0023]

[0024] λ=σ(w2(RELU((w1h t )+b1))+b2)

[0025] Among them, P nmt P represents the generation distribution of the model. final Let P represent the final fusion distribution. kNN The translation memory retrieval distribution is defined by τ, which represents the set threshold, and λ is the confidence score. After the input sentence x passes through the NMT model, the final feature vector h of the decoder at time t is obtained. t w1 is the weight matrix of the first feedforward layer of the confidence module, b1 is its bias vector, and ReLU is the activation function; w2 is the weight matrix of the second feedforward layer of the confidence module, b2 is its bias vector, and σ is the sigmoid activation function; the retrieval distribution P of the translation memory. kNN Represented as:

[0026]

[0027] Where d(.) represents the Euclidean distance function, and T is the temperature hyperparameter. h represents the hidden vector representation of the context stored in the translation knowledge base. t This is the context vector representation generated by the decoder at time t.

[0028] Further, in step 3.1, the adaptive k-value module is used to dynamically adjust the k-value retrieved from the translation knowledge base each time, including:

[0029] First, the nearest neighbor vector for each sample is retrieved based on the input features and the set maximum value maxk. Then, this vector is input into subsequent modules to dynamically adjust the value of k. For input features with high confidence, a smaller k value is chosen to reduce noise interference; while for input features with low confidence, a larger k value should be chosen to enhance the diversity and robustness of the retrieval results. The formula for calculating the k value is as follows:

[0030]

[0031] Where, λ t This represents the confidence level calculated at time t. h represents the average confidence level of the current batch. t q1 is the hidden vector of the decoder at time t obtained after the input sentence x passes through the target domain translation model. q1 is the weight matrix of the first feedforward layer in the adaptive k-value module, z1 is its bias vector, and ReLU is the activation function. q2 is the weight matrix of the second feedforward layer in the adaptive k-value module, z2 is its bias vector, and σ is the sigmoid activation function. Finally, the k-value is scaled down to between 1 and maxk using clamp.

[0032] Furthermore, Step 3.2 includes:

[0033] (1) Introduce a reinforcement learning strategy to adjust the relative weights of the loss term through the intermediate reward r; the formula for calculating the intermediate reward r is as follows:

[0034] r = where(P) final (y t ) = P gold (y t ))

[0035] Where r represents the intermediate reward, P final (y t P represents the final fusion distribution formed by combining the retrieval distribution of k-nearest neighbor knowledge with the prediction distribution of the target domain translation model at time t. gold (y t ) represents the distribution of the next true target word, where is used to evaluate the consistency of the largest candidate word in the two distributions and use it as the final output. If the candidate words are the same, the output value is 1; otherwise, the output value is 0.

[0036] (2) KL divergence is used as the loss function to optimize the combination accuracy of the target domain translation model and k-nearest neighbor knowledge, aiming to narrow the distance between the two distributions and optimize the loss L that minimizes the difference between the final fusion distribution and the distribution of the real target words. fg L fg The calculation formula is as follows:

[0037]

[0038] Where, r g Let represent the intermediate reward of the target domain translation model at time t, where x represents the source sentence, y represents the corresponding target sentence, and D represents the set of training parallel sentence pairs. This represents the distribution of the next true word at time t. Let KL[.] represent the final distribution fused from the retrieval distribution at time t and the predicted distribution of the target domain translation model. KL[.] is the KL divergence. When the intermediate reward r is 1, it indicates that the fused distribution is correct. However, setting the reward term to 0 will cause excessive model fluctuations and affect the model's convergence. Therefore, the reward is set to 1 / k. When the intermediate reward is 0, it indicates that the fused distribution is incorrect. In this case, a penalty term is set to 1 / j.

[0039] (3) By minimizing the final distribution P final and model distribution P nmt The KL divergence between the two is used to optimize the distribution of the translation model in the target domain. fn The calculation formula is as follows:

[0040]

[0041] Where, r n This represents the intermediate reward predicted by the model at time t. Let P represent the prediction distribution of the target domain translation model at time t, which is obtained by adjusting the prediction distribution P of the target domain translation model. nmt With the final fusion distribution P final The KL divergence between them gradually tends to converge to the fusion distribution P. final Similarly, set the reward to 1 / n and the penalty to 1 / m.

[0042] (4) The translation loss of the target domain translation model is as follows:

[0043]

[0044] Wherein, P(y i |x i ;θ) is the source sentence x i Translate into the target sentence y i The probability, n is the number of sentences, and θ is the target domain translation model parameter;

[0045] (5) Finally, the above loss terms are weighted to obtain the final target domain translation model optimization objective, as shown below:

[0046] L = L nmt +r n Lf n +r g L fg

[0047] Where L is the final training objective, L nmt It is translation loss, r n and r g L is a hyperparameter used to balance the weights of different loss functions. fn The loss is to minimize the difference between the final fusion distribution and the predicted distribution of the target domain translation model, L. fg It is the loss that minimizes the difference between the final fusion distribution and the distribution of the true target words.

[0048] Step 4 includes:

[0049] After training in Step 3, a fine-tuned target domain model is obtained. The training corpus for the target domain is then fed line by line into the fine-tuned domain-specific model to extract the hidden layer vector representations of the model's decoder. This process not only helps build an up-to-date translation knowledge base but also ensures that the model can better capture the features and language styles of the target domain. Subsequently, to maintain the timeliness of the knowledge base and reduce information redundancy, the old knowledge base from the previous round is deleted. This step ensures that the model relies only on the latest knowledge base at the current stage, thereby improving the model's accuracy and efficiency. By continuously updating the translation knowledge base, the model can adapt to constantly changing language use and domain characteristics, making it perform better and more reliably in translation tasks. Furthermore, this dynamic update strategy effectively prevents outdated knowledge or redundant information from negatively impacting translation quality, thus improving the overall system performance.

[0050] Step 5 includes:

[0051] Repeat Steps 3 and 4, optimizing the training process by adjusting the model's learning rate until the preset convergence condition is met. As the number of translation rounds increases, the learning rate is gradually reduced to ensure that the model converges more stably in the later stages of training. This strategy aims to enable the model to quickly adapt to the data in the early stages, while gradually reducing the learning rate near convergence helps avoid overfitting and oscillations, ensuring that the model captures more accurate domain information through fine-tuning.

[0052] This invention utilizes dynamic knowledge integration. First, it measures the confidence level of the decoder output distribution and then uses a gating mechanism to determine whether to integrate the k-nearest neighbor retrieval results at the current time step, thereby effectively adjusting the degree of k-nearest neighbor knowledge integration. Simultaneously, an adaptive k-value module further reduces interference from erroneous k-nearest neighbor knowledge integration. Then, during training, a distribution-guided loss is designed to gradually bring the model's output distribution closer to the target distribution, thereby improving translation accuracy.

[0053] The present invention also provides a domain machine translation system that dynamically incorporates k-nearest neighbor knowledge, the system comprising: a module for performing the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0054] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0055] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0056] The beneficial effects of this invention are:

[0057] 1. By training a basic model using a large amount of general domain data, it has good language representation capabilities, which can provide a good starting point and knowledge foundation for model training. The dynamic knowledge integration strategy measures the confidence of the decoder output distribution and combines it with a gating mechanism to intelligently decide whether to integrate the k-nearest neighbor retrieval results at the current time step, thereby effectively adjusting the degree of k-nearest neighbor knowledge integration.

[0058] 2. The method of this invention significantly improves the flexibility and adaptability of the model, enabling it to make more accurate translation decisions in changing contexts;

[0059] 3. The introduction of the adaptive k-value module in this invention further reduces the interference caused by the erroneous k-nearest neighbor knowledge, ensuring the stability and reliability of the translation results;

[0060] 4. During the training process, the distribution-guided loss of this invention makes the model output distribution gradually approach the target distribution, thereby improving the accuracy of translation;

[0061] 5. The method of the present invention has a superior advantage in adapting to domains with limited data, and is suitable for machine translation tasks involving multiple languages ​​and domains with low resources. Attached Figure Description

[0062] Figure 1 This is a flowchart from an embodiment of the present invention;

[0063] Figure 2 This is an overall architecture diagram in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of dynamic k-nearest neighbor knowledge in an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of probability fusion in an embodiment of the present invention. Detailed Implementation

[0066] Example 1: Figures 1-4 As shown, a domain machine translation method that dynamically incorporates k-nearest neighbor knowledge is described, the method comprising:

[0067] Step 1: Collect parallel data from the general domain and the target domain, preprocess the collected data, and then use the general domain data to train a basic teacher translation model;

[0068] Furthermore, Step 1 includes:

[0069] Step 1.1: Collect parallel data from the general domain and the target domain, and preprocess the collected data. Preprocessing includes deduplication and filtering. The purpose of preprocessing is to filter out low-quality data, thereby improving the overall reliability of the data. Based on the attributes of the target domain, organize the parallel datasets of the target domain into D = (S, D1, D2, ..., D...). n (1≤i≤n), where D i Let S be the corpus set for the i-th target domain; let S be the general domain data, and let S be the parallel pair of general domain data s and target domain data t, denoted as (s,t); the specific domain datasets constructed are shown in Table 1:

[0070] Table 1 Distribution of Datasets in the English and German Domains

[0071]

[0072] Step 1.2: Input the preprocessed general domain data S into the basic neural machine translation model in batches for training until the convergence condition is met, thereby constructing a basic teacher translation model Θ. base To improve English-German bilingual translation capabilities, the basic teacher translation model directly adopted a pre-trained neural machine translation (NMT) model, which was the winner of the German-to-English news translation task in the 2019 World Machine Translation Competition (WMT).

[0073] Step 2: Input the target domain data into the basic teacher translation model, extract the hidden layer representation of the model decoder, and build a translation knowledge base for the target domain;

[0074] Furthermore, step 2 includes:

[0075] The training data from the target domain is fed into the basic teacher translation model. base The last hidden vector of the decoder is selected as the key K, and the distribution of the next corresponding target word is taken as the value V. The entire target domain dataset is iterated through to construct a translation knowledge base for the target domain, which is represented as follows:

[0076]

[0077] Where (x,y)∈(X,Y) represents a parallel two-statement, (X,Y) represents the parallel data of the entire target domain, and f(x,y) represents the parallel two-statement. <t y represents the hidden vector representation of the translation context at time t. <t This indicates that the model has already predicted and generated the target sentence at prediction time t, y t To predict the truth value of the next target word, in (x,y), x represents the complete source language sentence, and y represents the complete target language sentence. For example, if the source sentence is "I love China," and time t is set to 2, the generated target sentence is... <sos> l( <sos>(Indicates the starting prediction symbol), the next prediction word y t For love; the building tool used is the FAISS toolkit, a library developed by Facebook in 2017. It can pre-train pre-built data storage vectors, cluster and quantize the vectors, thereby improving search speed and space efficiency.

[0078] Step 3: Fine-tune the basic teacher translation model using training corpus in the target domain to obtain the target domain translation model. During the fine-tuning process, relevant knowledge is dynamically extracted from the translation knowledge base, and the remaining corpus is gradually trained until the model fine-tuning is completed.

[0079] Furthermore, step 3 includes:

[0080] Step 3.1, as follows Figure 2 As shown, all parameters of the basic teacher translation model are frozen to preserve its general domain knowledge, and a domain-specific adapter layer is added at the top of the encoder and decoder to learn domain-specific knowledge; after the forward propagation of the decoder, an adaptive k-value (AD-K) module and a confidence module (CONF) are introduced.

[0081] Step 3.2: Fine-tune the target domain translation model after adding the adapter layer using parallel data from the target domain. The main parameters for fine-tuning are set as follows: Adam optimizer with β1, β2 = (0.9, 0.98), learning rate of 5e-04, max-tokens of 1024, warmup update steps of 6000, label smoothing value of 0.1, and dropout rate of 0.1. At each decoding time t, knowledge from the translation knowledge base is dynamically introduced to minimize the distance between the model's generated distribution, the final fused distribution, and the true distribution, thereby optimizing model performance. At the same time, a reinforcement learning strategy is introduced to further improve model performance by adjusting the relative weights of the loss term.

[0082] Furthermore, in step 3.1, as follows: Figure 3 As shown, the confidence module is used to calculate the confidence score of the target domain translation model for the current translated word at time t, thereby determining the weights in the fusion process. When the confidence score λ is higher than the preset threshold τ, it means that the model's translation accuracy has reached a sufficiently high level, so the retrieval step of the translation knowledge base is skipped. At this time, a small perturbation value ε is added to the distribution generated by the target domain translation model to replace the final fusion distribution at that time. The specific fusion process of the final fusion distribution is as follows: Figure 4 As shown; when the confidence score λ is lower than the set threshold τ, the translation knowledge base retrieval proceeds according to the normal process; the formulas for dynamic fusion and confidence score calculation are:

[0083]

[0084] λ=σ(w2(RELU((w1h t )+b1))+b2)

[0085] Among them, P nmt P represents the generation distribution of the model. final Let P represent the final fusion distribution. kNN The translation memory retrieval distribution is defined by τ, which represents the set threshold, and λ is the confidence score. After the input sentence x passes through the NMT model, the final feature vector h of the decoder at time t is obtained. t w1 is the weight matrix of the first feedforward layer of the confidence module, b1 is its bias vector, and ReLU is the activation function; w2 is the weight matrix of the second feedforward layer of the confidence module, b2 is its bias vector, and σ is the sigmoid activation function; the retrieval distribution P of the translation memory. kNN Represented as:

[0086]

[0087] Where d(.) represents the Euclidean distance function, and T is the temperature hyperparameter. h represents the hidden vector representation of the context stored in the translation knowledge base. t This is the context vector representation generated by the decoder at time t.

[0088] Further, in step 3.1, the adaptive k-value module is used to dynamically adjust the k-value retrieved from the translation knowledge base each time, including:

[0089] First, the nearest neighbor vector for each sample is retrieved based on the input features and the set maximum value maxk. Then, this vector is input into subsequent modules to dynamically adjust the value of k. For input features with high confidence, a smaller k value is chosen to reduce noise interference; while for input features with low confidence, a larger k value should be chosen to enhance the diversity and robustness of the retrieval results. The formula for calculating the k value is as follows:

[0090]

[0091] Where, λ t This represents the confidence level calculated at time t. h represents the average confidence level of the current batch. t q1 is the hidden vector of the decoder at time t obtained after the input sentence x passes through the target domain translation model. q1 is the weight matrix of the first feedforward layer in the adaptive k-value module, z1 is its bias vector, and ReLU is the activation function. q2 is the weight matrix of the second feedforward layer in the adaptive k-value module, z2 is its bias vector, and σ is the sigmoid activation function. Finally, the k-value is scaled down to between 1 and maxk using clamp.

[0092] Furthermore, Step 3.2 includes:

[0093] (1) Introduce a reinforcement learning strategy to adjust the relative weights of the loss term through the intermediate reward r; the formula for calculating the intermediate reward r is as follows:

[0094] r = where(P) final (y t ) = P gold (y t ))

[0095] Where r represents the intermediate reward, P final (y t P represents the final fusion distribution formed by combining the retrieval distribution of k-nearest neighbor knowledge with the prediction distribution of the target domain translation model at time t. gold (y t ) represents the distribution of the next true target word, where is used to evaluate the consistency of the largest candidate word in the two distributions and use it as the final output. If the candidate words are the same, the output value is 1; otherwise, the output value is 0.

[0096] (2) KL divergence is used as the loss function to optimize the combination accuracy of the target domain translation model and k-nearest neighbor knowledge, aiming to narrow the distance between the two distributions and optimize the loss L that minimizes the difference between the final fusion distribution and the distribution of the real target words. fg L fg The calculation formula is as follows:

[0097]

[0098] Where, r g Let represent the intermediate reward of the target domain translation model at time t, where x represents the source sentence, y represents the corresponding target sentence, and D represents the set of training parallel sentence pairs. This represents the distribution of the next true word at time t. Let KL[.] represent the final distribution fused from the retrieval distribution at time t and the predicted distribution of the target domain translation model. KL[.] is the KL divergence. When the intermediate reward r is 1, it indicates that the fused distribution is correct. However, if the reward term is set to 0, it will cause the model to fluctuate too much, affecting the convergence of the model. Therefore, the reward is set to 1 / k. When the intermediate reward is 0, it indicates that the fused distribution is incorrect. At this time, a penalty term is set, which is set to 1 / j, k is set to 2, and j is set to 1.

[0099] (3) By minimizing the final distribution P final and model distribution P nmt The KL divergence between the two is used to optimize the distribution of the translation model in the target domain. fn The calculation formula is as follows:

[0100]

[0101] Where, r n This represents the intermediate reward predicted by the model at time t. Let P represent the prediction distribution of the target domain translation model at time t, which is obtained by adjusting the prediction distribution P of the target domain translation model. nmt With the final fusion distribution P final The KL divergence between them gradually tends to converge to the fusion distribution P. final Similarly, set the reward to 1 / n, the penalty to 1 / m, n to 2.5, and m to 5.

[0102] (4) The translation loss of the target domain translation model is as follows:

[0103]

[0104] Wherein, P(y i |x i ;θ) is the source sentence x i Translate into the target sentence y i The probability, n is the number of sentences, and θ is the target domain translation model parameter;

[0105] (5) Finally, the above loss terms are weighted to obtain the final target domain translation model optimization objective, as shown below:

[0106] L = L nmt +r n L fn +r g L fg

[0107] Where L is the final training objective, L nmt It is translation loss, r n and r g L is a hyperparameter used to balance the weights of different loss functions. fn The loss is to minimize the difference between the final fusion distribution and the predicted distribution of the target domain translation model, L. fg It is the loss that minimizes the difference between the final fusion distribution and the distribution of the true target words.

[0108] Step 4: Input the target domain training corpus into the fine-tuned target domain translation model, update the translation knowledge base, and delete the knowledge base built in the previous round.

[0109] Step 4 includes:

[0110] After training in Step 3, a fine-tuned target domain model is obtained. The training corpus for the target domain is then fed line by line into the fine-tuned domain-specific model to extract the hidden layer vector representations of the model's decoder. This process not only helps build an up-to-date translation knowledge base but also ensures that the model can better capture the features and language styles of the target domain. Subsequently, to maintain the timeliness of the knowledge base and reduce information redundancy, the old knowledge base from the previous round is deleted. This step ensures that the model relies only on the latest knowledge base at the current stage, thereby improving the model's accuracy and efficiency. By continuously updating the translation knowledge base, the model can adapt to constantly changing language use and domain characteristics, making it perform better and more reliably in translation tasks. Furthermore, this dynamic update strategy effectively prevents outdated knowledge or redundant information from negatively impacting translation quality, thus improving the overall system performance.

[0111] Step 5: Repeat Step 3 to Step 4, dynamically adjusting the learning rate until the preset maximum number of iterations is reached, thereby obtaining the final machine translation model for the target domain, which is then used for data translation.

[0112] Step 5 includes:

[0113] Repeat Steps 3 and 4, optimizing the training process by adjusting the model's learning rate until the preset convergence condition is met. Here, the maximum number of training epochs is set to 100. As the number of translation epochs increases, the learning rate will gradually decrease to ensure that the model converges more stably in the later stages of training. This strategy aims to enable the model to quickly adapt to the data in the early stages, while gradually reducing the learning rate near convergence helps avoid overfitting and oscillations, ensuring that the model captures more accurate domain information through fine-tuning.

[0114] The present invention also provides a domain machine translation system that dynamically incorporates k-nearest neighbor knowledge, the system comprising: a module for performing the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0115] The basic teacher translation model training module is used to collect parallel data from the general domain and the target domain, preprocess the collected data, and then use the general domain data to train the basic teacher translation model.

[0116] A translation knowledge base module is constructed to input target domain data into the basic teacher translation model, extract the hidden layer representation of the model decoder, and construct the translation knowledge base for the target domain.

[0117] The fine-tuning module is used to fine-tune the basic teacher translation model using training corpus in the target domain to obtain the target domain translation model. During the fine-tuning process, relevant knowledge is dynamically extracted from the translation knowledge base, and the remaining corpus is gradually trained until the model fine-tuning is completed.

[0118] The update module is used to input the target domain training corpus into the fine-tuned target domain translation model, update the translation knowledge base, and delete the knowledge base built in the previous round.

[0119] The translation module is used to repeatedly execute the fine-tuning and update modules. By dynamically adjusting the learning rate until the preset maximum number of iterations is reached, the final machine translation model for the target domain is obtained and used for data translation.

[0120] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0121] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described domain machine translation method that dynamically incorporates k-nearest neighbor knowledge.

[0122] To illustrate the translation performance of this invention, a baseline system and this invention are compared. During the inference phase, this invention applies beam search with a beam size of 4. To evaluate translation quality, SacreBLEU is used to generate de-labeled BLEU scores for all models.

[0123] The baselines used are as follows:

[0124] 1) Transformer: The ultimate champion model in the German-English news task at WMT2019.

[0125] 2) V-kNN: The baseline model of kNN-MT, which generates a translation knowledge base based on the winning model of the WMT2019 German-English news task and integrates it into the decoding stage.

[0126] 3) A-kNN: An advanced variant of kNN-MT, which dynamically determines the use of k-nearest neighbor retrieval results and achieves more stable performance.

[0127] 4) R-kNN: The most advanced kNN-MT variant in retrieval optimization, which improves performance by dynamically calibrating k-nearest neighbor retrieval results and introducing more hyperparameters.

[0128] 5) Adapter: By adding an additional adapter module, the context representation and word embeddings are aligned, thereby optimizing the model's representation capabilities.

[0129] 6) INK: By introducing a few new parameters to smooth the representation space of k-nearest neighbor retrieval results, it surpasses the state-of-the-art kNN-MT system.

[0130] Table 2 shows the experimental results comparing the performance with the baseline model, with the best results indicated in bold.

[0131] Table 2. Baseline Model Comparison Experiment

[0132]

[0133] As shown in Table 2, our model achieved excellent BLEU scores across all domains, with an average BLEU score of 51.24, significantly outperforming existing baseline models and other enhancement methods, demonstrating its high adaptability in domain translation. In the low-resource captioning domain, despite containing only 30,000 data points, the model still achieved a BLEU score of 34.99, indicating its ability to effectively utilize limited data resources and generate more accurate translations under data-scarce conditions, showcasing good adaptability. In high-resource domains such as medicine and computer science, the BLEU scores reached 58.03 and 49.79 respectively, both exceeding other models such as INK. This achievement is attributed to the introduction of dynamic k-value adjustment, gating mechanisms, and distributed constraint loss, enabling the model to flexibly adjust the integration of k-nearest neighbor knowledge to adapt to the specific needs of each domain, thereby improving the accuracy of domain terminology translation. Although the R-kNN model performed best in the legal domain, our model remains the most robust across multiple domain tasks, especially in the low-resource domain, further validating the effectiveness of our improved methods.

[0134] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.< / sos> < / sos> < / sos> < / sos>

Claims

1. A domain machine translation method that dynamically incorporates k-nearest neighbor knowledge, characterized by: The method comprises: Step 1: Collect parallel data from the general domain and the target domain, pre-process the collected data, and then use the general domain data to train a basic teacher translation model; Step 2: Input the target domain data into the basic teacher translation model, extract the hidden layer representation of the model decoder, and build a translation knowledge base for the target domain; Step 3: Fine-tune the basic teacher translation model using the training corpus of the target domain to obtain the translation model of the target domain. During the fine-tuning process, relevant knowledge is dynamically extracted from the translation knowledge base, and the remaining corpus is gradually trained until the model fine-tuning is completed. Step 4: Input the target domain training corpus into the fine-tuned target domain translation model, update the translation knowledge base, and delete the knowledge base constructed in the previous round; Step 5. Repeat Step 3 to Step 4, and dynamically adjust the learning rate until the preset maximum number of iterations is reached, so as to obtain the final target domain machine translation model and use it for data translation.

2. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 1, characterized in that: The Step 1 includes: Step 1.1, collect parallel data of the general domain and the target domain, and preprocess the collected data; the preprocessing includes deduplication and filtering operations; according to the attributes of the target domain, the parallel data set of the target domain is sorted into D = (S, D1, D2, ..., D n )(1≤i≤n), where D i is the corpus of the i-th target domain; the general domain data is represented as S, and each domain data set contains parallel pairs of general domain data s and target domain t, denoted as (s, t); Step 1.2: Input the preprocessed general domain data into the basic neural machine translation model in batches for training until the convergence conditions are met, and construct the basic teacher translation model Θ base .

3. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 1, characterized in that: The step 2 comprises: Feed the training data of the target domain into the base teacher translation model Θ base , select the last hidden vector of the decoder as the key K, the next corresponding target word distribution as the value V, loop through the entire target domain data set, and build a translation knowledge base for the target domain, which is expressed as: Among them, (x,y)∈(X,Y) represents a parallel bilingual sentence, (X,Y) represents the parallel data of the entire target domain, and f(x,y <t ) represents the hidden vector representation of the translation context at time t, y <t It indicates that the model has predicted the target sentence at the prediction time t, yt is the true value of the next target word corresponding to the prediction, x in (x, y) represents the complete source language sentence, and y represents the complete target language sentence.

4. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 1, characterized in that: The step 3 includes: Step 3.1, freeze all parameters of the basic teacher translation model, and add a domain-specific adapter layer on the top layer of the encoder and decoder to learn domain-specific knowledge; after the forward propagation of the decoder, introduce the adaptive k-value module and the confidence module; Step 3.

2. Use the parallel data of the target domain to fine-tune the model after adding the adapter layer, that is, the target domain translation model; at each decoding time t, dynamically introduce the knowledge in the translation knowledge base to minimize the distance between the model-generated distribution, the final fusion distribution and the true distribution; at the same time, introduce a reinforcement learning strategy to improve the model performance by adjusting the relative weights of the loss terms.

5. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 4 is characterized in that: In the step Step3.1, the confidence module is used to calculate the confidence score of the target domain translation model for the current translation word at time t, so as to determine the weight in the fusion process; when the confidence score λ is higher than the preset threshold τ, the retrieval step of the translation knowledge base is skipped. At this time, a small perturbation value ε is added to the distribution generated by the target domain translation model to replace the final fusion distribution at this moment; when the confidence score λ is lower than the set threshold τ, the translation knowledge base is retrieved according to the normal process; the calculation formula of dynamic fusion and confidence is: λ=σ(w2(RELU((w1h t )+b1))+b2) Among them, P nmt represents the generative distribution of the model, P final represents the final fusion distribution, P kNN is the retrieval distribution of the translation memory, τ represents the set threshold, λ is the confidence score, and after the input sentence x passes through the NMT model, the final feature vector h of the decoder at time t is obtained t , w1 is the weight matrix of the first feed-forward layer of the confidence module, b1 is its bias vector, RELU is the activation function, w2 is the weight matrix of the second feed-forward layer of the confidence module, b2 is its bias vector, σ is the sigmoid activation function; the retrieval distribution P of the translation memory kNN It is expressed as: Where d(.) represents the Euclidean distance function, T is the temperature hyperparameter, The hidden vector representation of the context stored in the translation knowledge base, h t It is the context vector representation generated by the decoder at time t.

6. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 4, characterized in that: In the step 3.1, the adaptive k value module is used to dynamically adjust the k value retrieved from the translation knowledge base each time, including: First, the nearest neighbor vector of each sample is retrieved based on the input features and the set maximum value maxk; then, the vector is input into the subsequent module to dynamically adjust the size of the k value; for input features with high confidence, a small k value is selected to reduce noise interference; and for input features with low confidence, a large k value should be selected to enhance the diversity and robustness of the retrieval results; the calculation formula of the k value is as follows: Among them, λ t represents the confidence calculated at time t, represents the average confidence of the current batch, h t is the hidden vector of the decoder at time t obtained after the input sentence x passes through the target domain translation model, q1 is the weight matrix of the first feedforward layer in the adaptive k value module, z1 is its bias vector, RELU is the activation function, q2 is the weight matrix of the second feedforward layer in the adaptive k value module, z2 is its bias vector, σ is the sigmoid activation function, and finally clamp is used to scale the k value to between 1-maxk.

7. The domain machine translation method dynamically incorporating k-nearest neighbor knowledge according to claim 4, characterized in that: The Step 3.2 includes: (1) Introduce a reinforcement learning strategy and adjust the relative weight of the loss term through the intermediate reward r. The calculation formula of the intermediate reward r is as follows: r=where(P final (yt)=P gold (y t )) Among them, r represents the intermediate reward, P final (y t ) represents the final fusion distribution formed by the fusion of the retrieval distribution of k nearest neighbor knowledge and the prediction distribution of the target domain translation model at the tth moment, P gold (y t ) is the distribution of the next real target word, where is used to evaluate the consistency of the maximum candidate word in the two distributions and use it as the final output. If the candidate words are the same, the output value is 1, otherwise, the output value is 0; (2) KL divergence is used as the loss function to optimize the combination accuracy of the target domain translation model and k nearest neighbor knowledge, and to minimize the loss L between the final fusion distribution and the distribution of the true target word. fg , L fg The calculation formula is as follows: Among them, r g represents the intermediate reward of the target domain translation model when predicting at time t, x represents the source sentence, y represents the corresponding target sentence, and D represents the set of training parallel sentence pairs. represents the next real word distribution at time t, represents the final distribution of the fusion of the retrieval distribution at time t and the predicted distribution of the target domain translation model, KL[.] is the KL divergence; when the intermediate reward r is 1, it means that the fused distribution is correct, and the rewards are set to 1 / k respectively; when the intermediate reward is 0, it means that the fused distribution is wrong at this time, and the penalty term is set at this time, and it is set to 1 / j; (3) By minimizing the final distribution P final and the model distribution P nmt The KL divergence between them is used to optimize the distribution of the target domain translation model, L fn The calculation formula is as follows: Among them, r n represents the intermediate reward when the model predicts at time t, represents the prediction distribution of the target domain translation model corresponding to time t, by pulling in the prediction distribution P of the target domain translation model nmt And the final fusion distribution P final The KL divergence between them makes it gradually tend to the fusion distribution P final ; Similarly, set the reward term to 1 / n and the penalty term to 1 / m; (4) The translation loss of the target domain translation model is as follows: Among them, P(y i |x i ; θ) is the source sentence x i Translate to target sentence y i The probability of , n is the number of sentences, θ is the target domain translation model parameter; (5) Finally, the above loss items are weighted to obtain the final optimization target of the target domain translation model, which is expressed as follows: L=L nmt +r n L fn +r g L fg Among them, L is the final training target, L nmt is the translation loss, r n and r g It is a hyperparameter used to balance the weights of different loss functions. fn is the loss that minimizes the difference between the final fusion distribution and the predicted distribution of the target domain translation model, L fg is the loss that minimizes the difference between the final fusion distribution and the distribution of the true target words.

8. A domain machine translation system that dynamically incorporates k-nearest neighbor knowledge, characterized in that: The system comprises: a module for executing the domain machine translation method dynamically incorporating k-nearest neighbor knowledge as claimed in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the domain machine translation method with dynamic incorporation of k-nearest neighbor knowledge as claimed in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the domain machine translation method with dynamic incorporation of k-nearest neighbor knowledge as claimed in any one of claims 1 to 7 is implemented.

Images