A translation performance improvement method based on offline reinforcement learning

Abstract

The application discloses a translation performance improvement method based on offline reinforcement learning, a neural machine translation model is trained as a maximum likelihood estimation model using a maximum likelihood estimation paradigm; the maximum likelihood estimation model is used as a real sampling strategy, the offline reinforcement learning model is used as an updated strategy, and an importance sampling weight is calculated to update the offline reinforcement learning model in the training process; trajectory data is sampled through the maximum likelihood estimation model, and a reward function is obtained by summing the probability distribution of the target language output for the input source language; the importance sampling weight and the expected return are used, the decision output of the current training strategy is combined, a policy gradient method is used to train the offline reinforcement learning model, and the translation performance of the offline reinforcement learning model is improved. The application solves the operation that the model strategy needs to continuously sample and interact with the environment under the traditional reinforcement learning paradigm, and avoids the difficulty in training caused by the fact that the strategy falls into a zero reward area in the sampling and interaction process.

Description

Technical Field

[0001] This invention relates to a method for improving the performance of machine translation tasks in the field of natural language processing, specifically a method for improving translation performance based on offline reinforcement learning. Background Technology

[0002] Machine translation (MT) is a branch of computational linguistics, an experimental science that uses computers to automatically translate between various natural languages. Specifically, it's the process of using computers to convert one natural language (source language) into another (target language). Currently, the rapid development of deep learning technologies, represented by neural networks, has led to significant progress in tasks such as machine translation, and neural machine translation has become the mainstream method in the field. Common neural machine translation models employ an end-to-end encoder-decoder framework, transforming the machine translation task into a sequence-to-sequence conversion process. The encoder encodes the input source language sequence into a source language context vector, while the decoder receives the source language context vector from the encoder and converts it into a target language sequence.

[0003] Reinforcement learning (RL) is a paradigm in machine learning. Through policy updates and optimizations, an agent can maximize its reward for a set objective by interacting with its environment; it is an iterative process of continuous exploration and utilization. Unlike supervised learning, traditional reinforcement learning does not require pre-provided training data. Instead, it obtains reinforcement signals as learning information through the interaction between the agent's policy and the environment, thereby updating and optimizing the model's parameters. Reinforcement learning has been widely applied in many fields of artificial intelligence. However, in some settings, this online interactive data collection between the agent and the environment can be impractical, such as in autonomous driving and healthcare, where the cost of collecting interactive data is often very high. Offline reinforcement learning, on the other hand, allows the agent to utilize offline data and complete the update and optimization process without any interaction with the environment, making it a very promising direction.

[0004] Most existing neural machine translation decoding methods are based on autoregressive and maximum likelihood estimation paradigms. In this paradigm, the model's output translation is expected to be as similar as possible to the reference translation given the training data. However, this paradigm can lead to inconsistencies between training and inference objectives, as well as exposure bias caused by the teacher-forced autoregressive paradigm, resulting in the cumulative propagation of errors. The solutions to these problems align well with the optimization objectives of reinforcement learning, and these issues can be addressed using reinforcement learning paradigms. However, applying reinforcement learning to machine translation tasks causes the exploration space to expand exponentially. The agent's policy may get stuck in the zero-reward region and fail to effectively sample and interact with the environment, limiting the benefits of exploration. Therefore, adopting an offline reinforcement learning paradigm, training the model using only offline data in an offline setting, is undoubtedly a superior solution. Summary of the Invention

[0005] To address the problem that traditional reinforcement learning models require continuous sampling and interaction with the environment during training, which can easily lead to poor model performance due to getting stuck in zero-reward regions, this invention provides an offline reinforcement learning method to improve translation performance. This method uses offline reinforcement learning settings and maximum likelihood estimation to solve the reward function, avoiding interaction while making the translation results closer to human translation, thereby improving the performance of the translation system.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: This invention provides a method for improving translation performance based on offline reinforcement learning, comprising the following steps: 1) The neural machine translation model was trained using the maximum likelihood estimation paradigm as the maximum likelihood estimation model for the generation of subsequent trajectory data; 2) Using the maximum likelihood estimation model as the real sampling strategy, the offline reinforcement learning model is initialized using the maximum likelihood estimation model, and the offline reinforcement learning model is used as the strategy to be updated. The importance sampling weights are calculated and used to update the offline reinforcement learning model during the training process. 3) By sampling trajectory data through the maximum likelihood estimation model, the sum of the probability distributions of the target language output by the maximum likelihood estimation model is used as the reward function to ensure that the generated target language has a high overall probability under the maximum likelihood estimation model. 4) Using the calculated importance sampling weights and the expected reward obtained through the reward function, combined with the decision output of the current policy to be trained, the offline reinforcement learning model is trained using the policy gradient method to improve the translation performance of the offline reinforcement learning model.

[0007] In step 1), the neural machine translation model is trained using the maximum likelihood estimation paradigm as the maximum likelihood estimation model. This provides an initial strategy for the offline reinforcement learning model trained under the reinforcement learning paradigm, so that the reward value required for backpropagation update of the offline reinforcement learning model approximates human preferences and habits, thereby optimizing the perceptual quality of the generated sequence.

[0008] Step 2) The specific process is as follows: 201) Use an approximate probability distribution instead of the importance sampling product along the time step for each sampling action to avoid its sensitivity to optimization hyperparameters and reduce training convergence time; 202) Further approximate the probability distribution of the behavioral strategy to obtain the final importance sampling weights, as shown in the following formula: ; in, For time steps, Given the source language as input and the target language as output in the current decoding step, For the target words decoded at the current time step, Strategy to be updated; Step 3) The specific process is as follows: 301) Input source language sentences from offline data into a neural machine translation model trained using the maximum likelihood estimation paradigm. In this process, the probability distribution of the target statement output is used as the sampled trajectory; 302) Based on the defined reward function, calculate the sum of word probabilities of the generated sentence from the sampled trajectory as its expected reward. The definition of the reward function and the formula for the calculated expected return are as follows: ; ; in, For the reward function, The sampled trajectory state, Actions taken based on state decisions To determine the action probability distribution of the model trained using the maximum likelihood estimation paradigm, This represents the action probability distribution of the model trained using the maximum likelihood estimation paradigm at the current time step. For time steps, This represents the total number of time steps.

[0009] In step 4), the policy gradient method is used to train the model, training the gradient estimate of the objective function. as follows: ; in, For training data, The trajectory sampled from the training data, For time steps, The model policy trained using the offline reinforcement learning paradigm For expected returns.

[0010] The present invention has the following beneficial effects and advantages: 1. This invention solves the problem of the need for the model policy to continuously sample and interact with the environment in the traditional reinforcement learning paradigm by applying an offline reinforcement learning paradigm, thus avoiding the difficulty of training caused by the policy getting stuck in the zero-reward region during the sampling interaction process.

[0011] 2. This invention selects the reward obtained from the maximum likelihood estimation as an approximation for human translation, ensuring that the generated translated sentences have a high probability of success in human translation. This optimizes the perceived quality of the generated translation sequence and successfully improves the performance of the translation system. Attached Figure Description

[0012] Figure 1 This is a structural diagram of the translation performance improvement method based on offline reinforcement learning of the present invention; Figure 2 This is a flowchart of the model pre-training process involved in the present invention. Detailed Implementation

[0013] The present invention will now be further described with reference to the accompanying drawings.

[0014] This invention provides a method for improving translation performance based on offline reinforcement learning, the specific method structure of which is as follows: Figure 1 As shown, it includes the following steps: 1) The neural machine translation model is trained using the traditional Maximum Likelihood Estimate (MLE) paradigm as the maximum likelihood estimation model for the generation of subsequent trajectory data; 2) Using the maximum likelihood estimation model as the real sampling strategy, the offline reinforcement learning model is initialized using the maximum likelihood estimation model, and the offline reinforcement learning model is used as the strategy to be updated. The importance sampling weights are calculated and used to update the offline reinforcement learning model during the training process. 3) By sampling trajectory data through the maximum likelihood estimation model, the sum of the probability distributions of the target language output by the maximum likelihood estimation model is used as the reward function to ensure that the generated target language has a high overall probability under the maximum likelihood estimation model. 4) Using the calculated importance sampling weights and the expected reward obtained through the reward function, combined with the decision output of the current policy to be trained, the offline reinforcement learning model is trained using the policy gradient method to improve the translation performance of the offline reinforcement learning model.

[0015] In step 1), the neural machine translation model is trained using the maximum likelihood estimation paradigm as the maximum likelihood estimation model. This provides an initial strategy for the offline reinforcement learning model trained under the reinforcement learning paradigm, so that the reward value required for backpropagation update of the offline reinforcement learning model approximates human preferences and habits, thereby optimizing the perceptual quality of the generated sequence.

[0016] Maximum likelihood estimation (MLA) baseline models provide an initial policy for models trained in reinforcement learning paradigms. Since reward functions in text generation tasks such as machine translation are typically defined as sequence-level operations, the policy may get stuck in the zero-reward range during training. In this case, the trained model system struggles to receive effective learning signals and produces many meaningless outputs. Simultaneously, as a policy for subsequent sampling trajectories, the baseline model also allows the reward values ​​required for backpropagation updates to approximate human preferences, thereby optimizing the perceptual quality of the generated sequences.

[0017] In step 2), the maximum likelihood estimation model is used as the real sampling policy, and the offline reinforcement learning model is used as the policy to be updated. Importance sampling weights are then calculated. In many reinforcement learning algorithms, such as the policy gradient algorithm, maximizing the sum of the reward function requires trajectories derived from the probability distribution of the current policy to be updated. If this condition is not met, importance sampling must be used; otherwise, the calculated sum of rewards will not conform to the definition. In an offline setting that avoids interaction with the environment, it is impossible to sample the trajectory of the policy to be updated to estimate its expected reward. A different behavioral policy must be used to complete trajectory sampling. Therefore, the importance sampling weights are calculated using the two probability distributions mentioned above for model updates during training. The specific steps for calculating the importance sampling weights are as follows: 201) Use an approximate probability distribution instead of the importance sampling product of each sampling action along the time step to avoid its sensitivity to optimization hyperparameters and reduce training convergence time; when the distribution of the policy to be updated and the behavior policy are similar, using this biased estimate can effectively reduce the variance of the distribution. 202) Since it is difficult to abstract the behavior policies corresponding to offline datasets, the probability distribution of the behavior policies is further approximated. A constant empirical distribution is used to estimate the behavior policies so that it can be ignored during training, resulting in the final importance sampling weights. The constant empirical distribution and importance sampling weight formulas are as follows: ; ; Where N is the number of trajectories in the training data. The offline trajectory was sampled. For training data, For the calculated empirical distribution of constants, For importance sampling weights, This is a strategy pending update.

[0018] In step 3), the trajectory data is sampled using the maximum likelihood estimation model. The sum of the probability distributions of the target language output by the maximum likelihood estimation model for the input source language is used as the reward function. The specific steps for sampling and generating the reward are as follows: 301) Input the source language sentences from the offline data into a neural machine translation model trained using the maximum likelihood estimation paradigm, and use the output target sentence probability distribution as the sampled trajectory; 302) Based on the defined reward function, the sum of word probabilities generated from the sampled trajectory is calculated as the expected reward. In the offline setting, the distribution provided by maximum likelihood estimation can well approximate the probability of human translation. The definition of the reward function and the formula for the calculated expected reward are as follows: ; ; in, For the reward function, The sampled trajectory state, Actions taken based on state decisions To determine the action probability distribution of the model trained using the maximum likelihood estimation paradigm, This represents the action probability distribution of the model trained using the maximum likelihood estimation paradigm at the current time step. For time steps, This represents the total number of time steps.

[0019] In step 4), the calculated importance sampling weights and expected rewards are used, combined with the decision output of the current policy to be trained, to train the model using the policy gradient method. The overall flowchart of model training in this method is shown below. Figure 2 As shown, the gradient estimation of the training objective function is as follows: ; in, For training data, The trajectory sampled from the training data, For time steps, The model policy trained using the offline reinforcement learning paradigm For expected returns.

[0020] It may also include step 5), which evaluates the translation performance of the model after it has been trained for a certain number of rounds.

[0021] To verify the effectiveness of the method, the proposed offline reinforcement learning-based translation performance improvement method was tested on a machine translation task. Specifically, experiments were conducted on the IWSLT2014 German-English dataset, which contains approximately 160,000 bilingual sentence pairs. The baseline experiment used a standard Transformer model, comprising 6 encoder layers and 6 decoder layers, with a hidden layer dimension of 512. The experimental results are shown in the table below, where BLEU represents the quality of the machine translation system on the test set, with higher values ​​indicating better performance. Baseline system 34.81 Method of the present invention 35.63

[0022] Experiments show that the translation performance improvement method based on offline reinforcement learning proposed in this invention can improve the performance of machine translation systems on the basis of baseline systems when the offline reinforcement learning paradigm only uses offline data.

[0023] This invention, based on machine translation tasks, employs an offline reinforcement learning paradigm. By avoiding interaction with the environment through offline data sampling, it prevents the policy from getting stuck in the zero-reward region during the sampling process, successfully training a neural machine translation system. At the same time, it selects the maximum likelihood estimation distribution as the reward, optimizing the perceptual quality of the generated translation sequence and improving the performance of the translation system.

Claims

1. A method for improving translation performance based on offline reinforcement learning, characterized in that... Includes the following steps: 1) The neural machine translation model was trained using the maximum likelihood estimation paradigm as the maximum likelihood estimation model for the generation of subsequent trajectory data; 2) Using the maximum likelihood estimation model as the real sampling strategy, the offline reinforcement learning model is initialized using the maximum likelihood estimation model, and the offline reinforcement learning model is used as the strategy to be updated. The importance sampling weights are calculated and used to update the offline reinforcement learning model during the training process. 3) By sampling trajectory data through the maximum likelihood estimation model, the sum of the probability distributions of the target language output by the maximum likelihood estimation model is used as the reward function to ensure that the generated target language has a high overall probability under the maximum likelihood estimation model. 4) Using the calculated importance sampling weights and the expected reward obtained through the reward function, combined with the decision output of the current policy to be trained, the offline reinforcement learning model is trained using the policy gradient method to improve the translation performance of the offline reinforcement learning model; Step 3) The specific process is as follows: 301) Input source language sentences from offline data into a neural machine translation model trained using the maximum likelihood estimation paradigm. In this process, the probability distribution of the target statement output is used as the sampled trajectory; 302) Based on the defined reward function, calculate the sum of word probabilities of the generated sentence from the sampled trajectory as its expected reward. The definition of the reward function and the formula for the calculated expected return are as follows: ； ； in, For the reward function, The sampled trajectory state. Actions taken based on state decisions To determine the action probability distribution of the model trained using the maximum likelihood estimation paradigm, This represents the action probability distribution of the model trained using the maximum likelihood estimation paradigm at the current time step. For time steps, This represents the total number of time steps.

2. The translation performance improvement method based on offline reinforcement learning according to claim 1, characterized in that: In step 1), the neural machine translation model is trained using the maximum likelihood estimation paradigm as the maximum likelihood estimation model. This provides an initial strategy for the offline reinforcement learning model trained under the reinforcement learning paradigm, so that the reward value required for backpropagation update of the offline reinforcement learning model approximates human preferences and habits, thereby optimizing the perceptual quality of the generated sequence.

3. The translation performance improvement method based on offline reinforcement learning according to claim 1, characterized in that: Step 2) The specific process is as follows: 201) Use an approximate probability distribution instead of the importance sampling product along the time step for each sampling action to avoid its sensitivity to optimization hyperparameters and reduce training convergence time; 202) Further approximate the probability distribution of the behavioral strategy to obtain the final importance sampling weights, as shown in the following formula: ； in, For time steps, Given the source language as input and the target language as output in the current decoding step, For the target words decoded at the current time step, This is a strategy pending update.

4. The translation performance improvement method based on offline reinforcement learning according to claim 1, characterized in that: In step 4), the policy gradient method is used to train the model, training the gradient estimate of the objective function. as follows: ； in, For training data, The trajectory sampled from the training data, For time steps, The model policy trained using the offline reinforcement learning paradigm For expected returns.

Images