Machine translation method and device based on reinforcement learning, equipment, storage medium and program product

Abstract

The invention provides a machine translation method and device based on reinforcement learning, equipment, a storage medium and a program product, and relates to the technical field of translations, the method comprises the following steps: obtaining a large language model, the large language model comprising a translation task layer and a post-editing task layer; processing the first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value according to the candidate translation text set; processing second tuple data through the post-editing task layer to obtain a post-editing text set, and determining a second advantage value according to the post-editing text set; and based on the first advantage value and the second advantage value, performing reinforcement learning training on the large language model to obtain a target large language model, and generating a machine translation result based on the target large language model. The machine translation quality can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of translation technology, and in particular to a machine translation method, apparatus, device, storage medium, and program product based on reinforcement learning. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), reinforcement learning-based post-training techniques have become a key means to improve the quality of machine translation. Currently, mainstream reinforcement learning methods (such as the GRPO training paradigm based on automatic evaluation metrics) mainly focus on optimizing a single translation task, namely, the direct conversion from the source language to the target language. However, this optimization approach, which focuses solely on a single translation task, can easily limit the model to a superficial mapping between the source and target languages, making it difficult for the model to learn more idiomatic expressions, thus affecting the final machine translation quality. Summary of the Invention

[0003] This invention provides a hybrid training technique for machine translation and post-editing based on reinforcement learning (invention title) to address the issue that using traditional reinforcement learning training models in the prior art can affect the machine translation quality of large language models.

[0004] This invention provides a machine translation method based on reinforcement learning, comprising the following steps: Obtain a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; The translation task layer processes the first tuple data to obtain a candidate translation text set, and determines a first advantage value based on the candidate translation text set. The second tuple data is processed by the post-editing task layer to obtain a post-edited text set, and a second advantage value is determined based on the post-edited text set. Based on the first advantage value and the second advantage value, the large language model is trained by reinforcement learning to obtain a target large language model, and machine translation results are generated based on the target large language model.

[0005] According to a reinforcement learning-based machine translation method provided by the present invention, the first tuple data includes a first source language text and a first reference translation; wherein, the step of processing the first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set, includes: The first source language text is processed by the translation task layer to obtain a candidate translation text set; Based on the first reference translation and the candidate translation text set, a first advantage value is determined by a quality estimation model.

[0006] According to a reinforcement learning-based machine translation method provided by the present invention, the step of determining a first advantage value based on a quality estimation model using a first reference translation and the candidate translation text set includes: Based on the first reference translation, the quality estimation model is used to score each candidate translation text in the candidate translation text set to obtain a first score result set; Determine the first mean and the first standard deviation based on the first set of scores; Based on the first scoring result set, the first average value, and the first standard deviation, a first advantage value is determined for each candidate translation text in the candidate translation text set.

[0007] According to a reinforcement learning-based machine translation method provided by the present invention, the second tuple data includes a second source language text, a second language text to be edited, and a second reference translation; wherein, the step of processing the second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the post-edited text set, includes: Based on the second reference translation, the second language text to be edited is processed through the post-editing task layer to obtain a post-edited text set; The second language text to be edited is scored using a quality estimation model to obtain a basic score result; Based on the baseline score, the second source language text, the second reference translation, and the post-edited text set, a second advantage value is determined using a quality estimation model.

[0008] According to a reinforcement learning-based machine translation method provided by the present invention, the step of determining a second dominance value based on the base scoring result, the second source language text, the second reference translation, and the post-edited text set through a quality estimation model includes: Based on the second reference translation, the quality estimation model is used to score each post-edited text in the post-edited text set to obtain a second scoring result set; A second mean and a second standard deviation are determined based on the basic scoring results and the second scoring result set; Based on the baseline scoring results, the second scoring result set, the second average value, and the second standard deviation, a second advantage value is determined for each post-edited text in the post-edited text set.

[0009] According to a reinforcement learning-based machine translation method provided by the present invention, the step of training a large language model using reinforcement learning based on a first advantage value and a second advantage value to obtain a target large language model includes: Based on the first advantage value or the second advantage value, the objective function is optimized by maximizing the strategy, and the model parameters of the large language model are iteratively updated until the model converges, thus obtaining the target large language model.

[0010] The present invention also provides a machine translation device based on reinforcement learning, comprising the following modules: An acquisition module is used to acquire a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; The determination module is used to process the first tuple data through the translation task layer to obtain a candidate translation text set, and to determine a first advantage value based on the candidate translation text set; The determining module is used to process the second tuple data through the post-editing task layer to obtain a post-edited text set, and to determine a second advantage value based on the post-edited text set; The training module is used to perform reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, so as to generate machine translation results based on the target large language model.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the reinforcement learning-based machine translation method as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the reinforcement learning-based machine translation method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the reinforcement learning-based machine translation method as described above.

[0014] This invention provides a reinforcement learning-based machine translation method, apparatus, device, storage medium, and program product. It acquires a large language model, which includes a translation task layer and a post-editing task layer. The translation task layer processes first tuple data to obtain a candidate translation text set, and determines a first dominance value based on the candidate translation text set. The post-editing task layer processes second tuple data to obtain a post-edited text set, and determines a second dominance value based on the post-edited text set. Based on the first and second dominance values, the large language model is trained using reinforcement learning to obtain a target large language model, which then generates machine translation results. This invention solves the technical problem that traditional reinforcement learning training affects the machine translation quality of large language models. Compared to existing technologies, this invention constructs a translation task layer and a post-editing task layer within the large language model, and generates first and second dominance values ​​respectively within the same reinforcement learning training framework based on heterogeneous tuple data to collaboratively optimize model parameters. This method enables the model not only to learn basic cross-linguistic semantic transformations, but also to learn to refine the target language text through explicit post-editing tasks, thereby alleviating the problem of translation output not conforming to the expression habits of the target language. This design effectively improves the machine translation quality and robustness of large language models. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts of the reinforcement learning-based machine translation method provided by the present invention.

[0017] Figure 2 This is a model training framework diagram of the reinforcement learning-based machine translation method provided by the present invention.

[0018] Figure 3 This is the second flowchart illustrating the reinforcement learning-based machine translation method provided by this invention.

[0019] Figure 4 This is a schematic diagram of the structure of the machine translation device based on reinforcement learning provided by the present invention.

[0020] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined Figure 1 and Figure 3 The present invention describes a reinforcement learning-based machine translation method applicable to any reinforcement learning-based machine translation. The execution subject of this method can be an electronic device or a reinforcement learning-based machine translation device installed in the electronic device. The reinforcement learning-based machine translation device can be implemented by software, hardware, or a combination of both.

[0023] Figure 1 This is one of the flowcharts illustrating the reinforcement learning-based machine translation method provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain a large language model, wherein the large language model includes a translation task layer and a post-editing task layer.

[0024] It should be noted that the translation task layer is used to realize the conversion from source language text to target language text, that is, to establish a mapping relationship between source language text and target language text. This process primarily focuses on cross-linguistic semantic transfer, such as word-level or character-level translation. However, relying solely on this task is insufficient to further improve the high-level semantic quality of the target language.

[0025] It's important to note that the post-editing task layer is used to transform the initial translation result into an optimized translation. Specifically, given the source language text `src`, it establishes a mapping relationship (g: `tgt→tgt′`) from the lower-quality target language text `tgt` to the higher-quality target translation text `tgt′`. This task focuses on improving the high-level expressive power of the translation in the target language, including but not limited to word order adjustment, semantic consistency enhancement, and naturalness optimization. Its core lies in stimulating and utilizing the powerful target language generation and correction capabilities of the large language model itself, autonomously optimizing the initial translation through the post-editing process to generate a more fluent, natural, and semantically coherent translation. This mechanism effectively alleviates problems such as "output translation not conforming to target language expression habits" caused by relying solely on the initial translation.

[0026] Step 102: Process the first tuple data through the translation task layer to obtain a candidate translation text set, and determine the first advantage value based on the candidate translation text set.

[0027] It should be noted that the first tuple data can be used as a data sample for training the large language model. It corresponds to the cross-language translation task (MT task) and takes the form of a pair (src, tgt) containing source language text and a reference translation. The reference translation tgt does not participate in the forward generation but is only used for subsequent quality assessment and reward calculation. When training the large language model using the GRPO algorithm, the translation task layer can sample the first tuple data multiple times to generate a candidate translation text set containing N candidate translations. Subsequently, a quality estimation (QE) model (such as COMETKIWI) is used to automatically score each first candidate translation in the candidate translation text set, resulting in a set of scores. Then, based on this set of scoring results, the first advantage value is calculated. This first advantage value will serve as a key signal in reinforcement learning for subsequent policy gradient updates and model parameter optimization.

[0028] Step 103: Process the second tuple data through the post-editing task layer to obtain the post-edited text set, and determine the second advantage value based on the post-edited text set; It should be noted that the second tuple data can be used as data samples for training large language models. It corresponds to the monolingual post-editing task (PE task), and its form is a tuple (src, lst, tgt) containing the source language text, the target language text to be edited, and a reference translation. Here, lst is the initial translation to be optimized or the target language text with low quality (i.e., the target language text to be edited). The post-editing task layer will generate multiple post-editing optimization results based on the second tuple data through multiple sampling, forming a post-editing text set. Subsequently, the quality estimation (QE) model is used to score each post-editing candidate in both the target language text to be edited and the post-editing text set, obtaining the basic score result b. lst With a set of post-edited text scores Finally, the quality improvement brought about by post-editing (b) i -b lst The second advantage value is calculated as a reward signal.

[0029] Understandably, the first or second tuple data together constitute a hybrid dataset used to train a large language model. By proportionally or alternately mixing the first tuple data from the translation task with the second tuple data from the post-editing task in the hybrid dataset, the model can simultaneously receive and optimize learning signals from both types of tasks within a unified reinforcement learning framework. This hybrid mechanism not only supports the collaborative training and gradient update of translation and post-editing capabilities but also avoids catastrophic forgetting or capability fragmentation that might result from isolated task training, thus achieving end-to-end integrated policy learning from "basic translation" to "expression optimization."

[0030] Step 104: Based on the first advantage value and the second advantage value, perform reinforcement learning training on the large language model to obtain a target large language model, and generate machine translation results based on the target large language model.

[0031] In practical implementation, the first advantage value (from the translation task) and the second advantage value (from the post-editing task) can be used as key signals for policy gradient updates, enabling joint optimization of the large language model within a unified GRPO algorithm framework. Specifically, the advantage value, after standardization, will be used as a weighting factor in the loss function calculation, guiding the model parameters to update in a direction that yields higher rewards (i.e., better translation quality). During this process, the policy networks of the translation task layer and the post-editing task layer are trained simultaneously. The model not only learns how to perform cross-language conversion but also how to autonomously refine and optimize the initial translation.

[0032] Understandably, the target large language model obtained after training has internally enhanced its cross-language translation and monolingual post-editing capabilities. During the inference phase, the model can autonomously implement a two-stage optimization process: first, performing preliminary semantic transformation, and then refining and correcting the translation in terms of expression, thus achieving end-to-end optimization of the translation task from "semantic transmission" to "linguistic expression." This design not only effectively alleviates the catastrophic forgetting problem that may result from independent training in stages, but also breaks through the performance ceiling of a single-task training framework through the synergistic effect of dual tasks, ultimately significantly improving the model's overall translation quality and robustness in real-world scenarios, thereby generating higher-quality machine translation results.

[0033] In specific implementations, such as Figure 2As shown, the entire process begins with a mixed dataset containing two types of inputs: the first tuple data for the translation task and the second tuple data for the post-editing task. The model then executes policies for each type of input, generating diverse translation and post-editing candidates. Next, based on a unified quality assessment logic, the first advantage value for the translation task and the second advantage value for the post-editing task are calculated as reward signals for reinforcement learning. Finally, the two advantage values ​​are input into the policy optimization objective function. By calculating the loss and updating the model parameters, the dual-task collaborative training of translation generation and post-editing optimization is achieved, driving the model to build a complete "generation-optimization" capability loop.

[0034] This invention acquires a large language model, which includes a translation task layer and a post-editing task layer. The translation task layer processes first tuple data to obtain a candidate translation text set, and determines a first advantage value based on the candidate translation text set. The post-editing task layer processes second tuple data to obtain a post-edited text set, and determines a second advantage value based on the post-edited text set. Based on the first and second advantage values, the large language model is trained using reinforcement learning to obtain a target large language model, which then generates machine translation results. This invention solves the technical problem that traditional reinforcement learning training affects the machine translation quality of large language models. Compared to existing technologies, this invention constructs a translation task layer and a post-editing task layer within the large language model, and generates first and second advantage values ​​respectively within the same reinforcement learning training framework based on heterogeneous tuple data to collaboratively optimize model parameters. This method enables the model to not only learn basic cross-language semantic transformation but also learn to refine the target language text through explicit post-editing tasks, thereby alleviating the problem of translation output not conforming to the expression habits of the target language. This design effectively improves the machine translation quality and robustness of large language models.

[0035] Based on any of the above embodiments, the first tuple data includes a first source language text and a first reference translation; wherein, the step of processing the first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set, includes: The first source language text is processed by the translation task layer to obtain a candidate translation text set; Based on the first reference translation and the candidate translation text set, a first advantage value is determined by a quality estimation model.

[0036] It should be noted that the first source language text refers to the original text data to be translated; the first reference translation refers to a high-quality target language translation that corresponds to the first source language text and has been manually annotated or approved.

[0037] It should be noted that in reinforcement learning training, the translation task layer samples the first source language text multiple times to generate N candidate translation texts (i.e., a candidate translation text set). Subsequently, a quality estimation model (such as COMETKIWI) is used to compare and evaluate each candidate translation text in the candidate translation text set with the first reference translation, resulting in a set of corresponding score results. Then, based on this set of scores, the first advantage value is calculated. This advantage value reflects the relative performance of each candidate translation text with respect to the average performance of the current strategy, and serves as a key signal for policy gradient updates, driving the model to optimize towards generating higher-scoring translation texts.

[0038] Understandably, the process begins by sampling text from the same source language to generate multiple candidate translations. Then, a quality assessment model calculates a quality difference score between each candidate translation and a reference translation. Based on this score set, an independent advantage value reflecting the relative merit within the group is calculated for each candidate translation. Finally, this set of advantage values ​​is used to calculate the policy gradient of the large language model. This allows the model update to no longer rely on the absolute score of a single output, but rather to optimize based on the distribution of all possibilities under the current policy, thus significantly improving the stability and efficiency of training.

[0039] The reinforcement learning-based machine translation method provided in this invention generates a candidate translation text set based on the first tuple data in the translation task layer, and calculates the relative advantage value between the first reference translation and each candidate translation text through a quality estimation model, thus constructing a reinforcement learning optimization signal with internal comparison. This technical solution transforms the traditional "imitative" optimization that relies on a single reference answer into a strategy learning approach that guides the model to explore and strengthen better expressions among diverse candidate translations. This not only enables the model to more comprehensively evaluate its own strategy and learn deep linguistic rules beyond surface matching, but also provides a unified strategy evaluation benchmark for subsequent collaborative optimization with the post-editing task layer.

[0040] Based on any of the above embodiments, determining the first advantage value using a quality estimation model based on the first reference translation and the candidate translation text set includes: Based on the first reference translation, the quality estimation model is used to score each candidate translation text in the candidate translation text set to obtain a first score result set; Determine the first mean and the first standard deviation based on the first set of scores; Based on the first scoring result set, the first average value, and the first standard deviation, a first advantage value is determined for each candidate translation text in the candidate translation text set.

[0041] It should be noted that each candidate translation text in the candidate translation text set is independently scored using a quality estimation model, resulting in the first scoring result set. This set directly reflects the absolute performance of diverse outputs under the current strategy on specific quality dimensions (such as semantic fidelity and fluency). Subsequently, the first mean and first standard deviation of the first scoring result set are calculated. The mean represents the average quality benchmark of the current strategy when processing the first source language text, while the standard deviation quantifies the dispersion of the scoring results. Finally, a first advantage value is calculated for each candidate translated text based on the first scoring result set, the first mean, and the first standard deviation.

[0042] In practice, the formula for calculating the first advantage value is as follows: In the formula, This represents the average value of the first set of ratings. a represents the standard deviation of the first set of rating results. i This represents the score result of the i-th candidate translation text in the candidate translation text set.

[0043] Understandably, by subtracting the average, the optimization objective is shifted from pursuing an absolutely high score to "relatively selecting the best" among the current strategy outputs, effectively offsetting the systematic bias of the evaluation model and allowing the model to focus on self-improvement. Secondly, the scale normalization achieved by "dividing by the standard deviation" ensures the balance of gradient update amplitudes among different training samples, effectively preventing training oscillations and divergence, and guaranteeing efficient convergence and numerical stability of the large-scale parameter optimization process.

[0044] The machine translation method based on reinforcement learning provided in this invention scores the candidate translation text set sample by sample using a quality estimation model, and then normalizes the scores by combining the average and standard deviation of the scores. The resulting first advantage value not only retains the absolute quality information of a single candidate translation, but also eliminates the difference in scoring scale between different candidate sets through relative deviation calibration. This makes the advantage value uniformly comparable across different training batches and different input samples, avoiding the training instability problem caused by scale fluctuations when using a single absolute score as a reward signal.

[0045] Based on any of the above embodiments, the second tuple data includes a second source language text, a second language text to be edited, and a second reference translation; wherein, the step of processing the second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the set, includes: Based on the second reference translation, the second language text to be edited is processed through the post-editing task layer to obtain a post-edited text set; The second language text to be edited is scored using a quality estimation model to obtain a basic score result; Based on the baseline score, the second source language text, the second reference translation, and the post-edited text set, a second advantage value is determined using a quality estimation model.

[0046] It should be noted that the second source language text refers to the original text data to be translated; the second editable language text refers to the initial translation result generated based on the second source language text, or the target language text whose quality needs to be optimized; and the second translation refers to the high-quality target language translation that corresponds to the second source language text and has been manually annotated or approved.

[0047] It should be noted that in the reinforcement learning training, the post-editing task layer samples the second source language text multiple times to generate N post-editing optimization results (i.e., the post-edited text set). Subsequently, a quality estimation model (such as COMETKIWI) is used to compare and evaluate each post-edited text in the post-edited text set with the second source language text and the second reference translation, resulting in a set of corresponding post-editing scores. Simultaneously, the second language text to be edited is scored to obtain its basic score result b. lst Based on these post-editing scores and their differences from the baseline scores, a second advantage value is calculated, reflecting the relative merits of each post-edited text optimization. This advantage value serves as a key signal for updating the gradient of the post-editing task strategy, driving the model to optimize towards post-editing directions that produce higher quality improvements.

[0048] Understandably, the process begins by sampling the same source language text to generate multiple post-edited texts. Then, the quality assessment model calculates a quality difference score for each post-edited text against the second reference translation, based on the second source language text. Next, based on this score set and the difference between the post-edited text scores and the corresponding baseline scores of the second source language text, an independent advantage value (IOP) is calculated for each post-edited text, reflecting its relative optimization performance. Specifically, standardization is achieved by subtracting the average of all quality differences in the set from the quality score difference of each post-edited text and dividing by its standard deviation, thus obtaining the second advantage value. Finally, this set of second advantage values ​​is used to calculate the policy gradient of the large language model on the post-editing task. This allows the model to focus on improving post-editing strategies that bring greater quality improvements and suppressing ineffective editing directions, thereby driving the model to accurately and stably improve its ability to autonomously optimize translations.

[0049] The reinforcement learning-based machine translation method provided in this invention generates a post-edited text set based on the second tuple data in the post-editing task layer. A quality estimation model is used to calculate the quality improvement increment of the post-edited text relative to the second text to be edited, which is then used as the second advantage value. This constructs a reinforcement learning optimization signal centered on "optimization increment." This technical solution transforms the traditional "passive imitation optimization" that relies on a single reference translation into a strategy learning approach that guides the model to explore and reinforce quality improvement patterns between the text to be edited and optimization candidates. This not only enables the model to more accurately capture the language correction logic in post-editing scenarios and learn the optimization path from low-quality translations to high-quality expressions, but also provides a common incremental evaluation benchmark for collaborative optimization with the translation task layer. This allows the advantage value signals of the two tasks to have natural comparability and synergy within the GRPO training framework, ultimately driving the model to build a complete closed-loop capability of "translation generation - post-editing optimization."

[0050] Based on any of the above embodiments, the step of determining the second dominance value using a quality estimation model based on the basic scoring result, the second source language text, the second reference translation, and the post-edited text set includes: Based on the second reference translation, the quality estimation model is used to score each post-edited text in the post-edited text set to obtain a second scoring result set; A second mean and a second standard deviation are determined based on the basic scoring results and the second scoring result set; Based on the baseline scoring results, the second scoring result set, the second average value, and the second standard deviation, a second advantage value is determined for each post-edited text in the post-edited text set.

[0051] It should be noted that the second scoring result set is obtained by independently scoring each post-edited text in the post-edited text set using a quality estimation model. This directly reflects the absolute performance of each optimization candidate in terms of quality dimensions such as semantic fidelity, fluency, and grammatical regularity under the current post-editing strategy; while the previously obtained basic score results b lst The initial baseline performance of the text to be edited represents the initial performance of the text in the same quality dimension. Combining these two metrics quantifies the quality improvement brought about by a single post-edited text. Subsequently, the second average value calculated based on the base score and the second score result set represents the average optimized quality baseline of the current post-editing strategy when processing the text to be edited. The second standard deviation quantifies the dispersion of the scores of this group of post-edited texts, reflecting the magnitude of quality differences between different post-editing candidates. Finally, by calculating the quality improvement of a single post-edited text (b... i -bl stThe second advantage value of each post-edited text is obtained by normalizing it with the second mean and the second standard deviation. This calculation method retains the core value of "optimization increment" and eliminates the difference in scoring scale between different batches of post-editing tasks through statistical normalization. This allows the second advantage value to more accurately reflect the relative optimization value of a single post-editing candidate under the current strategy, providing reinforcement learning with a unified and stable reward signal, thereby supporting efficient iteration of dual-task collaborative training.

[0052] In practice, the formula for calculating the second advantage value is as follows: In the formula, This represents the average improvement in score of all edited texts relative to the second language text being edited, where blst represents the base quality score of the second language text being edited. Let bi represent the standard deviation of the improvement in the rating of all post-edited texts relative to the second language text to be edited, and let bi represent the rating of the i-th post-edited text in the set of post-edited texts.

[0053] Understandably, multiple post-edited texts are first generated by sampling the same second language text to be edited, resulting in a set of post-edited texts, where b i This represents the quality score obtained by the quality estimation model for the i-th edited text in the set; then, using the base quality score bl of the second text to be edited... st Based on this, calculate the quality improvement b for each post-edited text. i -b lst This quantifies the optimization effect of a single post-edit candidate text; then, it calculates the average Mean(bb) based on the quality improvement of all post-edited texts. lst ) and standard deviation Std(bb lst The mean reflects the average optimization level of the current post-editing strategy on this task, while the standard deviation quantifies the dispersion of quality improvement among different post-editing candidates; specifically, the quality improvement b of a single post-edited text is used as the metric. i -b lst Subtracting the average of all improvements and dividing by its standard deviation standardizes the quality improvement, resulting in a second advantage value that reflects the relative optimization value of a single post-edited text. Finally, this set of second advantage values ​​is used to calculate the policy gradient of the large language model on the post-editing task, enabling the model update to focus on improving post-editing strategies that bring greater quality improvements and suppressing ineffective editing directions, thereby driving the model to accurately and stably improve its ability to autonomously optimize translation.

[0054] Understandably, the simplified formula for calculating the second dominance value yields: Therefore, both translation and post-editing tasks can be processed uniformly within the same advantageous computing framework.

[0055] The reinforcement learning-based machine translation method provided in this invention constructs a reinforcement learning optimization signal centered on "relative optimization increment" by scoring the quality of the post-edited text set, combining the average and standard deviation of the base score of the text to be edited, and normalizing them. This constructs a second advantage value. This transforms the traditional "static evaluation" that relies on a single absolute score as a reward signal into a dynamic strategy learning that guides the model to explore and strengthen the relative optimization value among post-edited candidates in the same batch. This not only enables the model to more accurately quantify the quality improvement of each post-edited candidate and learn the optimization path from low-quality translation to high-quality expression, but also provides a common statistically normalized evaluation benchmark for collaborative optimization with the translation task layer. This allows the advantage value signals of the two tasks to have natural comparability and stability within the GRPO training framework, ultimately driving the model to build a complete closed loop of "translation generation-post-editing optimization".

[0056] Figure 3 This is the second flowchart illustrating the reinforcement learning-based machine translation method provided by this invention, as shown below. Figure 3 As shown, step 104 further includes step 1041: Step 1041: Based on the first advantage value or the second advantage value, optimize the objective function through a maximization strategy, iteratively update the model parameters of the large language model until the model converges, and obtain the target large language model.

[0057] It should be noted that when training a large language model, training samples can be randomly selected from a mixed dataset. If the training sample comes from a translation task (first tuple data), the corresponding first advantage value is used as the reward signal for that sample's output to optimize the model's cross-language conversion strategy. If the training sample comes from a post-editing task (second tuple data), the corresponding second advantage value is used as the reward signal to optimize the model's monolingual polishing and correction strategy. Through this task-aware signal routing mechanism, the model can simultaneously process and respond to learning signals from both tasks within the same training batch or iteration, thereby achieving collaborative updating and deep fusion of translation and post-editing capabilities. This design ensures the effectiveness of mixed training, enabling the model to internally allocate and coordinate the two capabilities to generate higher-quality translations.

[0058] It should be noted that the model parameters of a large language model can be updated by maximizing the GRPO objective function (i.e., the policy optimization objective function), the expression of which is: In the formula, This indicates sampling of a mixed dataset D (containing the first tuple data and the second tuple data). The input x can be either the first tuple data (src, tgt) or the second tuple data (src, lst, tgt). The old policy model is obtained from the previous round of parameter updates. A set of output instances generated for input x; This is the current training model; This indicates that the currently trained model generates the corresponding output given input x. The probability of; This indicates that the old policy model generates the corresponding output given input x. The probability; the clip(x,y,z) function restricts the probability ratio to the interval [1-ε,1+ε] (the value is 1-ε when x<1-ε; the value is 1+ε when x>1+ε). The hyperparameter ε controls the tightness of the constraint to avoid the policy update being too large, which would lead to training instability. The KL divergence penalty term is a hyperparameter. Here is the penalty coefficient for KL divergence, and KL divergence is... Used to constrain the distribution offset between the current model and the initial trained model.

[0059] The reinforcement learning-based machine translation method provided in this invention automatically routes corresponding advantage signals to different task samples in a mixed dataset and maximizes the mathematically constrained policy objective function. This drives the model parameters to continuously and smoothly update in the direction of maximizing expected rewards, effectively ensuring the convergence efficiency and numerical stability of large-scale language models in complex reinforcement learning training. This process promotes the co-evolution and deep integration of the model's basic translation capabilities and post-editing optimization capabilities within the same parameter space, ultimately resulting in a target large language model that internalizes advanced decision-making strategies. This model can not only independently complete high-quality cross-language translation but also autonomously and accurately polish the initial translation, thereby systematically improving the overall performance of the final translation in terms of semantic fidelity, language fluency, and naturalness of expression. This solves the problems of capability fragmentation and quality bottlenecks under the traditional single-task optimization framework.

[0060] The reinforcement learning-based machine translation apparatus provided by this invention will be described below. The reinforcement learning-based machine translation apparatus described below can be referred to in correspondence with the reinforcement learning-based machine translation method described above. For example... Figure 4 As shown, the reinforcement learning-based machine translation device includes: The acquisition module 10 is used to acquire a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; The determination module 20 is used to process the first tuple data through the translation task layer to obtain a candidate translation text set, and to determine a first advantage value based on the candidate translation text set; The determining module 20 is used to process the second tuple data through the post-editing task layer to obtain a post-edited text set, and to determine a second advantage value based on the post-edited text set; Training module 30 is used to perform reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, so as to generate machine translation results based on the target large language model.

[0061] Optionally, the determining module 20 is further configured to: The first source language text is processed by the translation task layer to obtain a candidate translation text set; Based on the first reference translation and the candidate translation text set, a first advantage value is determined by a quality estimation model.

[0062] Optionally, the determining module 20 is further configured to: Based on the first reference translation, the quality estimation model is used to score each candidate translation text in the candidate translation text set to obtain a first score result set; Determine the first mean and the first standard deviation based on the first set of scores; Based on the first scoring result set, the first average value, and the first standard deviation, a first advantage value is determined for each candidate translation text in the candidate translation text set.

[0063] Optionally, the second tuple data includes a second source language text, a second language text to be edited, and a second reference translation; wherein, the determining module 20 is further configured to: Based on the second reference translation, the second language text to be edited is processed through the post-editing task layer to obtain a post-edited text set; Based on the second source language text, the second reference translation, and the post-edited text set, a second dominance value is determined using a quality estimation model.

[0064] Optionally, the determining module 20 is further configured to: Based on the second reference translation, the second language text to be edited is scored using the quality estimation model, and the basic score result is obtained. Based on the second reference translation, the quality estimation model is used to score each post-edited text in the post-edited text set to obtain a second scoring result set; A second mean and a second standard deviation are determined based on the basic scoring results and the second scoring result set; Based on the baseline scoring results, the second scoring result set, the second average value, and the second standard deviation, a second advantage value is determined for each post-edited text in the post-edited text set.

[0065] Optionally, the training module 30 is further configured to: Based on the first advantage value or the second advantage value, the objective function is optimized by maximizing the strategy, and the model parameters of the large language model are iteratively updated until the model converges, thus obtaining the target large language model.

[0066] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a reinforcement learning-based machine translation method. This method includes: acquiring a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; processing first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set; processing second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the post-edited text set; and performing reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, thereby generating machine translation results based on the target large language model.

[0067] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the reinforcement learning-based machine translation method provided by the above methods. The method includes: acquiring a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; processing first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set; processing second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the post-edited text set; and performing reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, so as to generate machine translation results based on the target large language model.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the reinforcement learning-based machine translation method provided by the above methods. The method includes: acquiring a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; processing first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set; processing second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the post-edited text set; and performing reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, thereby generating machine translation results based on the target large language model.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine translation method based on reinforcement learning, characterized in that, include: Obtain a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; The translation task layer processes the first tuple data to obtain a candidate translation text set, and determines a first advantage value based on the candidate translation text set. The second tuple data is processed by the post-editing task layer to obtain a post-edited text set, and a second advantage value is determined based on the post-edited text set. Based on the first advantage value and the second advantage value, the large language model is trained by reinforcement learning to obtain a target large language model, and machine translation results are generated based on the target large language model.

2. The machine translation method based on reinforcement learning according to claim 1, characterized in that, The first tuple data includes a first source language text and a first reference translation; wherein, the step of processing the first tuple data through the translation task layer to obtain a candidate translation text set, and determining a first advantage value based on the candidate translation text set, includes: The first source language text is processed by the translation task layer to obtain a candidate translation text set; Based on the first reference translation and the candidate translation text set, a first advantage value is determined by a quality estimation model.

3. The machine translation method based on reinforcement learning according to claim 2, characterized in that, The step of determining the first advantage value based on the first reference translation and the candidate translation text set through a quality estimation model includes: Based on the first reference translation, the quality estimation model is used to score each candidate translation text in the candidate translation text set to obtain a first score result set; Determine the first mean and the first standard deviation based on the first set of scores; Based on the first scoring result set, the first average value, and the first standard deviation, a first advantage value is determined for each candidate translation text in the candidate translation text set.

4. The machine translation method based on reinforcement learning according to claim 1, characterized in that, The second tuple data includes a second source language text, a second language text to be edited, and a second reference translation; wherein, the process of processing the second tuple data through the post-editing task layer to obtain a post-edited text set, and determining a second advantage value based on the post-edited text set, includes: Based on the second reference translation, the second language text to be edited is processed through the post-editing task layer to obtain a post-edited text set; The second language text to be edited is scored using a quality estimation model to obtain a basic score result; Based on the baseline score, the second source language text, the second reference translation, and the post-edited text set, a second advantage value is determined using a quality estimation model.

5. The machine translation method based on reinforcement learning according to claim 4, characterized in that, The determination of the second dominance value based on the baseline scoring results, the second source language text, the second reference translation, and the post-edited text set, using a quality estimation model, includes: Based on the second reference translation, the quality estimation model is used to score each post-edited text in the post-edited text set to obtain a second scoring result set; A second mean and a second standard deviation are determined based on the basic scoring results and the second scoring result set; Based on the baseline scoring results, the second scoring result set, the second average value, and the second standard deviation, a second dominance value is determined for each post-edited text in the post-edited text set.

6. The machine translation method based on reinforcement learning according to claim 1, characterized in that, The step of training the large language model using reinforcement learning based on the first dominance value and the second dominance value to obtain the target large language model includes: Based on the first advantage value or the second advantage value, the objective function is optimized by maximizing the strategy, and the model parameters of the large language model are iteratively updated until the model converges, thus obtaining the target large language model.

7. A machine translation device based on reinforcement learning, characterized in that, include: An acquisition module is used to acquire a large language model, wherein the large language model includes a translation task layer and a post-editing task layer; The determination module is used to process the first tuple data through the translation task layer to obtain a candidate translation text set, and to determine a first advantage value based on the candidate translation text set; The determining module is used to process the second tuple data through the post-editing task layer to obtain a post-edited text set, and to determine a second advantage value based on the post-edited text set; The training module is used to perform reinforcement learning training on the large language model based on the first advantage value and the second advantage value to obtain a target large language model, so as to generate machine translation results based on the target large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the reinforcement learning-based machine translation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the reinforcement learning-based machine translation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the reinforcement learning-based machine translation method as described in any one of claims 1 to 6.

Images