A generative machine translation reward model based on an intra-group joint evaluation paradigm and a training method thereof

Abstract

The application provides a generative machine translation reward model based on an in-group joint evaluation paradigm and a training method thereof, and comprises the following steps: step 1, constructing training data; step 2, based on the training data, selecting a base model and performing supervised learning fine-tuning training to obtain a cold start group relative reward model; step 3, based on a verifiable reward reinforcement learning, optimizing the cold start group relative reward model to obtain a final group relative reward model; and step 4, using the final group relative reward model to optimize a target translation model to obtain an optimized translation model for a machine translation task.

Description

Technical Field

[0001] This invention relates to a machine translation reward model and its training method, particularly a generative machine translation reward model and its training method based on an intra-group joint evaluation paradigm. Background Technology

[0002] Machine translation (MT) is a natural language processing technology that automatically converts source language text into target language text. It is widely used in practical business scenarios such as cross-language information retrieval, international e-commerce, cross-border customer service, content export, and software localization. Modern machine translation systems typically use neural network sequence generation models (such as the Transformer architecture) to generate target language translations from source sentences.

[0003] In real-world applications, translation quality depends not only on surface fluency but also on meeting several criteria: semantic fidelity, information completeness, terminology / entity consistency, correct localization, appropriate style and register, and the ability to handle challenging samples such as complex sentence structures, implicit semantics, ambiguity resolution, and idioms / slang. To further improve model performance across these dimensions, the industry often employs reinforcement learning (RL) to optimize translation models: treating the translation model as a policy model, sampling and generating translations, and updating parameters based on reward signals to make the model more inclined to generate high-quality translations. Since machine translation is an open-ended generative task, rewards are often difficult to obtain directly through verification of correctness as in mathematical and programming tasks. Therefore, during training, reward models or automatic evaluators are commonly used to score the quality of candidate translations, thus providing learning signals for reinforcement learning.

[0004] For reward modeling in reinforcement learning training for machine translation, existing solutions typically adopt the Scalar Quality Metric (SQM) paradigm, which scores each candidate translation independently. Its core is to map the input pair (x,y) to a scalar score.

[0005] Although the SQM paradigm is currently the mainstream choice, existing techniques all have significant drawbacks when applied to GRPO (Group Relative Policy Optimization) training that relies on within-group relative ranking:

[0006] Among these challenges, Discriminative Reward Models (DRMs) lack explicit reasoning and interpretable comparison processes, making them ill-suited for complex translation scenarios. DRMs typically output scores or preference probabilities directly, lacking explicit reasoning and failing to grasp complex linguistic phenomena (such as long-distance dependencies, metaphors, and specific terminology constraints). On challenging datasets, DRM performance degrades drastically, failing to provide proper optimization guidance for policy models. Furthermore, because reinforcement learning allows policy models to explore autonomously, DRMs are susceptible to reward hijacking.

[0007] Existing GenRM (Generative Reward Model) based on the SQM paradigm suffers from low ranking accuracy and a significant "score saturation" problem. Despite possessing inference potential lacking in DRM, GenRM lags behind in ranking accuracy. This is because the method of scoring each translation independently lacks comparative context, making it difficult for the model to calibrate scores: it cannot distinguish subtle quality differences between candidate translations and is prone to overestimating superficially fluent but semantically flawed samples. This ultimately leads to severe score saturation, where the model tends to assign the same score (e.g., all full marks) to candidates of varying quality, causing the variance-normalized advantage value in GRPO to approach zero, resulting in gradient vanishing and ineffective policy updates. Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a generative machine translation reward model based on the intra-group joint evaluation paradigm and its training method.

[0009] To address the aforementioned technical problems, this invention discloses a generative machine translation reward model based on an intra-group joint evaluation paradigm and its training method, comprising the following steps:

[0010] Step 1, construct training data;

[0011] Step 2: Based on the training data, select a base model and perform supervised learning fine-tuning training to obtain the cold start group relative reward model;

[0012] Step 3: Optimize the cold start group relative reward model based on verifiable reward reinforcement learning to obtain the final group relative reward model;

[0013] Step 4: Optimize the target translation model using the final group relative reward model to obtain the optimized translation model, which can then be used for machine translation tasks.

[0014] Furthermore, the construction of training data described in step 1 includes:

[0015] Step 1-1: Construct a source sentence sample set to obtain the source sentence text data set for machine translation. Each source sentence is denoted as ;

[0016] Steps 1-2: Construct candidate translation groups for each source sentence. Collect its corresponding 1 candidate translation, construct a candidate translation set. , represented as:

[0017]

[0018] in, Indicates the first One candidate translation;

[0019] Steps 1-3 involve joint quality annotation within the group. Within the candidate translation group, structured annotation is performed, including: source sentence analysis text, detailed comparative analysis text between candidate translations, relative ranking within the group based on the analysis, and relative quality score vector consistent with the ranking.

[0020] Furthermore, step 2, which involves selecting the base model and performing supervised learning fine-tuning training, includes:

[0021] Step 2-1: Select a pre-trained generative language model as the base model and load the original pre-trained parameters in the base model as the initial state for subsequent supervised learning fine-tuning training.

[0022] Step 2-2: Design the input and output data formats and verification rules for the base model;

[0023] Steps 2-3: Construct the input sequence using a preset template that conforms to the input data format, and convert the source sentence... and candidate translation set Concatenate into a single-input sequence ;

[0024] Steps 2-4 involve supervised fine-tuning training, using the structured annotations from steps 1-3 as the supervision signal to train the pedestal model, obtaining the cold-start group relative reward model, and then training from a single input sequence. The system obtains structured output, including: intra-group comparative analysis, intra-group ranking results, and relative quality scores of each candidate translation within the group.

[0025] Furthermore, the input and output data formats and verification rules for the design base model described in step 2-2 include:

[0026] Step 2-2-1: Set the input format, the input data should include the source sentence. and candidate translation set And assemble them in order;

[0027] Step 2-2-2: Set the output format, which includes: candidate comparison analysis, predicted ranking, and a set of relative quality scores; among which,

[0028] Source sentence analysis and candidate comparison analysis involve generating an analytical text for the source sentence and comparing it with a set of candidate translations. Generate analytical text for item-by-item or pairwise comparisons;

[0029] Predictive ranking, which is the result of ranking candidate translations within a group;

[0030] The set of relative quality scores, i.e., the relative quality scores of the outputs and the candidate translations, is represented as:

[0031]

[0032] in, Corresponding to the One candidate translation The relative mass fraction;

[0033] Step 2-2-3, set up output verification and fault tolerance mechanisms, specifically including:

[0034] Format validation, checking the relative quality score length and the completeness of output fields;

[0035] Consistency verification involves checking the sorting results within a group based on the relative quality score.

[0036] If the verification fails, perform a retry to generate or rollback.

[0037] Furthermore, step 3, which involves optimizing the relative reward model for the cold-start group based on verifiable reward reinforcement learning, includes:

[0038] Step 3-1, constructing training augmentation data for verifiable reward reinforcement learning, specifically including:

[0039] Based on the training data constructed in step 1, the candidate translation group is randomly permuted in order, and subsets of different sizes are sampled from the original candidate translation group to form training augmentation data;

[0040] Step 3-2: Based on the training augmentation data, use the cold start group relative reward model to obtain the predicted in-group ranking results and relative quality scores;

[0041] Step 3-3: Calculate the composite reward based on the relative quality score labeled in the training augmentation data and the predicted relative quality score. This reward simultaneously models both coarse-grained and fine-grained ranking accuracy as feedback and encourages the model's output to conform to the format requirements.

[0042] Steps 3-4 involve using reinforcement learning to maximize the composite reward and updating the parameters of the cold-start group relative reward model to obtain the final group relative reward model.

[0043] Furthermore, the calculation of the compound reward described in step 3-3 includes:

[0044]

[0045] in, As a compound reward, Rewards are given based on the accuracy of sorting within the group. Perceive reward for score difference Gating; among which, the accuracy of sorting within the group is rewarded. The calculation method is as follows:

[0046]

[0047] in, Represents the number of combinations. and For any candidate translation pair The numbering of the candidate translations, For candidate translations Consistency score;

[0048] Gating The calculation method is as follows:

[0049] According to the output verification and fault tolerance mechanism set in step 2-2-3, if the verification is successful, then gating is set. otherwise ;

[0050] Score Difference Perceived Reward The calculation method is as follows:

[0051]

[0052] in, Preset error Mapping rules.

[0053] Furthermore, the candidate translations... Consistency score and score interpolation error The calculation method is as follows:

[0054]

[0055]

[0056] in, and Candidate translation pairs annotated in training augmented data The relative mass fraction, and It is the predicted relative quality score. For symbolic functions, This is an indicator function.

[0057] Furthermore, step 4, which involves optimizing the target translation model using the final group-relative reward model, includes:

[0058] Step 4-1: Obtain source sentence text data for optimizing the translation model;

[0059] Step 4-2, for each source sentence in the source sentence text data Candidate groups are generated using the translation model to be optimized. ;

[0060] Step 4-3, concatenate the text sequence Input the final group relative reward model to obtain the relative quality score vector. ;

[0061] Step 4-4, based on the relative quality fraction vector Calculate the advantage value for each candidate translation ;

[0062] Steps 4-5, based on the dominance value Update the parameters of the translation model to be optimized to obtain the optimal translation model.

[0063] Furthermore, step 4-4 describes the process based on the relative mass fraction vector. Calculate the advantage value for each candidate translation ,include:

[0064]

[0065] in, Representing vectors The mean, Representing vectors The standard deviation.

[0066] Furthermore, the intra-group joint quality labeling described in steps 1-3 can be performed using manual labeling or existing evaluation models.

[0067] Beneficial effects:

[0068] 1. The technical solution of the present invention uses a joint comparison mechanism within the same group for candidate translation groups of the same source sentence. The reward model obtains a unified intra-group reference, which can more stably distinguish fine-grained quality differences in machine translation, reduce the probability of intra-group ranking reversal and score saturation, and significantly improve the reliability of intra-group ranking. This provides a more effective advantage signal for group-based reinforcement learning such as GRPO.

[0069] 2. The technical solution output of this invention includes candidate translation comparison analysis and ranking criteria, which can make fuller use of the reasoning ability of large language models, while having better interpretability, facilitating training and debugging, quality auditing and locating the source of reward bias.

[0070] 3. The technical solution of this invention replaces the online invocation of ultra-large LLM review or manual review in the training loop with a smaller and deployable group relative reward model, which significantly reduces the latency and cost of machine translation reinforcement learning training and is suitable for large-scale multilingual and multi-domain training and continuous iteration.

[0071] 4. The technical solution of the present invention does not rely on the existence of a unique correct translation in machine translation tasks, and is especially suitable for scenarios where "the difference in translation quality is slight but has a significant impact on business" (such as terminology consistency, brand terminology, localization standards, compliant expressions, etc.), and can be extended to other open generation tasks. Attached Figure Description

[0072] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0073] Figure 1 A schematic diagram of the overall process for constructing the Group Relative Reward Model (GRRM).

[0074] Figure 2 A schematic diagram of the process for constructing training data for GQM.

[0075] Figure 3 This is a schematic diagram of the SFT cold start training process.

[0076] Figure 4 A schematic diagram of the RLVR optimization process.

[0077] Figure 5 This is a flowchart illustrating the process of applying GRRM to GRPO training for machine translation.

[0078] Figure 6 A template for adding prompts to GQM.

[0079] Figure 7 A diagram illustrating the prompt word template for GRRM. Detailed Implementation

[0080] The overall idea of ​​this invention is as follows: In the GRPO reinforcement learning training process for machine translation, existing reward models generally use SQM to score each sentence independently. This results in the inability to stably and finely distinguish candidate translation groups for the same source sentence, easily leading to inaccurate intra-group ranking and score saturation. Consequently, this causes problems such as advantage value decay, gradient vanishing, or incorrect update direction in GRPO training. This invention provides a machine translation reward modeling and optimization method based on Group Quality Metric (GQM). This method aims to fully leverage the powerful inference potential of generative models by introducing an intra-group context comparison mechanism, providing reward signals that are structurally highly aligned with the GRPO group relative update mechanism, thereby improving the optimization performance of translation models in complex semantic discrimination, localization, and challenging scenarios.

[0081] The technical solution proposed in this invention is as follows: A method for constructing a Group Relative Reward Model (GRRM) for reinforcement learning training in machine translation, addressing the key needs in Group Relative Policy Optimization (GRPO) training, and adopting the Group Quality Metric (GQM) paradigm: for the same source sentence Multiple candidate translations To conduct intragroup joint evaluation, the model is required to first perform fine-grained analysis on translation differences, and then output intragroup rankings and relative score vectors consistent with the rankings, thereby providing stable and distinguishable reward signals for GRPO.

[0082] The process of constructing a group relative reward model is as follows: Figure 1 As shown, it mainly includes the following three steps:

[0083] Step 1, GQM training data construction. For each source sentence to be translated (source language text data), construct candidate translation groups; perform joint annotation within each candidate translation group (target language text) to obtain initial instruction compliance capabilities that can be used to train the GRRM; the specific data construction process is as follows... Figure 2 As shown:

[0084] Step 1-1: Construct a sample set of source sentences. Obtain the set of source sentence data for machine translation. Each source sentence is denoted as Data can come from general corpora, business domain corpora, or challenge sets.

[0085] Steps 1-2: Construct candidate translation groups. For each source sentence... Construct a candidate translation set Candidate translations can be generated using existing translation models under different sampling parameters, or generated by multiple different models / systems, or manually annotated. The group size... It can be configured to any integer greater than 1.

[0086] Steps 1-3: Intra-group joint quality labeling. The labeling should use the prompt / rule requirements for "intra-group joint evaluation" and output the following:

[0087] (1) Detailed comparative analysis of the source text and candidate translations;

[0088] (2) The relative ranking within the group obtained based on this analysis;

[0089] (3) The relative score vector consistent with the sorting.

[0090] The annotation party can be a human annotator or a high-capability assessment model, etc.

[0091] Step 2, Cold Start Based on Supervised Fine-Tuning. Based on the intra-group joint labeled data obtained in Step 1, the base model is fine-tuned using supervised learning (SFT) to enable it to follow instructions for evaluating candidate groups, output according to a prescribed structure, and perform preliminary intra-group ranking and relative scoring. The specific process is as follows: Figure 3 As shown:

[0092] Step 2-1, Initialize the model. Select a pre-trained generative language model as the base model (e.g., a general large language model or a translation domain model), and load the original pre-trained parameters of the base model as the initial state for subsequent supervised fine-tuning (SFT).

[0093] Step 2-2, design the input and output data formats and validation rules, as follows:

[0094] To ensure that GRRM can be automatically invoked by the program in the training loop, this invention limits GRRM to a machine-parseable structured input / output format.

[0095] Input format. GRRM input includes at least the source text and a set of candidate translations, which can be represented as a pair of tuples. .in The source language text to be translated. For the set of candidate translations, All inputs are concatenated according to a predefined template, ensuring that comparisons between candidates occur within the same context window, thus guaranteeing "intra-group joint evaluation" from the input structure.

[0096] Output format. For each input tuple The relative reward model GRRM is instructed to generate structured output, which includes at least the following three parts: (1) candidate comparison analysis, for the candidate set (1) Generate detailed analysis texts for item-by-item or pairwise comparisons, including semantic analysis of the source sentences and evaluation of the sufficiency, fidelity, and fluency of translation candidates; (2) Predict ranking, providing the intra-group ranking results of candidate translations from best to worst, allowing for parallel relationships; (3) Relative quality score set, outputting a set of scores that correspond one-to-one with the candidates. ,in Corresponding candidate .

[0097] Output validation and fault tolerance. The structured parsing and validation module performs two types of validation strategies on the output: (1) Format validation, checking whether the fraction length is 100%. (1) Check if the fields are complete; (2) Check the consistency of the score relationship and whether it is consistent with the sorting field. If the check fails, a retry generation or rollback strategy can be executed (e.g., output 0) to ensure that the training loop is not interrupted.

[0098] Steps 2-3: Construct the input sequence. Using a preset template that conforms to the input format, ... Concatenate them into a single sequence input.

[0099] Steps 2-4 involve performing supervised fine-tuning training (SFT). Supervised learning is employed, using the structured annotations from step 1 as the supervision signal, enabling the base model to learn from... Generate structured output, including intra-group comparative analysis, ranking results, and relative quality scores of each candidate translation within the group.

[0100] Steps 2-5: After training, the cold start group relative reward model GRRM is obtained, which has the ability to stably perform "joint evaluation - output ranking - output relative score".

[0101] Step 3: Optimization based on Reinforcement Learning with Verifiable Rewards (RLVR). Building upon the cold-start group relative reward model GRRM, RLVR is further employed to optimize its analytical reasoning ability and within-group ranking accuracy, making it more suitable for the within-group reward ranking accuracy requirements of GRPO training. Supervision in the RLVR stage does not include labeled natural language analysis text; it only uses verifiable relative ranking / score relationships to encourage the model to autonomously explore the comparative reasoning process. The specific optimization process is as follows: Figure 4 As shown:

[0102] Step 3-1: Construct verifiable reward reinforcement learning (RLVR) training augmentation data. Based on the sample set from Step 1, the candidate translation group is subjected to (1) random permutation to reduce the model's dependence on fixed positions, and (2) sampling subsets of different sizes from the original group to form training data for the large-scale augmented RLVR stage. The templates designed in Step 2-2 are also used as input.

[0103] Step 3-2: On the constructed augmented data, use GRRM sampling to obtain the predicted ranking results and relative scores.

[0104] Step 3-3: Calculate a composite reward based on the relative quality scores labeled in the training augmentation data and the relative quality scores predicted by GRRM. This reward simultaneously models both coarse-grained and fine-grained ranking accuracy as feedback and encourages the model's output to conform to the required format. Composite Reward The calculation formula is:

[0105]

[0106] in, As a compound reward, Rewards are given based on the accuracy of sorting within the group. Perceive reward for score difference Gating; among which, the accuracy of sorting within the group is rewarded. The calculation method is as follows:

[0107]

[0108] in, Represents the number of combinations. and For any candidate translation pair The numbering of the candidate translations, For candidate translations Consistency score;

[0109] Gating The calculation method is as follows:

[0110] According to the output verification and fault tolerance mechanism set in step 2-2-3, if the verification is successful, then gating is set. otherwise ;

[0111] Score Difference Perceived Reward The calculation method is as follows:

[0112]

[0113] in, Preset error Mapping rules.

[0114] Furthermore, the candidate translations... The consistency score is calculated as follows:

[0115]

[0116] in, and Candidate translation pairs annotated in training augmented data The relative mass fraction, and It is the predicted relative quality score. For symbolic functions, This is an indicator function.

[0117] Steps 3-4 employ reinforcement learning algorithms (such as GRPO, PPO, etc.) to maximize the aforementioned composite reward and update the GRRM parameters to improve its ranking accuracy and discrimination ability for candidate translation groups, while ensuring that the model's output meets the format requirements.

[0118] Steps 3-5: After training is complete, output the final GRRM model.

[0119] Step 4: The GRRM model is used to provide relative reward scores within groups of candidate translations generated for the same source sentence during GRPO machine translation training. Based on these reward scores, the translation model's ability to automatically convert source language text into target language text is improved. The specific process is as follows: Figure 5 As shown:

[0120] Step 4-1, Source Sentence Sample Acquisition. The translation data used to optimize the translation model can be the same as that used to train GRRM, and since no annotation is required, a larger scale of data can be used.

[0121] Step 4-2, for each source sentence Using the translation model to be optimized, candidate groups are generated through sampling. .

[0122] Step 4-3, Input the final GRRM model output from step 3 to obtain the relative scores within groups. .

[0123] Step 4-4, according to Calculate the advantage value for each candidate .

[0124] Steps 4-5 involve updating the translation model to be optimized based on the advantage value to obtain the optimal translation model, which is then applied to the machine translation task.

[0125] In summary, the key point of this invention lies in the proposed method for constructing a Group Relative Reward Model (GRRM) for machine translation tasks. This method breaks the limitation of traditional machine translation reward models that score candidate translations independently one by one. By adopting a joint evaluation and scoring mode within groups, and combining supervised fine-tuning (SFT) and provable reinforcement learning (RLVR) in a two-stage optimization process, it achieves stable, fine-grained differentiation and high-precision ranking of candidate translation groups for the same source sentence. This provides reliable intra-group relative reward signals for training machine translation reinforcement learning methods such as GRPO.

[0126] Example:

[0127] This embodiment provides a machine translation optimization system and method based on the Group Relative Reward Model (GRRM). The hardware environment of this embodiment preferably employs a high-performance computing cluster (e.g., configured with an Nvidia A100 GPU), and the software environment is based on the deep learning framework PyTorch, the large language model supervised fine-tuning framework LLaMAFactory, and the reinforcement learning framework Verl.

[0128] Part 1: Training Data Construction and Preprocessing Module.

[0129] 1. Candidate Group Generation. Taking the Chinese-English translation task as an example, a subset of the TowerBlocks dataset containing approximately 18,000 data points is used. For each source sentence, multiple candidate translations are generated using a seed translation model (e.g., a finely tuned Qwen2.5-7B).

[0130] Group size distribution: To simulate real-world scenarios and cover different levels of difficulty, the candidate group size G is sampled from the set {2,3,4}, with a sampling frequency ratio of 1:1:3.

[0131] Reference translation injection: To ensure the existence of high-quality anchors within a group, the actual reference translation of the source sentence is injected into the candidate group with a probability of 0.5.

[0132] Sampling parameters: When generating candidates, the preferred temperature parameter is T=1.0 to increase diversity.

[0133] 2. GQM Labeling (Label Acquisition). The constructed candidate groups are input into a high-capability assessment model (e.g., Gemini-2.5-Pro) and labeled using the GQM (Group Quality Metric) cue word paradigm, such as... Figure 6 As shown, the group size is 3.

[0134] Part Two: GRRM Input / Output Interface Design

[0135] To ensure that the model output is parsable and verifiable, this embodiment has designed strict prompt word templates and output protocols.

[0136] 1. Input prompt template. The input text format received by the model, such as... Figure 7 As shown, the group size is 3.

[0137] 2. Output format. Follow the format below:

[0138] Analysis process;

[0139] Relative ordering within a group, such as A > B = C;

[0140] Relative fractional vectors, such as A: 8, B: 6, C: 6.

[0141] 3. Output parsing and validation. The system is configured with a parser to extract the relative order and relative score vectors within groups from the end of the text generated by the model.

[0142] Part Three: The SFT Cold Start Phase of the GRRM Model.

[0143] The base model (e.g., Qwen2.5-7B) was fully fine-tuned using the GQM-annotated data described above. This enabled the model to learn to follow the output format and initially align with the labeled scoring criteria. Training was performed for 3 epochs with a batch size of 64, a learning rate of 6e-6 (cosine decay), and a warmup ratio of 0.1.

[0144] Part Four: RLVR Optimization Phase.

[0145] The Group Sequence Policy Optimization (GSPO) algorithm is used to perform reinforcement learning on the SFT-adjusted model.

[0146] 1. RLVR Enhanced Data Processing. Online enhancement is performed on the above data:

[0147] Permutation: Randomly shuffle the order of candidate translations in the input prompt (e.g., swap the positions of candidate A and B), and adjust the labels accordingly to train the model's position invariance.

[0148] Subgroup sampling: A subset (e.g., 2 out of 4) is randomly selected from the G candidates of the original annotation to form new training samples to enhance the robustness of the model to different group contexts.

[0149] 2. RLVR Reward Function Design. This embodiment designs a composite reward:

[0150] Output validation gating: Based on the GRRM sampled output, the sorting results and scores are parsed, and gating is set during format validation and consistency checks. otherwise .

[0151] Ranking accuracy reward: Calculate the predicted score vector Consistency with true label scores .

[0152] Score Difference-Aware Reward: To encourage the model to not only rank correctly but also ensure that the score difference reflects the true quality gap, a discrete kernel function is introduced. Fractional difference error The mapping rule is...

[0153]

[0154] Score Difference Perceived Reward .

[0155] Final composite reward .

[0156] 3. Optimization Phase. The sampling group size is set to 8. The learning rate is set to a constant 1e-5, and the PPO batch size is 128. KL divergence penalty is not used.

[0157] Part 5: Effect Verification.

[0158] 1. Verification objective.

[0159] To verify the effectiveness of the Group Relative Reward Model (GRRM) in machine translation tasks, this embodiment verifies the effect from two dimensions: "improvement of general benchmark translation quality" and "emergence of reasoning ability for complex samples," and compares it with general large models and dedicated translation models.

[0160] 2. Translation strategy model and training data.

[0161] (1) Translation strategy model: Based on Qwen2.5-7B, in order to enable it to perform basic translation, SFT cold start was first performed on a small amount of translation data, and then reinforcement learning optimization was performed. In the reinforcement learning stage, GRRM was used to provide group relative scores as rewards to drive the update of the translation strategy model.

[0162] (2) Multilingual reinforcement learning data: TowerBlocks multilingual translation data was used in the reinforcement learning phase, covering 10 languages ​​and about 150,000 samples.

[0163] 3. Test dataset and evaluation metrics.

[0164] (1) General benchmark set: mainly based on Zh→En, and extended to the multilingual direction of En→X. The preferred test sets are: WMT23Zh→En; WMT24++ En→Zh; WMT24++ En→X (average of 7 languages);

[0165] (2) Complex sample benchmark set: Seed-X-Challenge (containing highly difficult samples such as idioms, slang, ambiguity, and cultural load, suitable for testing reasoning and fine-grained decision-making ability).

[0166] (3) Evaluation Metrics: The preferred evaluation metrics are those based on large language models, which better match human perception of quality compared to automatic metrics and offer better interpretability. For the Seed-X-Challenge benchmark, the translation difficulty annotations (expert annotations) provided by the dataset, along with reference translations, can be further incorporated into the evaluation prompts, enabling judges to make more informed judgments regarding the difficulties. This embodiment uses DeepSeek-R1-0528 as the evaluation model.

[0167] 4. Compare with baseline settings.

[0168] To demonstrate the sources of benefit and relative advantages of the method in this embodiment, the following comparison is set up:

[0169] Translation-specific models: TowerInstruct-13B, SeedX-PPO.

[0170] Internal comparison in this embodiment: Qwen2.5-7B-SFT (a cold-start model with basic translation capabilities) and its GRPO-optimized version (based on GRRM rewards).

[0171] 5. Main experimental results and conclusions.

[0172] The results are shown in Table 1. On common test sets such as WMT23 and WMT24++, this embodiment achieves significant improvements over Qwen2.5-7B-SFT in multiple language pairs, especially in the En→X multilingual direction. In Seed-X-Challenge, which emphasizes analysis and reasoning, this embodiment shows the most outstanding improvement and outperforms many open-source translation models.

[0173] Table 1. Experimental Results (Evaluation Indicators for Large Language Model)

[0174]

[0175] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a generative machine translation reward model based on an intra-group joint evaluation paradigm and its training method, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0176] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MCU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0177] This invention provides a generative machine translation reward model based on an intra-group joint evaluation paradigm and its training method. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A generative machine translation reward model based on an intra-group joint evaluation paradigm and its training method, characterized in that, Includes the following steps: Step 1, construct training data; Step 2: Based on the training data, select a base model and perform supervised learning fine-tuning training to obtain the cold start group relative reward model; Step 3: Optimize the cold start group relative reward model based on verifiable reward reinforcement learning to obtain the final group relative reward model; Step 4: Optimize the target translation model using the final group relative reward model to obtain the optimized translation model, which can then be used for machine translation tasks.

2. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm as described in claim 1, characterized in that, The construction of training data described in step 1 includes: Step 1-1: Construct a source sentence sample set to obtain the source sentence text data set for machine translation. Each source sentence is denoted as ; Steps 1-2: Construct candidate translation groups for each source sentence. Collect its corresponding 1 candidate translation, construct a candidate translation set. , represented as: ； in, Indicates the first One candidate translation; Steps 1-3 involve joint quality annotation within the group. Within the candidate translation group, structured annotation is performed, including: source sentence analysis text, detailed comparative analysis text between candidate translations, relative ranking within the group based on the analysis, and relative quality score vector consistent with the ranking.

3. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm as described in claim 2, characterized in that, Step 2, which involves selecting the base model and performing supervised learning fine-tuning training, includes: Step 2-1: Select a pre-trained generative language model as the base model and load the original pre-trained parameters in the base model as the initial state for subsequent supervised learning fine-tuning training. Step 2-2: Design the input and output data formats and verification rules for the base model; Steps 2-3: Construct the input sequence using a preset template that conforms to the input data format, and convert the source sentence... and candidate translation set Concatenate into a single-input sequence ; Steps 2-4 involve supervised fine-tuning training, using the structured annotations from steps 1-3 as the supervision signal to train the pedestal model, obtaining the cold-start group relative reward model, and then training from a single input sequence. The system obtains structured output, including: intra-group comparative analysis, intra-group ranking results, and relative quality scores of each candidate translation within the group.

4. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 3, characterized in that, The input and output data formats and verification rules for the design base model described in step 2-2 include: Step 2-2-1: Set the input format, the input data should include the source sentence. and candidate translation set And assemble them in order; Step 2-2-2: Set the output format, which includes: source sentence analysis and candidate comparison analysis, predicted ranking, and a set of relative quality scores; among which, Source sentence analysis and candidate comparison analysis involve generating an analytical text for the source sentence and comparing it with a set of candidate translations. Generate analytical text for item-by-item or pairwise comparisons; Predictive ranking, which is the result of ranking candidate translations within a group; The set of relative quality scores, i.e., the relative quality scores of the outputs and the candidate translations, is represented as: ； in, Corresponding to the One candidate translation The relative mass fraction; Step 2-2-3, set up output verification and fault tolerance mechanisms, specifically including: Format validation, checking the relative quality score length and the completeness of output fields; Consistency verification involves checking the sorting results within a group based on the relative quality score. If the verification fails, perform a retry to generate or rollback.

5. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 4, characterized in that, Step 3, which involves optimizing the relative reward model for the cold-start group based on verifiable reward reinforcement learning, includes: Step 3-1, constructing training augmentation data for verifiable reward reinforcement learning, specifically including: Based on the training data constructed in step 1, the candidate translation group is randomly permuted in order, and subsets of different sizes are sampled from the original candidate translation group to form training augmentation data; Step 3-2: Based on the training augmentation data, use the cold start group relative reward model to obtain the predicted in-group ranking results and relative quality scores; Step 3-3: Calculate the composite reward based on the relative quality score labeled in the training augmentation data and the predicted relative quality score; Steps 3-4 involve using reinforcement learning to maximize the composite reward and updating the parameters of the cold-start group relative reward model to obtain the final group relative reward model.

6. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 5, characterized in that, The calculation of compound reward described in step 3-3 includes: ； in, As a compound reward, Rewards are given based on the accuracy of sorting within the group. Perceive reward for score difference Gating; among which, the accuracy of sorting within the group is rewarded. The calculation method is as follows: ； in, Represents the number of combinations. and For any candidate translation pair The numbering of the candidate translations, For candidate translations Consistency score; Gating The calculation method is as follows: According to the output verification and fault tolerance mechanism set in step 2-2-3, if the verification is successful, then gating is set. otherwise ; Score Difference Perceived Reward The calculation method is as follows: ； in, Preset error Mapping rules.

7. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 6, characterized in that, Candidate translations Consistency score and score interpolation error The calculation method is as follows: ； ； in, and Candidate translation pairs annotated in training augmented data The relative mass fraction, and It is the predicted relative quality score. For symbolic functions, This is an indicator function.

8. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 7, characterized in that, Step 4, which involves optimizing the target translation model using the final group-relative reward model, includes: Step 4-1: Obtain source sentence text data for optimizing the translation model; Step 4-2, for each source sentence in the source sentence text data Candidate groups are generated using the translation model to be optimized. ; Step 4-3, concatenate the text sequence Input the final group relative reward model to obtain the relative quality score vector. ; Step 4-4, based on the relative quality fraction vector Calculate the advantage value for each candidate translation ; Steps 4-5, based on the dominance value Update the parameters of the translation model to be optimized to obtain the optimal translation model.

9. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 8, characterized in that, Step 4-4 describes the process based on the relative mass fraction vector. Calculate the advantage value for each candidate translation ,include: ； in, Representing vectors The mean, Representing vectors The standard deviation.

10. The generative machine translation reward model and its training method based on the intra-group joint evaluation paradigm according to claim 9, characterized in that, The joint quality labeling within the group described in steps 1-3 can be performed using manual labeling or existing evaluation models.

Images