A training method for a content-generating scorer based on preference data

Abstract

This invention provides a training method for a generated content scorer based on preference data. The method includes: supervising the fine-tuning of a multimodal large model using training data containing thought chain reasoning trajectories and final scores; during the fine-tuning process, masking the thought chain reasoning trajectories of a portion of the training data, retaining only the final scores as supervision signals to force the model to establish an implicit mapping from generated content input to point-level score output; constructing a pairwise reward mechanism using human preference labels, calculating the relative rewards between different outputs, and iteratively optimizing the model parameters based on the relative rewards to obtain the generated content scorer. This invention can train a high-precision generated content scorer that can assign point-level scores to generated content and simultaneously provide logically consistent and highly interpretable scoring reasons, thereby achieving a quantitative judgment of the quality of generated content, relying solely on human preference data.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically a training method for a content scoring generator based on preference data. Background Technology

[0002] Despite the advancements in generative models and their ability to synthesize vivid and realistic content, two major challenges remain. First, the generation quality is unstable, often marred by low-quality flaws, such as limb distortions common in static images and physical inconsistencies in generated videos. These defects severely undermine the realism and credibility of the content, increasing the difficulty of practical applications. Particularly in high-risk scenarios with stringent safety requirements, such as autonomous driving visualization, human-machine collaborative systems, and remote operation and monitoring, these flaws can easily escalate into significant decision-making risks and safety hazards. Second, even when some generated content possesses high visual quality, it often deviates from genuine human intent or fails to meet human preferences. Therefore, there is an urgent need to train a reliable quality scoring model (i.e., a scorer) based on human needs, providing clear direction for the iterative optimization of subsequent generative models.

[0003] Currently, the optimization paths for such scorers are mainly divided into two categories: discriminative and generative. Discriminative methods typically rely on human preference datasets, maximizing the model's win rate in predicting positive and negative samples to achieve a quality ranking ability that aligns with human preferences. However, this method often exhibits weak generalization performance and cannot incorporate thought chains (CoT) to encourage deeper model thinking, thus limiting further improvements in scoring accuracy. On the other hand, generative methods fully utilize the powerful cross-modal understanding capabilities of large multimodal models, fine-tuning the model by reducing the error between the output score and the true target score, thereby outputting quality scores that meet human expectations. However, this paradigm places extremely stringent requirements on data quality, heavily relying on manually pre-annotated point-level absolute scores. In practice, it is extremely difficult to standardize the evaluation criteria for absolute scores, with significant subjective differences among annotators, making the annotation difficulty far greater than simple relative preference comparisons. Furthermore, manually annotated scores are often discrete, which significantly limits the continuity and fine granularity of the scorer's output scores.

[0004] In conclusion, effectively utilizing human preference data, which has lower annotation costs, to train a generative quality scorer with CoT (Co-thinking) capabilities has become a core issue that urgently needs to be addressed. This will not only provide precise optimization directions for the further evolution of generative models, but also bring crucial interpretability to the scoring process. Summary of the Invention

[0005] The problem addressed by this invention is how to effectively utilize human preference data, which has lower annotation costs, to train a generative quality scorer with analytical capabilities.

[0006] To address the aforementioned problems, this invention provides a training method, apparatus, and storage medium for a content scoring generator based on preference data training.

[0007] In a first aspect, the present invention provides a training method for a content scoring generator based on preference data, comprising the following steps: Masked supervised fine-tuning stage: The multimodal large model is supervised and fine-tuned using training data containing thought chain reasoning trajectories and final scores; during the fine-tuning process, the thought chain reasoning trajectories of some training data are masked, and only the final scores are retained as supervision signals to force the model to establish an implicit mapping from the generated content input to the point-level score output. Reinforcement learning phase: A group relative strategy optimization algorithm is adopted to enable the model to generate batch outputs containing the thought chain reasoning process and point-level scores based on input prompts and generated content; a pairwise reward mechanism is constructed using human preference labels to calculate the relative reward between different outputs, and the model parameters are iteratively optimized in combination with the relative reward to obtain the generated content scorer.

[0008] Optionally, the mask supervision fine-tuning stage includes: Initial fine-tuning is performed on complete training data to make the thought chain reasoning trajectory and the final score visible to the model, so as to guide the model to establish a preliminary reasoning paradigm. In subsequent iterative fine-tuning, a masking mechanism is introduced, randomly selecting a preset proportion of training samples to mask their thought chain reasoning trajectories, and updating the gradient only based on the final score.

[0009] Optionally, the mask-supervised fine-tuning stage is executed through multiple rounds of iteration to inject domain knowledge related to human preferences into the model while suppressing the model's mechanical memorization of text sequences in the training data, thus preserving policy exploration space for the reinforcement learning stage.

[0010] Optionally, in the reinforcement learning phase, the pairwise reward mechanism is implemented in the following way: Extract a pair of generated content with human preference labels from the training set, and let the generated content scorer predict scores for the two as follows: and ; according to and The relative size is used to calculate the pairwise reward. This is to convert the human preference labels into supervisory signals for point-level absolute scores.

[0011] Optionally, the paired rewards Calculated using the following formula: ; Among them, when When the paired reward is a positive reward, In this case, the paired reward is a negative penalty.

[0012] Optionally, the reinforcement learning phase further includes: Calculate format compliance reward and quality inspection reward for each output in the batch output; The weighted sum of the pairwise rewards, format compliance rewards, and quality rewards is used to obtain the final reward for each output. The advantage value in the relative strategy optimization algorithm based on the final reward calculation group is used to update the model parameters.

[0013] Optionally, the final reward Calculated using the following formula: ; in, Rewards for format compliance As a reward for quality inspection, , , These are the weighting coefficients for the corresponding reward items.

[0014] Optionally, in the reinforcement learning phase, the group relative policy optimization algorithm updates the model policy by comparing the relative performance of different output trajectories under the same input, and introduces a KL divergence penalty term in the optimization objective to limit the deviation between the current policy and the reference policy.

[0015] Secondly, embodiments of the present invention provide a training apparatus for generating a content scorer, characterized in that it includes a memory and a processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the training method for the content generator trained based on preference data as described in the first aspect.

[0016] Thirdly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for a content scorer based on preference data training as described in the first aspect.

[0017] The beneficial effects of the training method for the content-generating scorer based on preference data training of the present invention are as follows: This invention uses a multimodal large-scale model as the base model and optimizes it into a high-precision generative quality scorer through a two-stage training paradigm of "supervised fine-tuning" and "reinforcement learning." Specifically: In the supervised fine-tuning stage, a masking mechanism is introduced for multiple rounds of fine-tuning, effectively preventing the model from falling into the catastrophic forgetting and overfitting trap of rote memorization of training set scores while injecting knowledge of human preference-related domains into the model; in the reinforcement learning stage, group-based relative policy optimization is introduced, prompting the model to spontaneously optimize its thinking logic in the exploration space, thereby outputting more accurate scalar scores. This invention innovatively constructs a pairwise reward calculation framework, which completely frees the model's optimization process from dependence on high-cost point-level absolute score data, and only requires efficient supervision of the reinforcement learning process based on human preference labels. Through the above two-stage training, a generative quality scorer that highly fits human preferences is finally obtained. This scorer can not only accurately quantify and score the generated content, but also output a thought chain with strong interpretability, thereby pointing out the optimization direction for subsequent iterations of the generative model. Attached Figure Description

[0018] Figure 1 This is a flowchart of the training method for the content generation scorer based on preference data in an embodiment of the present invention; Figure 2 This is a schematic diagram of the training method for the content generation scorer based on preference data in an embodiment of the present invention. Figure 3 This is a schematic diagram of a reinforcement learning method for pairwise relative policy optimization in an embodiment of the present invention; Figure 4 This is a structural block diagram of the training device for generating a content scorer in an embodiment of the present invention. Detailed Implementation

[0019] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0021] like Figure 1 As shown in the embodiment of the present invention, a training method for a content scoring generator based on preference data includes the following steps: S1, Masked Supervised Fine-Tuning Stage: Using training data containing thought chain reasoning trajectories and final scores, supervised fine-tuning of the multimodal large model is performed; during the fine-tuning process, the thought chain reasoning trajectories of some training data are masked, and only the final scores are retained as supervision signals to force the model to establish an implicit mapping from the input of generated content to the output of point-level scores. The mask supervision and fine-tuning phase includes: Initial fine-tuning was performed on complete training data to make the thought chain reasoning trajectory and final score visible to the model, in order to guide the model to establish a preliminary reasoning paradigm. In subsequent iterative fine-tuning, a masking mechanism is introduced, randomly selecting a preset proportion of training samples to mask their thought chain reasoning trajectories, and updating the gradient only based on the final score.

[0022] The mask-supervised fine-tuning stage is executed through multiple rounds of iteration. It is used to inject domain knowledge related to human preferences into the model while suppressing the model's mechanical memorization of text sequences in the training data, thus preserving policy exploration space for the reinforcement learning stage.

[0023] During supervised fine-tuning, the reasoning trajectory of the thought chain is selectively masked, so that some training data participates in supervised fine-tuning only with the final answer. This mechanism forces the model to establish an implicit mapping relationship from the generated content input to the decision output, guiding it to internalize the thinking framework rather than mechanically reconstructing the text data in the training set. Through a multi-round fine-tuning strategy, the model can both deeply learn domain knowledge corresponding to human preferences and effectively prevent overfitting, thus ensuring that the model maintains sufficient exploratory activity in the second round of reinforcement learning.

[0024] Furthermore, during the fine-tuning phase, the thought process of some data is selectively masked, retaining only the final decision answer for training. This mechanism forces the model to internalize the implicit mapping logic from multimodal inputs to the final output, avoiding rote memorization of training text. Through multiple rounds of masking fine-tuning, the model effectively suppresses overfitting while deeply absorbing human preference knowledge, thus preserving ample policy exploration space for subsequent reinforcement learning stages.

[0025] Specifically, in this embodiment, the multimodal large-scale model Qwen2.5-VL-7B, which possesses strong cross-modal understanding capabilities, is selected as the base model. Supervised fine-tuning (SFT) is first performed on it to inject domain prior knowledge related to human preferences. In this stage, the model learns how to analyze generated content, mimic human evaluation logic, and ultimately output accurate point-level scores. However, too many fine-tuning rounds can easily lead to overfitting—that is, the model tends to mechanically imitate text sequences in the training set, thus losing its ability for autonomous exploration and deep thinking. This severely compresses the search space in subsequent reinforcement learning stages and weakens the diversity of inference paths. Conversely, if the number of training steps is limited, the multimodal large-scale model may not learn enough domain knowledge.

[0026] To balance this contradiction, this invention introduces a masked supervised fine-tuning strategy. Specifically, the model is first fine-tuned on complete data, during which the thought process and the final score are visible to the model, guiding it to establish a preliminary reasoning paradigm and learn how to assess the fit between generated quality and human preferences. Subsequently, in subsequent iterative training, the thought process is masked, so that the model can only observe the final score during optimization. This strategy effectively reduces the model's over-reliance on pre-set reasoning paths while ensuring scoring accuracy.

[0027] S2, Reinforcement Learning Phase: A group relative policy optimization algorithm is adopted to enable the model to generate batch outputs containing thought chain reasoning processes and point-level scores based on input prompts and generated content; a pairwise reward mechanism is constructed using human preference labels to calculate the relative rewards between different outputs, and the model parameters are iteratively optimized in combination with the relative rewards to obtain a generated content scorer.

[0028] The mask supervision and fine-tuning phase includes: Initial fine-tuning was performed on complete training data to make the thought chain reasoning trajectory and final score visible to the model, in order to guide the model to establish a preliminary reasoning paradigm. In subsequent iterative fine-tuning, a masking mechanism is introduced, randomly selecting a preset proportion of training samples to mask their thought chain reasoning trajectories, and updating the gradient only based on the final score.

[0029] The mask-supervised fine-tuning stage is executed through multiple rounds of iteration. It is used to inject domain knowledge related to human preferences into the model while suppressing the model's mechanical memorization of text sequences in the training data, thus preserving policy exploration space for the reinforcement learning stage.

[0030] During the reinforcement learning phase, the batch outputs generated by the model are paired and a preference reward is calculated for each pair. This method cleverly utilizes human preference labels in the dataset to directly and inversely optimize the accuracy of the point-level scores output by the multimodal large model. Simultaneously, by providing quality control rewards and format compliance rewards for individual data points, the generation quality of the thought chain is continuously improved, further enhancing the accuracy and reliability of the scorer.

[0031] Specifically, to further enhance the model's logical reasoning ability and improve its scoring accuracy, this invention employs the Group Relative Policy Optimization (GRPO) algorithm, which iterates strategies by comparing the relative performance of different reasoning trajectories. Specifically, given text prompts and generated content, the training objective is to fine-tune the multimodal large model so that before outputting point-level scores between 1 and 5, it performs fine-grained analysis of the generated content and generates corresponding thought chain reasoning processes. The standard GRPO algorithm is derived from older scorer models. Based on input Generate a set of outputs Where n is the number of outputs generated. The dominance value of the i-th response is calculated by normalizing the reward of this set of responses. GRPO updates the scorer by truncating the objective function. We also add a KL penalty term. However, due to the lack of absolute true point-level scores in the dataset, we cannot directly estimate the GRPO advantage value by calculating the "error between the predicted score and the true score". To overcome this challenge, we propose a pairwise reward calculation mechanism: we combine the model's paired outputs using the native human preference labels in the dataset, and calculate rewards based on relative preferences and allocate them to the corresponding responses.

[0032] Extract a pair of generated content with human preference labels from the training set, and let the scorer predict scores for the two as follows: and Then, a pair of rewards will be given. The calculation logic is as follows: when Positive rewards are given when positive preferences are met, and negative penalties are given when they are not, thereby transforming relative preference labels into supervisory signals for point-level absolute scores.

[0033] For example, suppose a pair of generated content is extracted from the training set. The corresponding scorer output is: and The corresponding paired reward is: .

[0034] It should be noted that the reinforcement learning phase also includes: Calculate format compliance rewards and quality inspection rewards for each output in the batch output; The weighted sum of the pairwise rewards, format compliance rewards, and quality rewards is used to obtain the final reward for each output. The advantage value in the relative policy optimization algorithm is calculated based on the final reward and used to update the model parameters.

[0035] Specifically, to ensure the quality of the generated thought processes and standardize the output format, we have introduced an additional quality control reward system. and format compliance rewards .

[0036] In this embodiment, the reward for each output is: ; in, Rewards for format compliance As a reward for quality inspection, , , These are the weighting coefficients for the corresponding reward items.

[0037] During the reinforcement learning phase, the group relative policy optimization algorithm updates the model policy by comparing the relative performance of different output trajectories under the same input, and introduces a KL divergence penalty term into the optimization objective to limit the deviation between the current policy and the reference policy.

[0038] The final optimization process is described in the following formula: ; .

[0039] in, This represents the mean. Represents variance. This indicates the number of trajectories generated by a group. This indicates the length of the currently output token. Indicates the first The first output One token, Indicates the first The first output of each line One token, Control the cutoff range of the importance sampling ratio. This is the current strategy. Deviation from reference strategy The severity of the punishment.

[0040] To demonstrate the effectiveness of this invention, the following experiments were conducted for verification.

[0041] This experiment assesses the quality preference of paired generated images. The constructed generated content scorer is applied to a static generated image scenario to verify its accuracy in evaluating the quality of generated images. Specifically, the experiment requires the scorer to accurately identify and output the winning sample (which better aligns with human visual preferences) and the losing sample (which has lower quality) from a given pair of generated images.

[0042] Table 1. Performance comparison of generated image quality.

[0043] Whether targeting winning or losing samples, our model achieved the highest F1 score (0.531 and 0.671 respectively). Particularly in winning sample recognition, compared to the extreme biases of other models, our method achieved a robust balance between recall (0.462) and precision (0.624). This fully validates the superiority of the Pairwise Relative Policy Optimization (GRPO) reinforcement learning mechanism introduced in the second stage. By comparing the relative performance of different output trajectories, the model successfully overcame bias towards specific ratings and learned to make accurate decisions based on fine-grained thought chains (CoT) that truly align with human subjective preferences.

[0044] This experiment assesses the quality of generated videos based on quality preferences. The constructed generated content scorer is applied to a video generation scenario to verify its accuracy in evaluating the quality of generated videos. Table 2 Performance Comparison of Generated Video Quality

[0045] The model of this invention maintains highly competitive performance on general benchmarks. In the GenAI benchmark test, the model of this invention successfully outperformed other models that focus on single-frame visual understanding, and even slightly outperformed VideoScore2 and Qinsight on VideoGen. This indicates that the two-stage training strategy adopted in this invention not only endows the model with a keen sense of flaws, but also avoids catastrophic forgetting of general video quality perception capabilities.

[0046] like Figure 4 As shown in the figure, an embodiment of the present invention provides a training device for generating content scorers, comprising: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following methods: Masked supervised fine-tuning stage: The multimodal large model is supervised and fine-tuned using training data containing thought chain reasoning trajectories and final scores; during the fine-tuning process, the thought chain reasoning trajectories of some training data are masked, and only the final scores are retained as supervision signals to force the model to establish an implicit mapping from the generated content input to the point-level score output. Reinforcement learning phase: A group relative strategy optimization algorithm is adopted to enable the model to generate batch outputs containing the thought chain reasoning process and point-level scores based on input prompts and generated content; a pairwise reward mechanism is constructed using human preference labels to calculate the relative reward between different outputs, and the model parameters are iteratively optimized in combination with the relative reward to obtain the generated content scorer.

[0047] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0049] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a content-generating scorer based on preference data, characterized in that, Includes the following steps: Masked supervised fine-tuning stage: The multimodal large model is supervised and fine-tuned using training data containing thought chain reasoning trajectories and final scores; during the fine-tuning process, the thought chain reasoning trajectories of some training data are masked, and only the final scores are retained as supervision signals to force the model to establish an implicit mapping from the generated content input to the point-level score output. Reinforcement learning phase: A group relative strategy optimization algorithm is adopted to enable the model to generate batch outputs containing the thought chain reasoning process and point-level scores based on input prompts and generated content; a pairwise reward mechanism is constructed using human preference labels to calculate the relative reward between different outputs, and the model parameters are iteratively optimized in combination with the relative reward to obtain the generated content scorer.

2. The training method according to claim 1, characterized in that, The mask supervision fine-tuning phase includes: Initial fine-tuning is performed on complete training data to make the thought chain reasoning trajectory and the final score visible to the model, so as to guide the model to establish a preliminary reasoning paradigm. In subsequent iterative fine-tuning, a masking mechanism is introduced, randomly selecting a preset proportion of training samples to mask their thought chain reasoning trajectories, and updating the gradient only based on the final score.

3. The training method according to claim 1, characterized in that, The mask-supervised fine-tuning stage is executed through multiple rounds of iteration. It is used to inject domain knowledge related to human preferences into the model while suppressing the model's mechanical memorization of text sequences in the training data, thus preserving policy exploration space for the reinforcement learning stage.

4. The training method according to claim 1, characterized in that, During the reinforcement learning phase, the pairwise reward mechanism is implemented in the following way: Extract a pair of generated content with human preference labels from the training set, and let the generated content scorer predict scores for the two as follows: and ; according to and The relative size is used to calculate the pairwise reward. This is to convert the human preference labels into supervisory signals for point-level absolute scores.

5. The training method according to claim 4, characterized in that, The paired rewards Calculated using the following formula: ； Among them, when At that time, the paired reward is a positive reward. In this case, the paired reward is a negative penalty.

6. The training method according to claim 4, characterized in that, The reinforcement learning phase also includes: Calculate format compliance reward and quality inspection reward for each output in the batch output; The weighted sum of the pairwise rewards, format compliance rewards, and quality rewards is used to obtain the final reward for each output. The advantage value in the relative strategy optimization algorithm based on the final reward calculation group is used to update the model parameters.

7. The training method according to claim 6, characterized in that, The final reward Calculated using the following formula: ； in, Rewards for format compliance As a reward for quality inspection, , , These are the weighting coefficients for the corresponding reward items.

8. The training method according to claim 1, characterized in that, In the reinforcement learning phase, the group relative policy optimization algorithm updates the model policy by comparing the relative performance of different output trajectories under the same input, and introduces a KL divergence penalty term in the optimization objective to limit the deviation between the current policy and the reference policy.

9. A training device for a content scoring generator, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the training method for the content generator based on preference data training as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the training method for a content scorer based on preference data training as described in any one of claims 1 to 8.

Images