Model training method, data generation method, device, and storage medium

Abstract

The present disclosure provides a model training method, a data generation method, an apparatus and a storage medium, comprising: obtaining to-be-processed data, performing question and answer processing on the to-be-processed data, and determining a training set; wherein the training set at least includes one of the following: a plurality of questions, a plurality of answers corresponding to each question; for each answer, the reward model is used to judge the answer to obtain a plurality of judgment results; and the reference label corresponding to the answer is determined according to the judgment result; the reward value of each judgment result is determined according to the reference label; and the reward model is trained based on the reward value and the training set corresponding to the reward value.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, a data generation method, an apparatus, and a storage medium. Background Technology

[0002] The core operating logic of embodied intelligent devices lies in accurately perceiving and understanding multi-dimensional information such as visual and temporal information in multimodal videos, and combining this with environmental interaction needs to output decision-making instructions and action responses that conform to human expectations. The construction and optimization of the reward model is a key link that determines the performance of embodied intelligent devices and achieves precise alignment between the output of embodied intelligent devices and human preferences. Furthermore, the performance of the reward model directly affects the adaptability of embodied intelligent devices to complex scenarios, the accuracy of reasoning and judgment, and the effectiveness of actual industrial deployment.

[0003] However, in traditional reward model training methods, it is common to rely on preset fixed question-answer pairs, with corresponding standard reference responses written manually. Automated scripts are used to calculate the degree of matching between the output of the embodied intelligent device and the preset standard responses, thereby generating corresponding reward signals to provide supervision for the training of the reward model.

[0004] While this traditional approach automates reward signal generation to some extent and reduces the cost of real-time human intervention, its inherent reliance on a single, pre-defined standard response inherently makes it ill-suited to the practical application scenarios of multimodal video understanding and embodied intelligence. Particularly noteworthy is its strong dependence on standard responses, which renders it unsuitable for open-ended question-answering tasks. In real-world video understanding scenarios using embodied intelligence devices, reasonable responses to the same multimodal video content often exhibit diverse characteristics, such as different expressions and reasoning paths. A single standard response cannot comprehensively cover all reasonable semantic expressions. This limits the applicability of the reward model in open-domain tasks, resulting in a lack of flexibility and comprehensiveness in reward signal generation, failing to accurately match real-world application scenarios, thus impacting the training effect of the reward model and ultimately hindering the performance improvement of embodied intelligence devices. Summary of the Invention

[0005] This disclosure provides a model training method, data generation method, apparatus, and storage medium to address the problem that existing reward model training relies on a single standard response, which cannot adapt to the diversity of responses and open-ended question-and-answer requirements, resulting in inflexible reward signals and seriously affecting model training and the performance of embodied intelligent devices.

[0006] In view of the above problems, firstly, the present disclosure provides a model training method, including: Acquire the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, and multiple responses corresponding to each question; For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results; and a baseline label corresponding to the response is determined based on the judgment results. Based on the aforementioned benchmark labels, determine the reward value for each judgment outcome; The reward model is trained based on the reward value and the corresponding training set.

[0007] In conjunction with the first aspect, in one possible implementation, training the reward model based on the reward value and the corresponding training set includes: Based on the reward value, responses are filtered according to a preset strategy to determine the target responses after filtering; The first output data corresponding to the target response is used as the model update sample, and the reward model is updated through a reinforcement learning algorithm. The first output data includes at least one of the following: the target response, the reward value of the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0008] In conjunction with the first aspect, in one possible implementation, the type of the reward value includes a first type and a second type; The step of filtering responses according to the reward value and a preset strategy to determine the filtered target responses includes: For each response, the reward value of the first type is assigned to the first reward group and the reward value of the second type is assigned to the second reward group; Calculate the absolute value of the difference between the sum of the reward values ​​of the first reward group and the sum of the reward values ​​of the second reward group; If the absolute value of the difference is less than a first preset threshold, the response is determined as the target response after filtering.

[0009] In conjunction with the first aspect, in one possible implementation, the method further includes: If the absolute value of the difference is greater than or equal to a first preset threshold, the response and the reward value of the corresponding judgment result are deleted.

[0010] In conjunction with the first aspect, in one possible implementation, determining the baseline label corresponding to the response based on the decision result includes: For each response, the frequency of each type of decision result is counted among all the decision results corresponding to that response, and the decision result with the highest frequency is determined as the baseline label for that response.

[0011] In conjunction with the first aspect, in one possible implementation, for each response, a decision is made using a reward model to obtain multiple decision results, including: The data to be processed and the training set are input into the reward model. For each response, multiple independent judgments are made on the response to obtain the judgment result and the corresponding judgment reason for each judgment. The judgment result is a binary classification type, including two mutually exclusive judgment states.

[0012] In conjunction with the first aspect, in one possible implementation, determining the reward value for each judgment result based on the benchmark label includes: Each judgment result is matched with the corresponding response's baseline label, and a corresponding reward value is configured for each judgment result based on the matching result.

[0013] In conjunction with the first aspect, in one possible implementation, after determining the baseline label corresponding to each response, the method further includes: Based on the aforementioned benchmark labels, construct benchmark label annotation samples; Based on the baseline labeled samples, the base model of the reward model is subjected to supervised fine-tuning. The benchmark label-annotated samples include: benchmark labels, judgment results corresponding to the benchmark labels, judgment reasons corresponding to the judgment results, and the training set; The training set also includes the reasons for generating each response.

[0014] In conjunction with the first aspect, in one possible implementation, the data to be processed includes video data; The process of acquiring data to be processed, performing question and response processing on the data to be processed, and determining the training set includes: Acquire video data; The video data is questioned using a questioning model to generate multiple initial questions. A variance incentive mechanism is then used to filter the initial questions to obtain multiple filtered questions. For each question, based on the video data, multiple response models are used to process the question, resulting in multiple responses and the reasons for generating each response; The answer model includes the reward model.

[0015] Secondly, a data generation method is provided, including: Acquire the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results and corresponding judgment reasons; and a baseline label is determined for the response based on the judgment results. Based on the aforementioned benchmark labels, determine the reward value for each judgment outcome; Based on the reward value, responses are filtered according to a preset strategy to determine the target responses after filtering; Based on the target response, second output data is determined; the second output data includes at least one of the following: the target response, the benchmark label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0016] Thirdly, a model training device is provided, comprising: The first training set construction module is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, and multiple responses corresponding to each question; The first decision module is used to make a decision on each response using a reward model, obtaining multiple decision results; and to determine the baseline label corresponding to the response based on the decision results. The first reward value determination module is used to determine the reward value for each judgment result based on the benchmark label; The first update and output module is used to train the reward model based on the reward value and the training set corresponding to the reward value.

[0017] Fourthly, a data generation apparatus is provided, comprising: The second training set construction module is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; The second decision module is used to make a decision on each response using a reward model, obtaining multiple decision results and corresponding decision reasons; and determining the benchmark label corresponding to the response based on the decision results. The second reward value determination module is used to determine the reward value for each judgment result based on the benchmark label; The second update and output module is used to filter responses according to the reward value and a preset strategy to determine the filtered target response; and to determine the second output data according to the target response; the second output data includes at least one of the following: the target response, the benchmark label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0018] Fifthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of a model training method as described in the first aspect or in any possible embodiment of the first aspect; and / or the steps of a data generation method as described in the second aspect.

[0019] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a model training method, data generation method, apparatus, and storage medium, comprising: acquiring data to be processed; performing question and response processing on the data to be processed; and determining a training set; wherein the training set includes at least one of the following: multiple questions and multiple responses corresponding to each question; for each response, making a judgment on the response through a reward model to obtain multiple judgment results; and determining a benchmark label corresponding to the response based on the judgment results; determining a reward value for each judgment result based on the benchmark label; and training the reward model based on the reward value and the corresponding training set. The model training method provided by this disclosure breaks away from the traditional reliance on a single standard response, allowing the training set to support multiple reasonable responses to the same question; the benchmark label is autonomously determined based on the response judgment results, without manual pre-setting, adapting to the "reasonable response diversity" characteristics of multimodal video understanding and embodied intelligence scenarios, and solving the problem of adapting open-ended question answering. The reward value accurately reflects the accuracy of the judgment, adapting to the core tasks of embodied intelligence devices; the entire training process requires no manual intervention, allowing for flexible adjustment of data to adapt to different embodied intelligence devices, avoiding data privacy and intellectual property risks, adapting to sensitive scenarios, and facilitating large-scale promotion. Attached Figure Description

[0020] Figure 1 One of the flowcharts for the model training method provided in the embodiments of this disclosure; Figure 2 The second flowchart of the model training method provided in the embodiments of this disclosure; Figure 3 A flowchart of the model data generation method provided in the embodiments of this disclosure; Figure 4 A structural diagram of the training apparatus for the model provided in the embodiments of this disclosure; Figure 5This is a structural diagram of the data generation apparatus provided in an embodiment of the present disclosure. Detailed Implementation

[0021] This disclosure provides a model training method, a data generation method, an apparatus, and a storage medium. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit this disclosure. Furthermore, the embodiments and features described in this application can be combined with each other unless otherwise specified.

[0022] This disclosure provides a model training method, such as... Figure 1 As shown, it includes: S101. Obtain the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question; S102. For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results; and the baseline label corresponding to the response is determined based on the judgment results. S103. Determine the reward value for each judgment result based on the benchmark label; S104. Train the reward model based on the reward value and the corresponding training set.

[0023] In this embodiment of the disclosure, embodied intelligent devices refer to devices capable of perceiving the physical world, understanding the environment, and performing physical operations, such as humanoid robots. The core operating logic of embodied intelligent devices lies in accurately perceiving and understanding multi-dimensional information such as visual and temporal information in multimodal videos, and combining this with environmental interaction needs to output expected decision-making instructions and action responses. The construction and optimization of the reward model is a crucial step in determining the performance of embodied intelligent devices and achieving precise alignment between the output of embodied intelligent devices and human preferences. It directly affects the adaptability of embodied intelligent devices to complex scenarios, the accuracy of reasoning and judgment, and the effectiveness of actual industrial deployment.

[0024] However, traditional reward model training methods typically rely on pre-set standard responses. Automated scripts calculate the matching degree between the embodied intelligent device's output and the pre-set standard responses to generate corresponding reward signals, providing supervision for the training of the reward model. This approach has a fixed evaluation process: fixed question-answer pairs are pre-defined, corresponding standard responses are manually written, and then automated scripts quantify the similarity between the embodied intelligent device's output and the standard responses. This similarity score is then used as the core criterion for reward signal judgment. However, this strong dependence on standard responses makes it unsuitable for open-ended question-answering tasks. In real-world video understanding scenarios with embodied intelligent devices, reasonable responses to the same multimodal video content often exhibit diverse characteristics. For example, different expressions and reasoning paths may all conform to natural semantic logic and practical application needs, while a single standard response cannot comprehensively cover all reasonable semantic expressions. This limits the applicability of reward models in open-domain tasks, resulting in a lack of flexibility and comprehensiveness in reward signal generation. It fails to accurately match real-world human preferences, thus affecting the training effect of the reward model and ultimately hindering the performance improvement of embodied intelligent devices.

[0025] Another automatic evaluation method based on regular expression matching predefines specific regular expression templates to perform structured parsing and keyword matching on responses. The matching results are used as the basis for judging the correctness of the response, thereby generating a reward signal. This method has the advantages of low implementation cost and fast evaluation speed, but its application scenarios are severely limited. It is mainly suitable for simple tasks such as multiple-choice and fill-in-the-blank questions with fixed response formats, and cannot handle complex semantic and reasoning tasks. Regular expressions are essentially shallow matching based on string patterns, lacking the ability to understand the deep semantics of natural language and capture multimodal information associations. Furthermore, they cannot handle complex cognitive tasks commonly encountered in multimodal video understanding and embodied intelligent device applications, such as cross-temporal causal reasoning, action intent prediction, and environmental interaction decision-making. In these complex tasks, they cannot effectively evaluate the rationality and accuracy of the output results of embodied intelligent devices, causing the generated reward signal to lose its supervised meaning and failing to support the effective training of the reward model.

[0026] Another referee labeling method relies on strongly closed-source models (such as GPT-5). This method uses a high-performance, strongly closed-source large model as the "referee," leveraging its powerful semantic understanding and evaluation capabilities to judge responses and generate True / False labels, which serve as supervision signals for reward model training. While this method demonstrates good evaluation capabilities in complex tasks such as open-ended question answering and addresses the limitations of pre-defined standard responses to some extent, it suffers from both cost and data security risks, hindering large-scale application. Reliance on strongly closed-source models leads to high costs and data security risks. API calls incur significant fees, which increase linearly with the scale of training data for the reward model, significantly increasing the cost of technology development and application. Furthermore, the multimodal video data and model output data involved in training need to be uploaded to third-party closed-source model servers, posing potential risks of data privacy leaks and intellectual property infringement. For embodied intelligent devices in sensitive scenarios (such as medical, military, and core industrial scenarios), such data security risks are unacceptable, severely limiting the applicability of this method.

[0027] Another method, Reinforcement Learning from Human Feedback (RLHF), uses a large language model to align human preferences. This requires a large number of human annotators to perform pairwise comparisons of multiple sets of results output by embodied intelligent devices, labeling results that conform to human preferences to form a human preference dataset. The reward model is then trained using methods such as Direct Preference Optimization (DPO). This method can capture human preferences well and improve the rationality of the reward model's evaluation. However, the high cost and low scalability of manual annotation are significant problems, severely limiting the training efficiency and large-scale application of the reward model. The cost of manual annotation increases linearly with the scale of the labeled data, and the annotation quality is affected by various factors such as the subjective cognitive level, professional ability, and focus of the human annotators, making it difficult to guarantee the consistency and accuracy of the labeled data. As the scale of annotation increases, the marginal utility gradually decreases, further exacerbating the cost pressure of this method and limiting the efficiency and scalability of reward model training.

[0028] In this embodiment, data to be processed is acquired, and a training set is constructed by questioning and responding to the data. A reward model then judges the responses in the training set and determines a baseline label. Based on the baseline label, a reward value is calculated for each judgment result. Finally, the reward value and the corresponding training set data are used to train the reward model, thereby improving its performance. The data to be processed can be video data originating from the working environment of the embodied intelligent device. Questioning and responding models are used to process the data. The questioning model asks questions about the scenarios and information included in the data, generating multiple questions that fit the actual needs of the embodied intelligent device. The responding model responds to each question, generating multiple responses. These questions and responses are automatically generated by the questioning and responding models, without the need for a single pre-defined standard response, allowing multiple response formats to coexist. Finally, the questions and corresponding responses are integrated to form a training set. The training set includes at least multiple questions and multiple responses for each question, ensuring that the data covers the diverse needs of the embodied intelligent open question-and-answer scenario. The reward model evaluates response quality, generates judgment results, and determines benchmark labels for measuring response quality, providing a standard for subsequent reward value calculation. This replaces traditional manual labeling or closed-source model referee labeling, reducing costs and ensuring objectivity. For each response to each question in the training set, the response is input into the reward model, which evaluates the quality of the response (e.g., whether it meets the scenario requirements, aligns with human preferences, and whether the reasoning is reasonable) and outputs the corresponding judgment result. The reward model can make multiple judgments on a single response, with each response corresponding to multiple judgment results. Based on all judgment results for the same response, statistical analysis, such as mode voting and weighted statistics, is used to determine the benchmark label corresponding to that response, i.e., the standard judgment result for that response, used to subsequently measure the accuracy of the reward model's judgment results.

[0029] Furthermore, the degree of fit between the benchmark label and the decision result is converted into a quantified reward value, generating the supervision signal required for training the reward model. Using the benchmark label as the standard, each decision result is compared with the corresponding benchmark label: if the decision result perfectly matches the benchmark label (e.g., both the decision result and the benchmark label are True), a higher reward value is assigned to that decision result, indicating that the reward model's decision was accurate; if the decision result does not match the benchmark label, a lower reward value is assigned based on the degree of fit, indicating that the reward model's decision was biased. Finally, a corresponding reward value is assigned to each decision result, forming a correspondence between decision results and reward values, providing a quantified supervision basis for the training of the reward model. Using the quantified reward value as the supervision signal, combined with the "question-response" data in the training set, the evaluation accuracy of the reward model is iteratively optimized, gradually adapting it to the needs of multimodal video understanding and embodied intelligent device application scenarios. The reward value, along with the corresponding training data (i.e., the response and question corresponding to that reward value), is input into the reward model to be trained. Adaptive training methods, such as reinforcement learning, are used to drive the reward model to learn the correlation between response quality, judgment results, and reward values. It learns how to accurately judge response quality and gradually optimizes the reward model parameters. Through a preset number of iterative training rounds, most judgment results of the reward model are made to conform to the baseline label, and the overall reward value tends to be optimized. Ultimately, the accuracy of the reward model's judgment is improved, enabling it to accurately adapt to open scenarios of multimodal video understanding and embodied intelligence without relying on manual annotation or closed-source models.

[0030] This application breaks away from the traditional reliance on a single standard response. The training set allows for multiple reasonable responses to the same question. The benchmark label is autonomously determined based on the decision result of the response, rather than being manually preset. This perfectly adapts to the characteristic of "diverse reasonable responses" in multimodal video understanding and embodied intelligent device application scenarios, solving the deficiency of being unable to adapt to open-ended question-answering tasks. The reward value can accurately reflect the accuracy of the reward model's decision, adapting to core embodied intelligence tasks such as complex semantics and temporal reasoning. The training process requires no manual intervention and can flexibly adjust the content of the data to be processed and the training set according to changes in the working environment of embodied intelligent devices, adapting to the training needs of different types of embodied intelligent devices. It also avoids the risks of data privacy leakage and intellectual property infringement, adapting to the application needs of embodied intelligent devices in sensitive scenarios, and facilitating large-scale promotion and application.

[0031] In another embodiment of this disclosure, the reward model is trained based on the reward value and the corresponding training set, including: Step 1: Based on the reward value, filter the responses according to the preset strategy to determine the target responses after filtering; Step 2: Use the first output data corresponding to the target response as the model update sample, and update the reward model using a reinforcement learning algorithm; The first output data includes at least one of the following: the target response, the reward value of the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0032] In this embodiment of the disclosure, such as Figure 2As shown, based on the reward value 207, high-quality target responses 208 are selected according to a preset strategy. Then, the multi-dimensional first output data associated with the target responses is used as model update samples, and the reward model is updated using a reinforcement learning algorithm 204. For step 1 above, high-quality target responses with reward values ​​meeting the requirements are selected from the massive responses in the training set, and low-quality, worthless response data is eliminated, reducing ineffective training costs, improving the efficiency and accuracy of the reward model update, and adapting to the needs of video data question-and-answer scenarios in the working environment of embodied intelligent devices. For each response, based on the reward value of each judgment result, a preset strategy is invoked to eliminate responses that do not meet the requirements, and the responses that meet the requirements are determined as target responses. For example, the preset strategy can be threshold filtering, selecting responses with a total reward value ≥ the reward filtering threshold as target responses; the preset strategy can be a reward hacking behavior intervention mechanism: judging by the absolute value difference between the two types of reward values, eliminating biased responses to obtain target responses; the preset strategy can also be ranking filtering: selecting the top N responses with the highest total reward values ​​as target responses. For step 2 above, extract the first output data corresponding to all target responses. Use this first output data as model update samples and update the reward model using a reinforcement learning algorithm. The first output data includes at least one of the following: the target response, the reward value of the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response. The judgment reason corresponding to the judgment result can refer to the basis for the reward model to give the judgment result to the response, such as "the response did not accurately match the question context". The generation reason corresponding to the target response can refer to the logical basis for the answer model to generate the response to the question, such as "derived based on the video temporal information in the data to be processed". The reinforcement learning algorithm can be the Group Relative Policy Optimization (GRPO) algorithm. Call the GRPO reinforcement learning algorithm and input the model update samples into the reward model to be trained. The GRPO algorithm will group the samples according to the response type / question scenario, optimize the judgment strategy of each group based on the reward value feedback, avoid the bias caused by a single strategy; optimize the parameters of the reward model during the algorithm iteration process, so that the judgment result of the reward model is more in line with the actual needs, and gradually improve the judgment accuracy. The first output data includes contextual dimensions such as the judgment reasoning corresponding to the judgment result and the generation reasoning corresponding to the target response. Compared with single response data, this allows the reward model to learn the "logical rationality" of the response, rather than relying solely on surface matching. The grouping optimization strategy of the GRPO algorithm further improves the adaptability of the reward model to different scenarios and reduces judgment bias. Pre-set strategies filter out low-quality and abnormal responses, using only high-quality target responses to train the model, reducing invalid computation, shortening the iteration cycle of the reward model, and reducing time and resource costs.

[0033] In another embodiment of this disclosure, the reward value includes a first type and a second type; In step 1 above, responses are filtered according to a preset strategy based on the reward value to determine the target responses after filtering, including: Step 1: For each response, assign the reward value of the first type to the first reward group and the reward value of the second type to the second reward group; Step 2: Calculate the absolute value of the difference between the sum of the reward values ​​of the first reward group and the sum of the reward values ​​of the second reward group; Step 3: If the absolute value of the difference is less than the first preset threshold, the response is determined as the target response after screening.

[0034] In this embodiment of the disclosure, reward hacking is a biased anomaly that occurs during the training of a reward model. To obtain a higher reward, the reward model ignores the input context (such as information related to the data to be processed) and fixes the output to either positive or negative labels, leading to distorted judgment results and decreased training effectiveness. Figure 2As shown, based on the reward value 207, the response is filtered through a reward-based hacking intervention mechanism to obtain the filtered target response 208. Based on the first and second types of reward values, target response filtering is completed through grouping, summation, difference calculation, and threshold comparison, achieving accurate detection and intervention of Reward Hacking behavior. The reward value types include first and second types. Each judgment result is matched with the corresponding response's baseline label, and a corresponding reward value is configured for each judgment result based on the matching result. For example, judgment results that match the corresponding response's baseline label are configured as first-type reward values, such as a reward value of 1; judgment results that do not match the corresponding response's baseline label are configured as second-type reward values, such as a reward value of 0. For step one above, for each response, all its corresponding reward values ​​are sorted out and categorized according to reward value type: all first-type reward values ​​corresponding to the response are uniformly grouped into the first reward group, for example, reward values ​​of 1 are grouped into the True bucket. All second-type reward values ​​corresponding to the response are uniformly grouped into the second reward group, for example, reward values ​​of 0 are grouped into the False bucket. For step two above, the summation of all reward values ​​in the first and second reward groups is performed to obtain the total reward value of the first reward group (True bucket reward sum) and the total reward value of the second reward group (False bucket reward sum). The difference between the two total reward values ​​is calculated, and the absolute value of this difference is taken. The magnitude of this absolute value directly reflects the degree of deviation between the two bucket reward values. The smaller the absolute value, the more balanced the rewards in the two buckets, and the less biased the reward model is in its decision-making regarding the response; the larger the absolute value, the greater the deviation between the two bucket reward values, and the higher the probability of the reward model exhibiting Reward Hacking behavior. For step three above, based on the Reward Hacking detection results, target response screening is completed. Stable and unbiased target responses are retained, while responses with abnormal bias and their corresponding training groups are removed, ensuring the accuracy of subsequent reward model training. If the absolute value of the difference is less than the first preset threshold, the reward evaluations of the first reward group (True bucket) and the second reward group (False bucket) tend to be balanced. The model's judgment on the response is not significantly biased, and there is no reward hacking behavior. The quality of this response is stable and the judgment is reliable. It is identified as the target response after screening and used for subsequent update training of the reward model. By using the bucket difference threshold mechanism to divide the reward value into True bucket and False bucket, and by comparing the absolute value of the difference with the threshold, the bias problem of the reward model fixed output positive or negative labels and ignoring context can be accurately identified. This triggers the intervention mechanism of discarding abnormal training groups, avoiding biased data from the source from affecting the judgment accuracy of the reward model, and ensuring the rationality and accuracy of the reward model training.

[0035] In another embodiment of this disclosure, the method further includes: If the absolute value of the difference is greater than or equal to the first preset threshold, delete the response and the reward value of the corresponding judgment result.

[0036] In this embodiment, if the absolute value of the difference is greater than or equal to a first preset threshold, the reward values ​​of the first reward group (True bucket) and the second reward group (False bucket) deviate too much, indicating that the model exhibits reward hacking behavior. This means that the reward model, in pursuit of high rewards, ignores contextual information and outputs fixed positive or negative labels. In this case, an intervention mechanism is triggered to discard the entire training group corresponding to that response, preventing such biased and anomalous data from entering subsequent reward model training, thus preventing training distortion and ensuring the training quality of the reward model.

[0037] In another embodiment of this disclosure, step S102 above, determining the baseline tag corresponding to the response based on the judgment result, includes: For each response, the frequency of each type of decision result is counted among all the decision results corresponding to that response, and the decision result with the highest frequency is determined as the baseline label for that response.

[0038] In this embodiment of the disclosure, such as Figure 2 As shown, based on multiple decision results 205 corresponding to the same response, a baseline label 206 for that response is determined through frequency statistics and mode voting, thus providing an objective and unified supervision standard for subsequent reward value calculation and model training. For each response, all decision results made by the reward model are first collected, and then different types of decision results are classified and counted to obtain the frequency of each type of decision result. After the statistics are completed, the decision result with the most frequent occurrences is selected as the final baseline label for that response, representing the consensus result of the majority of decisions. Using a majority voting mechanism to determine the baseline label can integrate the information of multiple decision results, reduce the impact of single decision errors, and improve label accuracy. The baseline label is closer to the true quality distribution, providing a more accurate supervision signal for subsequent reward value calculation, making the reward model learning more stable and the decisions more reasonable.

[0039] In another embodiment of this disclosure, in step S102 above, for each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results, including: The data to be processed and the training set are input into the reward model. For each response, multiple independent judgments are made on the response to obtain the judgment result and the corresponding judgment reason for each judgment. The judgment result is a binary classification type, including two mutually exclusive judgment states.

[0040] In this embodiment of the disclosure, such as Figure 2As shown, the data to be processed 201 and the training set are input into the reward model 204. Multiple independent decisions are made for each response, resulting in a binary classification decision 205 and corresponding decision reasons. This provides a reliable, diverse, and interpretable basis for subsequent baseline label determination. The data to be processed and the training set containing questions and responses are input into the current reward model 204, providing the reward model 204 with all the necessary context and data foundation for decision-making. The reward model performs multiple independent decision operations for each response, allowing it to output multiple decisions under different inference paths, improving the robustness and reliability of the results. The binary classification decision result and the corresponding decision reason are output. The decision result includes two mutually exclusive decision states, such as a positive label or a negative label. The decision reason is used to support the decision logic and improve training interpretability. Multiple independent decisions reduce the random error and fluctuation of a single decision, making the subsequently obtained baseline labels more stable and reliable. Each decision outputs a decision reason, making the reward model's decisions traceable and interpretable, which helps improve the rationality and reliability of the reward model training.

[0041] In another embodiment of this disclosure, step S103 above, determining the reward value for each judgment result based on the benchmark label, includes: Each judgment result is matched with the corresponding response's baseline label, and a corresponding reward value is configured for each judgment result based on the matching result.

[0042] In this embodiment of the disclosure, such as Figure 2 As shown, each decision result 205 output by the reward model 204 is compared and matched with the corresponding baseline label 206. A corresponding reward value 207 is assigned to each decision result based on the matching result, thus forming a standardized supervision signal that can be used for model training. For each independent decision of each response, the decision result is compared with the established baseline label for that response to determine whether the decision result is the same as the baseline label, forming a matching result. If the decision result is consistent with the baseline label, a positive or higher reward value is configured. If the decision result is inconsistent with the baseline label, a zero or lower reward value is configured. Using the baseline label as a unified standard ensures that the reward value allocation rules are consistent and fair, avoiding human intervention. The reward value directly reflects the accuracy of the decision, enabling the reward model to clearly learn the correct decision direction, improving convergence speed and training stability. The size and range of the reward value can be flexibly set to adapt to different training strategies and reinforcement learning algorithms.

[0043] In another embodiment of this disclosure, after determining the baseline tag corresponding to each response, the method further includes: Step 1) Construct baseline label annotation samples based on baseline labels; Step 2) Based on the baseline labels, perform supervised fine-tuning of the base model of the reward model; The baseline label-annotated samples include: baseline labels, corresponding judgment results for the baseline labels, judgment reasons corresponding to the judgment results, and training sets; The training set also includes the rationale for each response.

[0044] In this embodiment of the disclosure, such as Figure 2 As shown, a benchmark label annotation sample, including multi-dimensional information such as judgment reasons and generation reasons, is constructed based on the benchmark label 206. Supervised fine-tuning of the base model of the reward model 204 is then performed based on this benchmark label annotation sample, enabling the base model to learn stable and interpretable judgment logic and improving the accuracy, reliability, and generalization ability of the reward model 204. The determined benchmark labels and related information are integrated to form a complete benchmark label annotation sample that can be used for supervised fine-tuning. The benchmark labels are used as standard supervision signals, and the judgment results and judgment reasons corresponding to the benchmark labels are included in the sample. Training set data containing multiple questions, multiple responses for each question, and reasons for response generation are also included in the sample to obtain multi-dimensional benchmark label annotation samples, i.e., supervised fine-tuning data (SFT). The base model can refer to an artificial intelligence model with general capabilities that is pre-trained on large-scale data. The base model of the reward model can be the fundamental model on which the reward model is based, serving as the core carrier of the overall structure and parameters of the reward model. Supervised fine-tuning refers to the supervised training of a pedestal model using pre-labeled benchmark samples to optimize its parameters and improve its decision-making ability. The constructed benchmark samples are input into the pedestal model of the reward model, and parameters are optimized through supervised learning. During supervised fine-tuning, the pedestal model not only learns the mapping relationship between decision results and benchmark labels but also learns the decision logic and generation rationality from the decision reasons and generation reasons, making the model output more stable and interpretable. Benchmark samples contain multi-dimensional information, providing the pedestal model with complete context and logical basis, significantly improving the fine-tuning effect. Supervised fine-tuning of the pedestal model improves the reward model's understanding, reasoning, and decision-making abilities from the bottom up, making subsequent reward model training more stable and converging faster.

[0045] In another embodiment of this disclosure, the data to be processed includes video data; In step S101 above, the data to be processed is obtained, questions and responses are performed on the data to be processed, and the training set is determined, including: Step 1) Obtain video data; Step 2) Use the questioning model to ask questions about the video data to generate multiple initial questions. Use a variance incentive mechanism to filter the initial questions to obtain multiple filtered questions. Step 3) For each question, based on the video data, multiple response models are used to process the questions and obtain multiple responses and the reasons for generating each response; The response model includes a reward model.

[0046] In this embodiment of the disclosure, such as Figure 2As shown, video data is used as the data to be processed (201). First, the video data is acquired, and then initial questions are generated through the questioning model (202). High-quality questions are selected using a variance incentive mechanism. Finally, for each selected question, multiple response models (203) are used to generate multiple responses and reasons for each response, combined with the data to be processed (201). The questions, responses, and reasons for generation are then integrated to obtain the training set. Regarding step one above, according to the training requirements of the reward model, video data from the target scene is collected or acquired, such as the working environment of embodied intelligent devices or multimodal video understanding scenarios. This ensures that the video data is clear and complete, containing key information related to the training requirements, such as scene interaction, object movement, and environmental features. The acquired video data is preprocessed, such as format unification, noise reduction, and temporal alignment, to remove invalid information and ensure that the video data can be effectively identified and analyzed by the questioning and response models, laying the data foundation for subsequent steps. Regarding step two above, the preprocessed video data is input into the questioning model. The questioning model comprehensively analyzes the visual content, temporal changes, and scene logic of the video data to extract key information from the video. For example, based on object type, motion trajectory, and scene requirements, multiple initial questions closely related to the video content are generated. These initial questions must cover multi-dimensional information of the video, ensuring a sufficient number. The variance incentive mechanism is the core mechanism for filtering initial questions. By calculating the variance between initial question scores, it characterizes the diversity and differentiation of questions, incentivizing and retaining questions that fall within the middle range (ideal range), have high diversity, and cover multi-dimensional information of the video, while eliminating redundant, repetitive, or simplistic questions. The variance incentive mechanism filters the generated initial questions. For example, it evaluates the initial questions using a judging model, obtains initial question scores, and calculates the variance between all initial question scores. The middle range (ideal range) and extreme range of variance are pre-defined. Variance within the preset middle range indicates high discriminative power. Variance within the preset extreme range (maximum or minimum range) represents an extreme state, where extremely small variance and extremely high mean are judged as "superficial questions," and extremely low mean are judged as "unsolvable logical deadlock questions." The variance reward mechanism is essentially an automated filter that accurately eliminates "superficial questions" that all response models can answer correctly and "invalid questions" that all response models answer incorrectly, thus locking in highly discriminative questions that are at the cognitive boundary of the model. This results in a final set of filtered questions, ensuring both diversity and effectiveness. For each filtered question (as described in step three), multiple differentiated and reasonable responses and corresponding justifications are generated, enriching the diversity of the training set and providing interpretable contextual information for subsequent reward model decisions and training, ensuring the high quality of the training set. For each question, it is input along with the video data obtained in step one into multiple independent response models.Each response model, based on video data, reasons and analyzes the question, outputs a corresponding response, and generates a reason for that response. The reason for generation includes which information from the video the response is based on and through what logic it was generated. Because multiple response models are used, and each model has different reasoning logic and focus, multiple differentiated responses and corresponding reasons will be generated for the same question. Collecting all responses and their reasons from all response models ensures that each question corresponds to multiple responses, and each response has a clear reason for generation. Finally, the selected questions, the multiple responses for each question, and the reasons for each response are integrated to determine the complete training set. Response models can include reward models. In this case, the reward model simultaneously acts as both a "contestant" and a "referee." As a contestant, the reward model generates responses based on video data and the question; as a referee, the reward model performs multiple independent judgments on each response. This dual-role design significantly reduces system complexity and deployment costs, and mitigates bias problems caused by role confusion through frequency statistics and mode voting. The response model includes a reward model, which participates in the response generation process early in the training process. This allows the reward model to become familiar with the logical relationship between video data, questions, and responses in advance, making it easier to converge during subsequent decision-making and training, thus significantly improving the overall training efficiency.

[0047] Based on the same disclosed concept, this disclosure also provides a data generation method. Since the principle of the problem solved by this data generation method is similar to that of the aforementioned model training method, the implementation of this data generation method can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0048] This disclosure provides a data generation method, such as... Figure 3 As shown, it includes: S301. Obtain the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; S302. For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results and corresponding judgment reasons; and the baseline label corresponding to the response is determined based on the judgment results. S303. Determine the reward value for each judgment result based on the benchmark label; S304. Based on the reward value, the responses are filtered according to the preset strategy to determine the target responses after filtering; S305. Based on the target response, determine the second output data; the second output data shall include at least one of the following: the target response, the baseline label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0049] In this embodiment, data to be processed is acquired, and questions and answers are processed on it. The questions, answers, and reasons for generation are integrated to obtain a training set. The reward model makes multiple judgments on each answer, obtaining multiple judgment results and reasons. Based on the statistical analysis of the judgment results, a baseline label for the answer is determined. Each judgment result is matched with the baseline label of the corresponding answer, and a corresponding reward value is assigned to each judgment result based on the matching result. Based on the reward value, answers are filtered according to a preset strategy, selecting high-quality answers and removing invalid data to obtain target answers that meet the requirements. The data related to the target answer is integrated, and second output data is output. The second output data includes at least one of the following: the target answer, the baseline label corresponding to the target answer, the judgment result corresponding to the target answer, the reason for the judgment result, the question corresponding to the target answer, and the reason for generation corresponding to the target answer. During the training of the reward model, the second output data is output synchronously, ensuring close integration and reducing training complexity and cost.

[0050] Based on the same disclosed concept, this disclosure also provides a model training device. Since the principle of solving the problem by these devices is similar to that of the aforementioned model training method, the implementation of the device can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0051] The training apparatus for the model provided in the embodiments of this disclosure, such as Figure 4 As shown, it includes: The first training set construction module 401 is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, and multiple responses corresponding to each question; The first decision module 402 is used to make a decision on each response using a reward model, obtain multiple decision results, and determine the baseline label corresponding to the response based on the decision results. The first reward value determination module 403 is used to determine the reward value of each judgment result based on the benchmark label; The first update and output module 404 is used to train the reward model based on the reward value and the training set corresponding to the reward value.

[0052] Based on the same disclosed concept, this disclosure also provides a data generation apparatus. Since the principle of the problem solved by these apparatuses is similar to that of the aforementioned data generation method, the implementation of the apparatus can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0053] The data generation apparatus provided in the embodiments of this disclosure, such as Figure 5 As shown, it includes: The second training set construction module 501 is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; The second decision module 502 is used to make a decision on each response using a reward model, obtain multiple decision results and corresponding decision reasons, and determine the benchmark label corresponding to the response based on the decision results. The second reward value determination module 503 is used to determine the reward value of each judgment result based on the benchmark label; The second update and output module 504 is used to filter responses according to the reward value and a preset strategy to determine the filtered target response; and to determine second output data based on the target response; the second output data includes at least one of the following: the target response, the benchmark label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

[0054] Based on the same disclosed concept, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the model training method as described in any of the above embodiments; and / or the steps of the data generation method.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0056] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0057] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0058] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0059] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A model training method, characterized in that, include: Acquire the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, and multiple responses corresponding to each question; For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results; and a baseline label corresponding to the response is determined based on the judgment results. Based on the aforementioned benchmark labels, determine the reward value for each judgment outcome; The reward model is trained based on the reward value and the corresponding training set.

2. The method as described in claim 1, characterized in that, The step of training the reward model based on the reward value and the corresponding training set includes: Based on the reward value, responses are filtered according to a preset strategy to determine the target responses after filtering; The first output data corresponding to the target response is used as the model update sample, and the reward model is updated through a reinforcement learning algorithm. The first output data includes at least one of the following: the target response, the reward value of the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

3. The method as described in claim 2, characterized in that, The reward value includes a first type and a second type; The step of filtering responses according to the reward value and a preset strategy to determine the filtered target responses includes: For each response, the reward value of the first type is assigned to the first reward group and the reward value of the second type is assigned to the second reward group; Calculate the absolute value of the difference between the sum of the reward values ​​of the first reward group and the sum of the reward values ​​of the second reward group; If the absolute value of the difference is less than a first preset threshold, the response is determined as the target response after filtering.

4. The method as described in claim 3, characterized in that, The method further includes: If the absolute value of the difference is greater than or equal to a first preset threshold, the response and the reward value of the corresponding judgment result are deleted.

5. The method as described in claim 1, characterized in that, The step of determining the baseline label corresponding to the response based on the judgment result includes: For each response, the frequency of each type of decision result is counted among all the decision results corresponding to that response, and the decision result with the highest frequency is determined as the baseline label for that response.

6. The method as described in claim 1, characterized in that, For each response, a reward model is used to make a judgment on that response, resulting in multiple judgment outcomes, including: The data to be processed and the training set are input into the reward model. For each response, multiple independent judgments are made on the response to obtain the judgment result and the corresponding judgment reason for each judgment. The judgment result is a binary classification type, including two mutually exclusive judgment states.

7. The method as described in claim 1, characterized in that, The step of determining the reward value for each judgment result based on the benchmark label includes: Each judgment result is matched with the corresponding response's baseline label, and a corresponding reward value is configured for each judgment result based on the matching result.

8. The method as described in claim 1, characterized in that, After determining the baseline label for each response, the following is also included: Based on the aforementioned benchmark labels, construct benchmark label annotation samples; Based on the baseline labeled samples, the base model of the reward model is subjected to supervised fine-tuning. The benchmark label-annotated samples include: benchmark labels, judgment results corresponding to the benchmark labels, judgment reasons corresponding to the judgment results, and the training set; The training set also includes the reasons for generating each response.

9. The method as described in claim 1, characterized in that, The data to be processed includes video data; The process of acquiring data to be processed, performing question and response processing on the data to be processed, and determining the training set includes: Acquire video data; The video data is questioned using a questioning model to generate multiple initial questions. A variance incentive mechanism is then used to filter the initial questions to obtain multiple filtered questions. For each question, based on the video data, multiple response models are used to process the question, resulting in multiple responses and the reasons for generating each response; The answer model includes the reward model.

10. A data generation method, characterized in that, include: Acquire the data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; For each response, a reward model is used to make a judgment on the response, resulting in multiple judgment results and corresponding judgment reasons; and a baseline label is determined for the response based on the judgment results. Based on the aforementioned benchmark labels, determine the reward value for each judgment outcome; Based on the reward value, responses are filtered according to a preset strategy to determine the target responses after filtering; Based on the target response, second output data is determined; the second output data includes at least one of the following: the target response, the benchmark label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

11. A training device for a model, characterized in that, include: The first training set construction module is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, and multiple responses corresponding to each question; The first decision module is used to make a decision on each response using the reward model, obtaining multiple decision results; and to determine the baseline label corresponding to the response based on the decision results. The first reward value determination module is used to determine the reward value of each judgment result based on the benchmark label; The first update and output module is used to train the reward model based on the reward value and the training set corresponding to the reward value.

12. A data generation apparatus, characterized in that, include: The second training set construction module is used to acquire data to be processed, perform question and response processing on the data to be processed, and determine the training set; wherein, the training set includes at least one of the following: multiple questions, multiple responses corresponding to each question, and the reason for generating each response; The second decision module is used to make a decision on each response using a reward model, obtaining multiple decision results and corresponding decision reasons; and determining the benchmark label corresponding to the response based on the decision results. The second reward value determination module is used to determine the reward value for each judgment result based on the benchmark label. The second update and output module is used to filter responses according to the reward value and a preset strategy to determine the filtered target response; and to determine the second output data according to the target response; the second output data includes at least one of the following: the target response, the benchmark label corresponding to the target response, the judgment result corresponding to the target response, the judgment reason corresponding to the judgment result, the question corresponding to the target response, and the generation reason corresponding to the target response.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the model training method as described in any one of claims 1 to 9; and / or the steps of the data generation method as described in claim 10.

Images