Intelligent content evaluation and optimization method and system based on multi-criteria preference learning

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent content evaluation and optimization method and system based on multi-standard preference learning, the method comprising: constructing target user preference data, generating an evaluation trajectory including evaluation criteria and judgment results; based on the evaluation trajectory, screening samples and assigning credits through sorting and consistency rules to obtain preference pair training data; adopting response supervision fine-tuning and direct preference optimization strategies to jointly train a generative reward model; reorganizing the original evaluation trajectory into alternating acceptance and rejection samples to form long-thought chain training data, and further training to obtain a final model; and evaluating and optimizing the alignment between the generated text and the user's preferences through the final model. The present invention improves the alignment of the language model with human preferences and the overall performance through multi-stage optimization and process supervision, solves the problems of compound error and data sparsity of the traditional reward model, and performs excellently in out-of-distribution evaluation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent content evaluation and optimization method and system based on multi-criteria preference learning. Background Art

[0002] In the field of reward modeling for large language models (LLMs), the development of traditional reward models faces numerous technical bottlenecks. Currently, mainstream reward models include discriminative reward models and generative reward models (GenRMs), but both models have significant flaws.

[0003] For generative reward models, the core pain points are the difficulty in verifying the correctness of chain of thought (CoT) trajectories and the problem of compound errors in multi-dimensional evaluation scenarios. Specifically, GenRMs improve the interpretability of LLMs by synthesizing critical opinions, but the chain of thought trajectories generated during the reasoning process lack a reliable verification mechanism. Because the synthesis process is highly dependent on the model's own reasoning ability, the generated critical opinions may contain logical incoherence or factual errors, resulting in distorted reward signals. When faced with multiple evaluation criteria (such as usefulness, simplicity, consistency, etc.), the superposition of multi-dimensional indicators will amplify the impact of local errors in the chain of thought, resulting in compound errors, making it difficult for the model to achieve effective self-optimization through the "criticism-improvement" mechanism, seriously hindering the performance improvement of the model in open domain scenarios.

[0004] Traditional reward models (especially discriminative reward models) are constrained by the difficulty of explicitly modeling complex preference relationships and the lack of ability to optimize key criteria. Existing models often approximate preference functions through simplified ranking or comparison structures, making it difficult to capture the complex hierarchical, conditional, and conflicting structures that may exist in human preferences. For example, when dealing with complex objectives such as "usefulness" and "simplicity", the model lacks a clear goal decomposition capability and is unable to independently control and optimize key evaluation criteria, resulting in uncontrollable weights of each dimension during the optimization process, which ultimately affects the output quality. In addition, traditional models rely on implicitly learning ranking functions from human preferences. This implicit learning mechanism not only makes the model behavior opaque and poorly controllable, but also makes it difficult to cope with changes in task objectives or inconsistent preference criteria.

[0005] In summary, the thought chain trajectory verification defects and multi-dimensional compound error problems of the generative reward model in the existing technology, as well as the traditional model's insufficient modeling of complex preference relationships and lack of key criterion optimization capabilities, have become the core technical bottlenecks that restrict the accuracy and effectiveness of the reward model. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to solve the problems that the correctness of the thought chain trajectory in the generative reward model is difficult to verify, the performance improvement is difficult due to the compound errors of multi-dimensional evaluation, and the traditional model is difficult to explicitly model complex preference relationships and cannot optimize key criteria individually.

[0007] To solve the above technical problems, the present invention provides an intelligent content evaluation and optimization method and system based on multi-criteria preference learning, which includes the following steps:

[0008] Step S1: construct the target user's preference data, and obtain the corresponding evaluation criteria and judgment results based on the preference data to generate an evaluation trajectory;

[0009] Step S2: Based on the evaluation trajectory, sample screening and credit allocation are performed using sorting rules and consistency rules to obtain preference pair data training samples;

[0010] Step S3: Based on the preference pair data training samples, the generative reward model is jointly trained using a response supervision fine-tuning strategy and a direct preference optimization strategy to obtain a trained generative reward model;

[0011] Step S4: reorganizing the original evaluation trajectory into alternating evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further training the trained generative reward model using the long thought chain training data to obtain a final generative reward model;

[0012] Step S5: Evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

[0013] In one embodiment of the present invention, in step S1, the method for generating an evaluation trajectory includes:

[0014] For a given data set , based on the user query text and the candidate response pairs to be evaluated , generating evaluation dimensions in the form of multiple text sequences related to the target task , as the root node of the preference tree;

[0015] For each evaluation dimension , according to the conditional probability distribution of the evaluation criteria Sampling generation evaluation details , according to the conditional probability distribution of the judgment result Sampling generates judgment results , the judgment result Include responses to candidates and Rating results of and based on the scoring results The obtained preference response judgment results;

[0016] Each evaluation dimension Corresponding evaluation rules and judgment results As a leaf node, it constitutes a branch structure of the preference number. Each path from the root node to the leaf node forms an evaluation track containing evaluation dimensions, evaluation rules and judgment results. ;

[0017] in, is the user's preferred true label, represents the number of samples in the dataset, Indicates the index number of the sample, represents the number of evaluation dimensions, The index number representing the evaluation dimension or evaluation criterion.

[0018] In one embodiment of the present invention, a method for obtaining a preferred response judgment result based on the scoring result includes: and Rating, score and ,if ,but is a better preference response, otherwise For a better preference response.

[0019] In one embodiment of the present invention, in step S2, the method for obtaining the preference pair data training sample includes:

[0020] For each evaluation dimension and its corresponding evaluation trajectory , calculate the candidate response and Rating results of and , and get the difference ;

[0021] According to the difference Sort all evaluation dimensions to obtain a sorted evaluation dimension sequence , the original unsorted evaluation dimension sequence is denoted as , in order to obtain the evaluation dimension preference pair before and after sorting ;

[0022] For each evaluation dimension ,generate Different evaluation criteria and the corresponding Judgment results ,statistics The judgment results support Better than Number of times and support Better than Number of times ,Compare and The size of , we can get the judgment results that tend to be consistent and the judgment results that tend to be conflicting;

[0023] Based on the judgment result of the bias consistency, select the score difference The judgment results greater than the threshold and their corresponding scoring rules are taken as acceptance samples; the judgment results with conflicting biases and their corresponding evaluation rules are taken as rejection samples to obtain the preference pairs of evaluation rules. and judgment result preference ;

[0024] The evaluation dimension preference 、The evaluation rules prefer and the judgment result preference Together they constitute the preference pair data training samples.

[0025] In one embodiment of the present invention, in step S3, the method for obtaining the trained generative reward model includes:

[0026] Based on the preference pair data training sample, a first sample data set is constructed and the second sample dataset ;

[0027] in, Indicates the query text entered by the user. Indicates that the evaluation dimension accepts samples, Indicates that the evaluation dimension rejects the sample; Indicates that the evaluation criteria accept samples, Indicates that the evaluation criteria reject the sample; Indicates that the judgment result accepts the sample. Indicates that the judgment result rejects the sample; represents the number of the first sample data set, represents the number of the second sample data set;

[0028] Construct an initial generative reward model, take the first sample set as input, and calculate the generated preference label results With input sample Original preference label for , optimizing the training parameters of the initial generative reward model based on the cross entropy loss value to obtain an optimized generative reward model;

[0029] The preference pairs in the second sample data set 、 、 The optimized generative reward model is trained as input, and the model parameters are optimized so that the probability of the model generating an acceptance sample is higher than that of a rejection sample, thereby obtaining a trained generative reward model.

[0030] In one embodiment of the present invention, the loss function for training the optimized generative reward model includes a loss function for evaluating dimension preference pairs, as follows:

[0031] ,

[0032] in, represents the generative reward model with the currently trained policy model parameter θ, represents the model pre-trained by fine-tuning the response supervision, Indicates the mathematical expectation; Indicates including Preference for datasets; Represents the sigmoid function; is a hyperparameter used to control the current strategy Deviation from the reference strategy degree.

[0033] In one embodiment of the present invention, a method for obtaining a final generative reward model includes:

[0034] Based on the preference pair data training samples, the acceptance samples of the original evaluation trajectory are obtained , and reject samples , the receiving sample and the rejection sample Alternate arrangement to obtain long thinking chain training data , For the evaluation trajectories accept samples, For the evaluation trajectories reject samples;

[0035] Based on the long thought chain training data, the trained generative reward model is retrained to obtain a final generative reward model.

[0036] Based on the same inventive concept, the present invention also provides an intelligent content evaluation and optimization system based on multi-criteria preference learning, which includes the following modules:

[0037] A preference data construction module is used to construct the preference data of the target user and obtain the corresponding evaluation criteria and judgment results based on the preference data to generate an evaluation trajectory;

[0038] A sample screening and credit assignment module is used to perform sample screening and credit assignment based on the evaluation trajectory using sorting rules and consistency rules to obtain preference pair data training samples;

[0039] a joint training optimization module, configured to jointly train the generative reward model based on the preference pair data training samples using a response supervision fine-tuning strategy and a direct preference optimization strategy to obtain a trained generative reward model;

[0040] A long thought chain training module is used to reorganize the original evaluation trajectory into an alternating arrangement of evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further train the trained generative reward model using the long thought chain training data to obtain a final generative reward model;

[0041] The content evaluation and optimization module is used to evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

[0042] The present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected via the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the intelligent content evaluation and optimization method based on multi-standard preference learning.

[0043] The present invention also provides a computer storage medium storing a computer software product, wherein the computer software product includes several instructions for enabling a computer device to execute the intelligent content evaluation and optimization method based on multi-standard preference learning.

[0044] The above technical solution of the present invention has the following advantages over the prior art:

[0045] This paper proposes a standard-based preference tree framework that reconstructs the generative reward modeling process into a standard-based preference tree form to optimize the synthetic standard trajectory. By introducing predefined rules and Monte Carlo tree expansion strategies, an effective process supervision mechanism is introduced into the generative modeling process to ensure the correctness of the reasoning chain in long thought chains, thereby significantly reducing the compound error of the synthetic thought chain. At the same time, the present invention uses sorting rules and consistency rules to enable large-scale language models to more accurately identify and prioritize key criteria and explicitly model complex preference relationships. This approach significantly improves the accuracy and effectiveness of reward modeling, ultimately improving the alignment of the language model with human preferences and overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 1 is a flow chart of an intelligent content evaluation and optimization method based on multi-criteria preference learning provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a specific implementation flow of an intelligent content evaluation and optimization method based on multi-criteria preference learning provided in an embodiment of the present invention;

[0049] Figure 3 This is a graph of the evaluation results of the impact of the number of criteria on model performance;

[0050] Figure 4 Schematic diagram of the structure of an intelligent content evaluation and optimization system based on multi-criteria preference learning provided in an embodiment of the present invention;

[0051] Explanation of the accompanying drawings in the specification: 100, preference data construction module; 200, sample screening and credit allocation module; 300, joint training optimization module; 400, long thinking chain training module; 500, content evaluation and optimization module. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0053] Example 1:

[0054] like Figure 1 and Figure 2 As shown, the present invention provides an intelligent content evaluation and optimization method and system based on multi-criteria preference learning, which includes the following steps:

[0055] Step S1: construct the target user's preference data, obtain the corresponding evaluation criteria and judgment results based on the preference data, and generate the evaluation trajectory using the Monte Carlo expansion strategy;

[0056] Step S2: Based on the evaluation trajectory, sample screening and credit allocation are performed using sorting rules and consistency rules to obtain preference pair data training samples;

[0057] Step S3: Based on the preference pair data training samples, the generative reward model is jointly trained using the response supervised fine-tuning (RSFT) strategy and the direct preference optimization (DPO) strategy to obtain a trained generative reward model;

[0058] Step S4: reorganizing the original evaluation trajectory into alternating evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further training the trained generative reward model using the long thought chain training data to obtain a final generative reward model;

[0059] Step S5: Evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

[0060] It can be seen from the above technical solution that the present invention constructs user preference data and generates evaluation trajectories, uses sample screening and credit allocation to obtain high-quality preference pair data, and combines multiple training strategies to perform multi-stage optimization training on the generative reward model. Ultimately, it can accurately evaluate and optimize the degree of alignment between the generated text and user preferences. It has a high degree of personalization, accuracy and adaptability, and can effectively improve the quality of content generation and user satisfaction.

[0061] Furthermore, in this embodiment, in step S1, preference data of the target user is constructed, and corresponding evaluation criteria and judgment results are obtained based on the preference data. The method of generating an evaluation trajectory using a Monte Carlo expansion strategy includes:

[0062] For a given data set , using a large language model (such as LLaMA3.1-8B) based on the user query question text and the candidate response pairs to be evaluated , and According to the query text The two answer texts obtained generate evaluation dimensions in the form of multiple text sequences related to the target task. , , as the root node of the preference tree; as the core component of the intelligent content evaluation system, each evaluation dimension It represents a perspective on content evaluation. Typical dimensions include semantic relevance, information usefulness, and clarity of expression. Its definition is highly coupled with the task semantic space and constitutes the basic framework for multi-criteria preference learning.

[0063] For each evaluation dimension , according to the conditional probability distribution of the evaluation criteria By adjusting the temperature parameter, we can generate specific evaluation criteria by sampling from the probability distribution output by the language model. The detailed rules are a computable decomposition of the evaluation dimensions. By constructing a hierarchical semantic mapping relationship, the abstract standards are transformed into specific judgment criteria, including but not limited to scoring criteria such as ("5 points = complete and unambiguous content") and example explanations. The purpose is to standardize the consistency and repeatability of the model evaluation process by clearly defining the boundaries of excellence in the evaluation dimensions;

[0064] According to the conditional probability distribution of the judgment result , sampling generates judgment results , the judgment result Based on the evaluation dimension and its corresponding evaluation rules The decision-making process of quantitatively evaluating the model output through probability distribution sampling or decision tree reasoning, specifically including:

[0065] Response to candidate and Explicit scoring results for ,in ;

[0066] A textual explanation of the scoring basis (e.g., “Response 4 points on the 'clarity' dimension for using simpler terminology");

[0067] Based on the scoring results The result of the preference response judgment is, for example, if ,but is a better preference response, otherwise For a better preference response.

[0068] Each evaluation dimension Corresponding evaluation rules and judgment results As leaf nodes, a branch structure of preference numbers is formed, each from the root node (evaluation dimension ) to the leaf node (judgment result ) forms a complete evaluation trajectory , including full process information from input to final scoring;

[0069] in, is the user's preferred true label, represents the number of samples in the dataset, Indicates the index number of the sample, represents the number of evaluation dimensions, The index number representing the evaluation dimension or evaluation criterion.

[0070] Furthermore, in step S2, in order to ensure that the samples have high confidence and consistency and improve the model's ability to model human preferences, based on the evaluation trajectory Methods for obtaining preference pair data training samples by performing sample screening and credit assignment using ranking rules and consistency rules include:

[0071] For each evaluation dimension and its corresponding evaluation trajectory , calculate the candidate response and Rating results of and , and get the difference , Indicates the confidence level of the model in this evaluation dimension. The larger the difference, the more confident the judgment. Compare Higher confidence level;

[0072] Query for the same input text and candidate response pairs , all evaluation dimensions generated , according to the difference Sort all evaluation dimensions from large to small to obtain the sorted evaluation dimension sequence ,in is the evaluation dimension with the largest difference (the most critical criterion); the original unsorted evaluation dimension sequence is recorded as , in order to obtain the evaluation dimension preference pair before and after sorting , this pair represents the preference relationship between the high-confidence standard sequence and the original standard sequence, and is used to train the model to preferentially activate key standards;

[0073] For each evaluation dimension ,generate Different evaluation criteria And the corresponding 4 judgment results ,statistics The judgment results support Better than Number of times and support Better than Number of times ,Compare and If at least three judgment results choose the same response (such as ), it is considered a “consistent judgment”; if the judgment results conflict (such as 2 options , 2 options ), it is considered as an "inconsistent judgment", and ultimately the judgment results tending to be consistent and the judgment results tending to be conflicting are obtained;

[0074] Based on the judgment result of the bias consistency, select the score difference The judgment results greater than the threshold and their corresponding scoring rules are taken as acceptance samples; the judgment results with conflicting biases and their corresponding evaluation rules are taken as rejection samples to obtain the preference pairs of evaluation rules. and judgment result preference ;

[0075] The evaluation dimension preference 、The evaluation rules prefer and the judgment result preference Together they constitute the preference pair data training samples.

[0076] To explain the above method of obtaining high-quality preference pair training data samples, take the evaluation dimension "clarity" as an example:

[0077] The first step is to calculate the difference in scores: if response A gets 4 points and response B gets 2 points, , this dimension has medium priority in the sorting;

[0078] Step 2: Consistency check: Generate 4 judgment results. If 3 of them think "A is clearer" and 1 thinks "B is clearer", then the majority agrees and the sample is accepted.

[0079] Step 3: Preference pair generation: sorted evaluation dimension sequence In the Ranked 3rd; consistent evaluation criteria The corresponding judgment result is "4 points = concise and unambiguous expression". "A gets 4 points, B gets 2 points"; inconsistent evaluation criteria (e.g. conflicting evaluation criteria) were rejected.

[0080] Furthermore, in this embodiment, in step S3, based on the preference pair data training samples, the RSFT strategy and the DPO strategy are used to jointly train the generative reward model to obtain the trained generative reward model, which includes:

[0081] Based on the preference pair data training sample, a first sample data set is constructed and the second sample dataset ;

[0082] in, Indicates the query text entered by the user. Indicates that the evaluation dimension accepts samples, Indicates that the evaluation dimension rejects the sample; Indicates that the evaluation criteria accept samples, Indicates that the evaluation criteria reject the sample; Indicates that the judgment result accepts the sample. Indicates that the judgment result rejects the sample; represents the number of the first sample data set, represents the number of the second sample data set;

[0083] Construct an initial generative reward model, take the first sample set as input, perform supervised learning on the model, and calculate the generated preference label results With input sample Original preference label for The cross entropy loss value is used to optimize the training parameters of the initial generative reward model to train the model to generate high-quality evaluation dimensions, evaluation criteria, and judgment results, thereby obtaining an optimized generative reward model, namely, the SyncPL-SFT model.

[0084] The preference pairs in the second sample data set 、 、 As input, with the goal of maximizing the probability of generating accepted samples and minimizing the probability of rejecting samples, the optimized generative reward model is trained, and the model parameters are optimized so that the probability of the model generating accepted samples is higher than that of rejecting samples, thereby obtaining the trained generative reward model, namely the SyncPL-DPO model.

[0085] Specifically, the loss function for training the optimized generative reward model (SyncPL-DPO model) includes a loss function for evaluation dimension preference pairs, a loss function for evaluation criterion preference pairs, and a loss function for judgment result preference pairs, wherein the loss function for evaluation dimension preference pairs is as follows:

[0086] ,

[0087] in, represents the generative reward model with the currently trained policy model parameter θ, represents the model pre-trained by fine-tuning the response supervision, Indicates the mathematical expectation; Indicates including Preference for datasets; Represents the sigmoid function; is a hyperparameter used to control the current strategy Deviation from the reference strategy the extent of

[0088] The loss function for the evaluation criteria preference pair and the loss function for the judgment result preference pair are similar to the above functions and are expressed as follows:

[0089] ,

[0090] .

[0091] Through the above-mentioned joint training mechanism based on RSFT strategy and DPO strategy, the model can gradually learn to prefer more reasonable evaluation dimensions, evaluation criteria and judgment results in complex evaluation paths.

[0092] Furthermore, in this embodiment, the method for obtaining the final generative reward model includes:

[0093] Inspired by the "Long CoT" capability in the OpenAI o1 series model, based on the preference pair data training samples, we get the acceptance samples of the original evaluation trajectory. , and reject samples , the receiving sample and the rejection sample Alternate arrangement to obtain long thinking chain training data , For the evaluation trajectories accept samples, For the evaluation trajectories reject samples;

[0094] In order to enhance the model's self-reflection and long-range reasoning capabilities, based on the long thought chain training data, the trained generative reward model (SyncPL-DPO model) is retrained to obtain the final generative reward model, namely the SyncPL-o1 model.

[0095] To comprehensively evaluate the SyncPL-o1 model's performance in terms of evaluation quality and generation stability in generated answer text, this study tested the reward model using three different human-preferred benchmarks: RewardBench, Auto-J, and MT-Bench. The baseline models included the state-of-the-art closed-source generative models: GPT-4, GPT4-o, and DeepSeek V3. For the Discriminative Reward Model (RM), the LLaMA3-8B RM, trained with the BT objective function and hyperfeedback, was selected. For the Generative Reward Model (GenRM), the Prometheus2 model, trained with synthetic data from GPT-4, was selected. Furthermore, the Cloud model utilized LLaMA3-405B, using a hybrid approach to generate Chain of Thought (CoT) reasoning paths for model training.

[0096] Experiment 1: Validates the SyncPL model's reward modeling capabilities under out-of-distribution (OOD) conditions. This experiment uses the Qwen2.5-32B-Instruct-GPTQ-Int8 model to generate Chain of Thought (CoT) data, which is then used to train the LLaMA3.1-8B model, employing a distillation model as its underlying architecture. SyncPL builds a fine-grained reward model using conversational data from UltraFeedback and OffsetBias. These datasets have significantly different data distributions from the benchmarks, so the experimental results can be considered out-of-distribution evaluations. The experimental results, shown in Table 1, show that through response-supervised fine-tuning (RSFT), SyncPL significantly improves alignment with human preferences, achieving a 2.8-point improvement in alignment score compared to the LLaMA3.1-8B distillation model. Furthermore, the SyncPL-DPO model improves reward modeling capabilities by 3.8 points through a direct preference optimization strategy. SyncPL-o1 is trained on synthetic data generated by SyncPL-SFT. Its advantage lies in its ability to generate negative data to enhance the model's reflective ability. Through reinforcement learning (RL), SyncPL-o1 achieves excellent performance, with its evaluation score (84.2) exceeding the LLaMA3.1-70B model (84.0). This result is mainly attributed to the use of standard judgment data in the SyncPL optimization process. The introduction of process supervision signal ensures the correctness of the reasoning chain in the long thinking chain (CoT), thereby fully tapping the potential of the model. In Table 1, the " Evaluation models marked with "" represent self-measured experimental data, and unmarked evaluation models represent experimental data that have been published in existing research.

[0097] Table 1:

[0098]

[0099] Experiment 2: Evaluate the judgment performance of the model under in-distribution (ID) conditions. This experiment compares the model fine-tuned on Mistral-7B-Instruct using Fennec training data. This dataset contains 3K dialogue samples from Auto-J and chain of thought (CoT) trajectory data annotated by GPT-4. To ensure a fair comparison, the consistency (AGR) indicator is used as the evaluation criterion. In addition, coherence (CNS) is also used as one of the performance indicators, but it only measures position consistency without considering correctness. The experimental results in Table 2 show that the SyncPL-RSFT and SyncPL-DPO models perform significantly better than most baseline methods on the Auto-J dataset, achieving the highest consistency score of 58.41. In Table 2, the " Evaluation models marked with "" represent self-measured experimental data, and unmarked evaluation models represent experimental data that have been published in existing research.

[0100] Table 2:

[0101]

[0102] Experiment 3: Evaluating Conversation Correction Capabilities. The key function of the Generative Reward Model (GenRM) is to identify and correct flaws in conversations to improve their overall quality. To assess this capability, this experiment tested the model's ability to evaluate and optimize conversations. On the multi-turn conversation evaluation benchmark MT-Bench, conversations generated by Alpaca-13B and LLaMA2-7B Chat were evaluated and optimized. Unlike previous pairwise evaluation methods, MT-Bench uses GPT-4 to score AI responses on a scale of 1 to 10 and establishes a ranking relationship based on these scores. Specifically, multiple judgments are generated for each response, and responses scoring below 3 are optimized. As shown in Table 3, the optimization significantly improves the scores of both Alpaca-13B and LLaMA2-7B Chat. Furthermore, the model using the Direct Preference Optimization (DPO) strategy outperforms the Response Supervised Fine-tuning (RSFT) model in terms of correction performance, demonstrating that the improved evaluation feedback mechanism can significantly improve model evaluation quality.

[0103] Table 3:

[0104]

[0105] Experiment 4: Verify the scalability of the SyncPL branch. This experiment uses the RewardBench benchmark framework to evaluate the performance scalability of models at different training stages under multiple standard conditions. Figure 3 As shown, with increasing the number of evaluation criteria, the performance of the Distill and SyncPL-RSFT models significantly improves on various task datasets, including the Chat, Chat Hard, Safety, and Reasoning datasets. In particular, in the Reasoning task, the application of multiple evaluation criteria significantly improves the model's judgment accuracy, with maximum improvements of +18.2% and +12.6%, respectively. Furthermore, by optimizing the selection of key criteria, the SyncPL-DPO model is able to make accurate judgments using only a single criterion, further validating the effectiveness of the proposed preference optimization method. The SyncPL-o1 model uses the Chain of Thought (CoT) format and is trained using key criteria extracted from the SyncPL-RSFT model. Experimental results demonstrate that the proposed method not only ensures stable performance improvements but also achieves better optimization efficiency. Specifically, the SyncPL-o1 model outperforms other models on almost all sub-datasets, fully demonstrating its superior performance and scalability under multi-criteria conditions.

[0106] Furthermore, the proposed method has wide application. In scientific research writing assistance, it conducts multi-dimensional analysis of review drafts based on structured criteria such as technical accuracy, generates interpretable recommendation rationales, and optimizes review quality. In open source community content curation, it constructs evaluation trajectories based on community dynamic preferences, outputs explanatory recommendation rationales, and forms an adaptive content ecosystem. Through "standard-rule-judgment" trajectory modeling, rule screening, and joint training, this technology achieves explicit modeling and conflict resolution of complex preferences in various scenarios, ensures the interpretability of the evaluation process, and continuously optimizes the model, providing a universal solution for deeply aligning AI systems with human preferences.

[0107] Example 2:

[0108] Based on the same inventive concept as that of Example 1, the present invention also provides an intelligent content evaluation and optimization system based on multi-standard preference learning, which is used to implement the steps of the intelligent content evaluation and optimization method based on multi-standard preference learning described in Example 1.

[0109] like Figure 4As shown, the intelligent content evaluation and optimization system based on multi-standard preference learning includes: a preference data construction module 100, a sample screening and credit allocation module 200, a joint training optimization module 300, a long-term thinking chain training module 400 and a content evaluation and optimization module 500;

[0110] The preference data construction module 100 is used to construct the preference data of the target user, and obtain the corresponding evaluation criteria and judgment results based on the preference data to generate an evaluation trajectory;

[0111] The sample screening and credit allocation module 200 is used to perform sample screening and credit allocation based on the evaluation trajectory through sorting rules and consistency rules to obtain preference pair data training samples;

[0112] The joint training optimization module 300 is used to jointly train the generative reward model based on the preference pair data training samples using a response supervision fine-tuning strategy and a direct preference optimization strategy to obtain a trained generative reward model;

[0113] The long thought chain training module 400 is used to reorganize the original evaluation trajectory into alternating evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further train the trained generative reward model using the long thought chain training data to obtain the final generative reward model;

[0114] The content evaluation and optimization module 500 is used to evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

[0115] The present embodiment proposes an intelligent content evaluation and optimization system based on multi-standard preference learning, which is used to implement the aforementioned intelligent content evaluation and optimization method based on multi-standard preference learning. Therefore, the specific implementation methods of the intelligent content evaluation and optimization system based on multi-standard preference learning can be seen in the embodiment part of the aforementioned intelligent content evaluation and optimization method based on multi-standard preference learning. For example, the preference data construction module 100, the sample screening and credit allocation module 200, the joint training optimization module 300, the long thinking chain training module 400 and the content evaluation and optimization module 500 are respectively used to implement the steps S1, S2, S3, S4 and S5 of the intelligent content evaluation and optimization method based on multi-standard preference learning in embodiment one. Therefore, its specific implementation methods can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, they will not be repeated here.

[0116] Example 3:

[0117] The present invention also provides an electronic device, which includes a processor, a memory and a bus system, wherein the processor and the memory are connected via the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the intelligent content evaluation and optimization method based on multi-standard preference learning described in Example 1.

[0118] Example 4:

[0119] The present invention also provides a computer storage medium storing a computer software product, wherein the computer software product includes several instructions for enabling a computer device to execute the intelligent content evaluation and optimization method based on multi-standard preference learning described in Example 1.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0124] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. An intelligent content evaluation and optimization method based on multi-criteria preference learning, characterized in that: The following steps are involved: Step S1: Construct the target user's preference data, and obtain the corresponding evaluation criteria and judgment results based on the preference data to generate an evaluation trajectory, including: For a given data set , based on the user query text and the candidate response pairs to be evaluated , generating evaluation dimensions in the form of multiple text sequences related to the target task , as the root node of the preference tree; For each evaluation dimension , according to the conditional probability distribution of the evaluation criteria Sampling generation evaluation details , according to the conditional probability distribution of the judgment result Sampling generates judgment results , the judgment result Include responses to candidates and Rating results of and a preference response judgment result obtained based on the scoring result; wherein the method for obtaining the preference response judgment result based on the scoring result comprises: and Rating, score and ,if ,but is a better preference response, otherwise for a more optimal preference response; Each evaluation dimension Corresponding evaluation rules and judgment results As a leaf node, it constitutes a branch structure of the preference number. Each path from the root node to the leaf node forms an evaluation track containing evaluation dimensions, evaluation rules and judgment results. ; in, is the user's preferred true label, represents the number of samples in the dataset, Indicates the index number of the sample, represents the number of evaluation dimensions, The index number representing the evaluation dimension or evaluation criteria; Step S2: Based on the evaluation trajectory, sample screening and credit allocation are performed using sorting rules and consistency rules to obtain preference pair data training samples, including: For each evaluation dimension and its corresponding evaluation trajectory , calculate the candidate response and Rating results of and , and get the difference ; According to the difference Sort all evaluation dimensions to obtain a sorted evaluation dimension sequence , the original unsorted evaluation dimension sequence is denoted as , in order to obtain the evaluation dimension preference pair before and after sorting ; For each evaluation dimension ,generate Different evaluation criteria and the corresponding Judgment results ,statistics The judgment results support Better than Number of times and support Better than Number of times ,Compare and The size of , we can get the judgment results that tend to be consistent and the judgment results that tend to be conflicting; Based on the judgment result of the bias consistency, select the score difference The judgment results greater than the threshold and their corresponding scoring rules are taken as acceptance samples; the judgment results with conflicting biases and their corresponding evaluation rules are taken as rejection samples to obtain the preference pairs of evaluation rules. and judgment result preference ; The evaluation dimension preference 、The evaluation rules prefer and the judgment result preference Together they constitute the preference pair data training samples; Step S3: Based on the preference pair data training samples, the generative reward model is jointly trained using a response supervision fine-tuning strategy and a direct preference optimization strategy to obtain a trained generative reward model; Step S4: reorganizing the original evaluation trajectory into alternating evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further training the trained generative reward model using the long thought chain training data to obtain a final generative reward model; Step S5: Evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

2. The intelligent content evaluation and optimization method based on multi-criteria preference learning according to claim 1 is characterized in that: In step S3, the method for obtaining the trained generative reward model includes: Based on the preference pair data training sample, a first sample data set is constructed and the second sample dataset ; in, Indicates the query text entered by the user. Indicates that the evaluation dimension accepts samples, Indicates that the evaluation dimension rejects the sample; Indicates that the evaluation criteria accept samples, Indicates that the evaluation criteria reject the sample; Indicates that the judgment result accepts the sample. Indicates that the judgment result rejects the sample; represents the number of the first sample data set, represents the number of the second sample data set; Construct an initial generative reward model, take the first sample set as input, and calculate the generated preference label results With input sample Original preference label for , optimizing the training parameters of the initial generative reward model based on the cross entropy loss value to obtain an optimized generative reward model; The preference pairs in the second sample data set 、 、 The optimized generative reward model is trained as input, and the model parameters are optimized so that the probability of the model generating an acceptance sample is higher than that of a rejection sample, thereby obtaining a trained generative reward model.

3. The intelligent content evaluation and optimization method based on multi-criteria preference learning according to claim 2 is characterized in that: The loss function for training the optimized generative reward model includes a loss function for evaluating dimension preference pairs, as follows: ， in, represents the generative reward model with the currently trained policy model parameter θ, represents the model pre-trained by fine-tuning the response supervision, Indicates the mathematical expectation; Indicates including Preference for datasets; Represents the sigmoid function; is a hyperparameter used to control the current strategy Deviation from the reference strategy degree.

4. The intelligent content evaluation and optimization method based on multi-criteria preference learning according to claim 1 is characterized in that: The methods to obtain the final generative reward model include: Based on the preference pair data training samples, the acceptance samples of the original evaluation trajectory are obtained , and reject samples , the receiving sample and the rejection sample Alternate arrangement to obtain long thinking chain training data , For the evaluation trajectories accept samples, For the evaluation trajectories reject samples; Based on the long thought chain training data, the trained generative reward model is retrained to obtain a final generative reward model.

5. An intelligent content evaluation and optimization system based on multi-criteria preference learning, characterized in that: The steps of implementing the intelligent content evaluation and optimization method based on multi-criteria preference learning as claimed in claim 1 are as follows: the intelligent content evaluation and optimization system based on multi-criteria preference learning comprises the following modules: A preference data construction module is used to construct the preference data of the target user and obtain the corresponding evaluation criteria and judgment results based on the preference data to generate an evaluation trajectory; A sample screening and credit assignment module is used to perform sample screening and credit assignment based on the evaluation trajectory using sorting rules and consistency rules to obtain preference pair data training samples; a joint training optimization module, configured to jointly train the generative reward model based on the preference pair data training samples using a response supervision fine-tuning strategy and a direct preference optimization strategy to obtain a trained generative reward model; A long thought chain training module is used to reorganize the original evaluation trajectory into an alternating arrangement of evaluation trajectory acceptance samples and evaluation trajectory rejection samples to form long thought chain training data, and further train the trained generative reward model using the long thought chain training data to obtain a final generative reward model; The content evaluation and optimization module is used to evaluate and optimize the alignment between the text content generated based on the target user's query and their preferences through the final generative reward model.

6. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus system, the processor and the memory are connected through the bus system, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the intelligent content evaluation and optimization method based on multi-standard preference learning as described in any one of claims 1 to 4.

7. A computer storage medium, characterized in that The computer storage medium stores a computer software product, which includes several instructions for enabling a computer device to execute the intelligent content evaluation and optimization method based on multi-standard preference learning as described in any one of claims 1 to 4.

Images