Closed-loop reward optimization method and system based on multi-modal human preference alignment

Abstract

This invention relates to the field of power grids, specifically disclosing a closed-loop reward optimization method based on multimodal human preference alignment, comprising the following steps: S1: Candidate generation step: The intelligent monitoring robot receives multimodal input information, and the main model generates multiple candidate outputs; S2: Reward evaluation step: The candidate outputs are scored for cross-modal language consistency and task relevance using a unified reward model, and internal confidence calibration is performed using a self-reward and confidence calibration module. Natural language criticism and negative penalty scores are generated using a critical reward module; S3: Reward fusion step: The unified reward model score, the self-reward and confidence calibration module score, and the critical reward module score are weighted and fused to obtain a total reward value; S4: Criticism transformation step: The natural language criticism generated by the critical reward module is transformed into pseudo-preference pair data; S5: Closed-loop update step: The fused total reward value and the transformed pseudo-preference pair data are used to update the main model parameters.

Description

Technical Field

[0001] This invention belongs to the field of power grids, and in particular relates to a closed-loop reward optimization method and system based on multimodal human preference alignment. Background Technology

[0002] Current research on intelligent monitoring robots relies on human preference alignment methods based on manual annotation, such as RLHF (Reinforcement Learning with Human Feedback) and DPO (Direct Preference Optimization). RLHF collects a large number of human preference pairs to train a reward model, which ranks or scores candidate outputs. Reinforcement learning is then used to optimize the main model using the reward signal from the RLHF. DPO, on the other hand, is a direct optimization method that replaces RLHF. It does not require complex reinforcement learning but directly adjusts the main model parameters based on the relative probabilities of preferred and rejected outputs. In practical production applications, preference alignment models are built using the experience and annotations of human monitors. For example, manual annotations are used to judge the quality of robot-generated logs and the rationality of anomaly handling suggestions, which are then used as reward signals to train the robot model. This method is mature and stable, and can capture human subjective values ​​and aesthetic preferences to a certain extent. However, its shortcomings cannot be ignored. Power grid monitoring scenarios are complex, requiring a large number of manual samples as preference data, making annotation extremely costly, especially in multimodal tasks where the annotation difficulty is far greater than that of plain text. Furthermore, different monitors may make different judgments in the same scenario, making it difficult to form a stable reward model. The scale of manually labeled samples is limited. As tasks become more complex, samples for scenarios such as sudden accidents and complex cross-site linkages are scarce. Manual labeling is difficult to cover long-tail scenarios, resulting in insufficient generalization ability of the reward model.

[0003] Some rule-based or self-scoring reward mechanisms, such as CSR-VLMs, propose having a large model generate multiple candidate outputs, then using the model's own confidence estimation or an internal discriminator to score these candidates. To avoid "model self-praise," they introduce calibration mechanisms, such as comparison with external auxiliary models or designing constraints to suppress overconfidence. In the case of no or few annotations, pseudo-preference pairs are constructed to further optimize the main model. This approach significantly reduces reliance on manual annotation, improves the automation level of the model training loop, and achieves results similar to human preferences in tasks such as image description and image-text response. However, the illusion problem or logical errors inherent in the model itself may be repeatedly reinforced during the self-reward process. Furthermore, existing CSR methods are insufficiently comprehensive, often limited to single data sources, lacking a comprehensive consideration of image, speech, and time-series signals. Most rely solely on simple confidence or comparison methods, making it difficult to dynamically adapt to different task requirements.

[0004] Another type of research, Unified Reward Model (URM), focuses on training a unified reward model capable of handling tasks such as text (logs, reports), voice interactions (operation monitoring), and visual signals (screen monitoring). This model is jointly trained on multiple tasks to learn shared quality assessment capabilities across tasks. Unified reward models improve the consistency of reward signals across different tasks, reduce fragmentation issues in reward models, facilitate deployment in large-scale multimodal systems, and possess good scalability. However, they have some drawbacks. First, they lack personalized constraints; unified models often ignore the specificities of tasks or modalities, and different modalities require different evaluation criteria, which a single unified reward model cannot adequately address. Furthermore, if the model's reward is incorrect, the entire training loop may be misled, and the bias is difficult to correct.

[0005] The shortcomings of existing methods in intelligent monitoring robot scenarios are mainly manifested in the following aspects: over-reliance on human experience, poor scalability, unstable rules and self-monitoring mechanisms that easily amplify errors, lack of cross-modal unified and personalized integration, difficulty in covering complex business, and lack of closed-loop feedback in reward signals, making it difficult to support robot self-evolution. Therefore, there is an urgent need for a novel closed-loop optimization method that integrates self-reward, unified reward, and critical feedback to adapt to the long-term stable operation requirements of power grid monitoring robots in multimodal scenarios. Summary of the Invention

[0006] This invention proposes a closed-loop reward optimization method for intelligent monitoring robots based on multimodal human preference alignment. In power grid centralized control and maintenance scenarios, monitoring robots need to simultaneously process images, voice, text, and time-series signals, and output candidate solutions in key tasks such as anomaly handling, log generation, and operation monitoring. However, most existing systems rely on fixed rules or manual annotation, failing to form a self-evolving closed loop. This invention constructs a unified candidate generation-reward evaluation-criticism feedback-training update loop, enabling the robot to continuously self-optimize during actual operation.

[0007] To solve the above problems, the technical solution provided by the present invention is as follows: First technical solution: A closed-loop reward optimization method based on multimodal human preference alignment, including the following steps: S1: Candidate generation step: The intelligent monitoring robot receives multimodal input information, and the main model generates multiple candidate outputs, including anomaly handling suggestions, log drafts, and operation monitoring prompts; S2: Reward evaluation steps: Use a unified reward model to score the cross-modal language consistency and task relevance of candidate outputs, use a self-reward module to perform internal confidence calibration scoring, and use a critical reward module to generate natural language criticisms and corresponding negative penalty scores. S3: Reward Fusion Step: The unified reward model score, the self-reward and confidence calibration module score, and the critical reward module score are weighted and fused to obtain the total reward value; S4: Criticism Transformation Steps: Transform the natural language criticisms generated by the criticism reward module into pseudo-preference pairs; S5: Closed-loop update steps: Use the merged total reward value and the transformed pseudo-preference pair data to update the main model parameters.

[0008] Preferably, the formula for the total reward value in step S3 is as follows: ; The above formula: R total The total reward value, The score represents the unified reward model score, which mainly measures the matching degree between candidate outputs and tasks and cross-modal consistency, such as whether logs cover alarm signals and whether operation monitoring prompts are synchronized with screen displays. The score represents the self-reward and confidence calibration module score, which reflects the robot's confidence estimate and logical consistency of its own output; The score represents the critical reward module score, which includes both positive reinforcement signals and negative penalties based on criticisms, used to correct potential biases in the model. parameter The validation set can be automatically tuned, or adjustments can be made during operation using a dynamic weighting mechanism.

[0009] Preferably, the dynamic adjustment method for the weight coefficients α, β, γ is as follows: based on the reliability score of each reward source on the short-term validation set, the weights are calculated using the Softmax function, so that reward sources with high reliability occupy a higher weight in the fusion.

[0010] Preferably, the criticism transformation step specifically includes: Use a critical reward module to diagnose defects in candidate outputs; Generate corresponding correction suggestions based on the defects; Improved candidate outputs are automatically generated based on the suggested corrections. ; Compare the original candidate output y with the improved candidate output Constructing pseudo-preference pairs (y, ).

[0011] Preferably; S5: The closed-loop update step adopts the Direct Preference Optimization (DPO) algorithm, which uses the fused reward value and pseudo preference to calculate the loss function on the data and update the parameters of the main model.

[0012] Preferably, the training method for the unified reward model includes: Construct multimodal contrastive learning samples and encode the input state x and candidate output y into semantic vectors; Define the InfoNCE loss function to shorten the vector distance between positive sample pairs and widen the vector distance between negative sample pairs; The cross-modal language consistency score is calculated using a cosine similarity function.

[0013] Preferably, the workflow of the self-reward and confidence calibration module includes: Temperature scaling calibration is applied to the raw Logits output by the master generator model, and the calibrated probability distribution is calculated. The highest probability value is used as the confidence score; When the confidence score is lower than a preset threshold, a negative penalty is introduced into the total reward value, and a manual review mechanism is triggered.

[0014] Preferably, the critical reward module is implemented using a fine-tuned large language model, and its input prompts include: a structured description of the multimodal input state, the specific content of the candidate output, and predefined evaluation dimensions. The critical reward module outputs a structured JSON format containing ratings and natural language feedback.

[0015] Technical effects of this technical solution: This invention addresses the problems of existing methods, such as high reliance on human intervention, rigid rules, difficulty in correcting biases, and superficial criticism, by introducing mechanisms such as multi-source reward fusion, pseudo-preference pair generation through criticism, dynamic weight optimization, and closed-loop training. The robot no longer merely executes commands mechanically but possesses the ability to continuously learn, proactively improve, and self-evolve. The main effects are as follows: 1. This paper proposes for the first time to automatically transform criticisms into pseudo-preference pairs, making criticisms not only explanatory feedback, but also a data source that drives the self-evolution of the model.

[0016] 2. Innovatively, Unified Reward (URM), Self-Reward and Confidence Calibration (CSR), and Critical Reward (CBR) are combined into a hierarchical fusion reward function, which not only ensures cross-modal consistency but also provides personalized constraints for different tasks.

[0017] 3. The criticism module makes the reward signal interpretable, so that the results given by the robot are traceable; at the same time, the self-reward calibration mechanism avoids the model's "overconfidence" from reinforcing errors.

[0018] 4. Establish a complete closed loop from candidate generation, reward evaluation, criticism correction to training and updating, so that the robot has the ability to learn while running and adapt to the long-term needs of complex power grid environments.

[0019] 5. By using pseudo-preference pairs and a self-evolutionary mechanism, the reliance on manual annotation and rule base updates is significantly reduced, thereby lowering operation and training costs.

[0020] The second technical solution is a closed-loop reward optimization system based on multimodal human preference alignment, which includes a perception and candidate generation layer, a reward evaluation and criticism layer, and a closed-loop optimization layer. The perception and candidate generation layer receives inputs from multiple sources and generates candidate outputs through the main model. The reward evaluation and criticism layer consists of three parts: the Unified Reward Model (URM), the Critical Reward Module (CBR), and the Self-Reward Module (CSR), which respectively provide cross-modal consistency evaluation, model internal confidence correction, and criticism feedback generation. The closed-loop optimization layer achieves automatic expansion of training data and model iteration through reward function fusion and pseudo-preference pair generation mechanism.

[0021] This technical solution's system framework not only supports real-time feedback during online operation, but also seamlessly integrates with the existing log system, reporting system, and operation monitoring interface of the monitoring robot, forming an adaptive and self-evolving closed-loop system. This loop not only works on experimental datasets but also updates in real time during actual monitoring operations, enabling the robot to gradually break free from its dependence on manual annotation and rule bases, and possess long-term evolution capabilities. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments or prior art, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts of the present invention; Figure 2 This is the second schematic diagram of the process of the present invention. Detailed Implementation

[0024] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0025] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0026] The following section will provide a detailed and systematic explanation of the theoretical foundation and key technologies used in this invention. We will start with the most basic mathematical concepts and gradually explain how each theory is applied to the "intelligent monitoring robot," facilitating an understanding of the entire invention's working mechanism and reliability assurance.

[0027] The fundamental framework to understand first is the intuition of Markov Decision Processes (MDPs) and Reinforcement Learning (RL). View the decision-making of a monitoring robot during operation as a sequential problem: at any given moment, the information "observed" by the robot (screen images, telemetry timings, voice commands, historical logs, etc.) constitutes the environmental state s; the robot selects an action a based on its current policy. For example, generating a log draft, proposing a solution, or issuing a voice prompt. The environment provides an immediate reward r, indicating the merit of the action in terms of business objectives, such as log completeness or the correctness of the solution. Within this framework, the robot seeks the policy that maximizes the cumulative reward over long-term operation. This idea can be expressed mathematically as follows: for a policy π (where the policy is a conditional probability distribution π(a|s)), its state-value function satisfies the Bellman equation: ; Where γ∈(0,1) is the discount factor, representing the weight of future returns. Intuitively, this equation illustrates that "current state value = current immediate return + expected future state value". For engineers, understanding this means that what we need to optimize is long-term performance, not just whether a particular output looks "correct".

[0028] Based on the above framework, one common approach to optimizing policies is the policy gradient method, with its core theorem called the "policy gradient theorem": it derives the performance improvement of the policy as a desired gradient form. ; Where J(θ) is the objective (expected cumulative return) of the policy parameter θ, and This is the action value function. In engineering terms, this formula tells us that when we want to improve a policy, we should update the parameters in a direction that increases the "weighted log probability," where the weights are the value estimates of the actions (better actions are amplified more). In the scenario of surveillance robots, this means that if a generated log or action suggestion proves more useful in real-world operation, the training algorithm will increase the probability of generating that type of output.

[0029] In practical engineering, directly using traditional RL to drive multimodal large language models to learn in industrial scenarios presents two difficulties: first, it requires explicit and frequent real-time numerical reward signals; second, directly using RL to update large generative models suffers from stability and computational cost issues. To address these challenges, two widely adopted alternative approaches have emerged: reward modeling based on preferences (RLHF) + RL, and direct preference optimization (DPO). The underlying concepts and their specific applications in this invention will be explained below.

[0030] Within the RLHF framework, we first construct a reward model using expert-collected or high-quality data. Here, x represents the input, and y represents the output generated by the robot. Reward models are often trained with a "pairwise preference" training objective: given two candidate outputs... (Better) and (Poor) The reward model should satisfy This training loss is often written as log-odds loss: ; in It's the logistic function. After training the reward model, the traditional RLHF approach is to... Using environmental rewards to run RL to update the main model. However, "using RL" has problems with high complexity and unstable convergence. DPO provides a simpler, smaller-scale, but very useful alternative: directly using preference pairs as training signals and using a probability ratio (logit ratio)-based objective to directly optimize the policy. The core objective of DPO can be expressed as: for each preference pair Hopeful Strategy It reflects preference in the logarithmic probability difference, while relative to a certain reference strategy ( Maintain a reasonable distance.

[0031] A common form of DPO aims to maximize: ; The intuitive explanation is: we want to increase the relative preference of the policy for better outputs, but at the same time, we use π_ref as a baseline constraint to prevent the policy from deviating too much from the original language model. In monitoring robots, this method can directly translate expert preferences for log quality and handling solutions into adjustments to the model's generation probability, without requiring expensive online RL environment simulations. The advantage of DPO is that it is simpler to implement and more stable in training, but the disadvantage is that it still relies on pairwise preference data. This is why this invention introduces "automatic conversion of critical annotations into pseudo-preference pairs" to reduce manual costs.

[0032] Next, we will explain the sources and technical details of the three main reward components in our scheme. The first type is the Unified Reward Model (URM), whose core purpose is to measure the performance of candidate outputs in terms of "cross-modal consistency" and "task relevance". A URM typically consists of a multimodal encoder and a small scoring head. The multimodal encoder encodes each modality of the input into a vector representation, and then combines these vectors to calculate a consistency score. Technically, a commonly used approach is to train the modality alignment encoder using contrastive learning and InfoNCE loss. The key formula for InfoNCE is: ; in These are vectors corresponding to the modalities. sim(·,·) is usually the cosine similarity. It's a temperature hyperparameter. This training method brings "true pairs" closer together in the vector space, while squeezing "false pairs" further apart. Applied to monitoring robots, URM can measure multimodal consistency such as "whether the device number mentioned in the log matches the number in the screen OCR" and "whether the intent of the voice command is consistent with the current alarm situation," thus providing an objective score. .

[0033] The second category is the self-reward and confidence calibration module (CSR), which aims to provide a reliable estimate of the "confidence" of the model's output. Simple softmax probabilities cannot represent true uncertainty, therefore calibration and uncertainty estimation techniques are needed. Common calibration methods include temperature scaling, which involves dividing logits by the temperature T before performing softmax. ; Minimizing the negative log-likelihood by selecting an appropriate T on the validation set allows the output probability to more closely approximate the true confidence level. Furthermore, more advanced CSRs incorporate uncertainty estimation techniques. These methods provide "confidence intervals" or entropy information, enabling the robot to reduce its confidence and trigger human intervention when facing unfamiliar or uncertain scenarios. To measure calibration quality, the "Expected Calibration Error (ECE)" is used as an indicator, its approximate calculation form being: ; Here, the output probability is divided into M intervals. is the number of samples in the m-th bucket, and acc and conf are the actual accuracy and average confidence score within the bucket, respectively. In practice, we would use the confidence score as... A higher reward is given when the confidence level is high and consistent with the URM; conversely, if the confidence level is low, the reward is reduced or a manual review is triggered.

[0034] The third type is Critical Reward (CBR), which is one of the key innovations of this invention: we not only provide a score, but also "actionable improvement suggestions." Specifically, we use a model capable of "diagnostic and suggestion outputs" to analyze candidate outputs and generate structured criticism items. These criticisms serve both as explanatory information for operations personnel to understand and as automatically "converted" into pseudo-preference pairs: that is, using criticisms to generate "corrected outputs." This constitutes a new preference pair ( The text is then processed (y), and then added to the training set. Key technologies enabling this step include editable text and structured transformation rules, as well as automated verification strategies to avoid misleading results from low-quality criticism. This process of turning criticism into training data significantly reduces the need for manual annotation and allows the reward model to continuously learn from the robot's actual errors.

[0035] When combining the above three signals into the final numerical reward, we adopt a hierarchical weighted fusion strategy and allow dynamic adjustment of the weights. The total reward is expressed as: ; In engineering, we don't fix α, β, and γ; instead, we dynamically allocate weights using "signal reliability." For example, we can calculate a reliability score for each signal based on its reliability on its short-term validation set, and then use softmax to map the three values ​​to weights. Formally, this can be written as: ; in • is the reliability obtained from the t-th round of evaluation, and λ controls the sensitivity. Thus, it automatically has a high weight when the URM performs well; when the model encounters new equipment that causes a decrease in the URM signal-to-noise ratio, the system can temporarily rely on CSR or manual intervention to ensure robustness.

[0036] Example 1: To verify the effectiveness of this invention, we conducted systematic experiments on the power grid monitoring robot task dataset and two mainstream public multimodal datasets, MS-COCO and MSRVTT. The experimental objective was to comprehensively evaluate the advantages of this invention in multimodal consistency, task accuracy, interpretability, and closed-loop self-evolutionary capability.

[0037] In the power grid monitoring robot scenario, we constructed an experimental set using 120,000 historical monitoring logs, over 50,000 anomaly and alarm signals, and approximately 100 hours of operation monitoring voice data provided by a provincial power company. Based on this, we compared four types of methods: ① traditional rule-based methods; ② the RLHF method relying solely on manual annotation; ③ a single unified reward model (URM); and ④ the closed-loop reward optimization method proposed in this invention. The results are as follows: ; As can be seen, the proposed solution significantly outperforms other methods in log completeness, accuracy of handling suggestions, and error detection rate in operation monitoring, while also exhibiting faster response speed and higher multimodal consistency score. This indicates that the criticism-to-pseudo-preference pair and the multi-source reward fusion mechanism significantly improve the stability and robustness of the intelligent monitoring robot in complex tasks.

[0038] To verify the generality of this scheme, we also conducted comparative experiments on two public multimodal datasets.

[0039] Experiment 1: MS-COCO Image Description Task This task requires the model to generate natural language descriptions based on input images. Evaluation metrics include BLEU-4, METEOR, and CIDEr. Comparison methods include supervised learning baselines, RLHF, DPO, URM, and the proposed solution. Results are as follows: ; The results showed that the present invention improved the CIDEr score by 9.4 points compared to URM, indicating that the generated descriptions were more in line with human preferences, and also improved the BLEU-4 and METEOR scores.

[0040] Experiment 2: MSRVTT Video Question and Answer Task This task requires the model to answer content-related questions after watching a video. The evaluation metrics are Top-1 accuracy and multimodal consistency score. Experimental results are as follows: ; The results show that the present invention improves the Top-1 accuracy by 4.5 percentage points compared with URM and improves the multimodal consistency by 0.06, proving that the present invention performs better in cross-modal fusion and robustness in complex time-series tasks.

[0041] In summary, this invention significantly outperforms mainstream methods such as RLHF, DPO, and URM on both industry-specific power grid monitoring datasets and open public datasets like MS-COCO image description and MSRVTT video question answering. This demonstrates that the proposed mechanism of "multi-source reward fusion + critical pseudo-preference pairs + closed-loop optimization" possesses universality and leading advantages across tasks and modalities, ensuring that the model maintains higher accuracy, consistency, and self-evolution capabilities in different application scenarios.

[0042] It should be noted that the above-mentioned technical features can be combined with each other to form various embodiments not listed above, all of which are considered to be within the scope of this invention specification; and, for those skilled in the art, improvements or modifications can be made based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A closed-loop reward optimization method based on multi-modal human preference alignment, characterized in that; Includes the following steps: S1: Candidate generation step: The intelligent monitoring robot receives multimodal input information, and the main model generates multiple candidate outputs, including anomaly handling suggestions, log drafts, and operation monitoring prompts; S2: Reward evaluation steps: Use a unified reward model to score candidate outputs for cross-modal language consistency and task relevance, use a self-reward and confidence calibration module to score internal confidence, and use a critical reward module to generate natural language criticisms and corresponding negative penalty scores. S3: Reward Fusion Step: The unified reward model score, the self-reward and confidence calibration module score, and the critical reward module score are weighted and fused to obtain the total reward value; S4: Criticism Transformation Steps: Transform the natural language criticisms generated by the criticism reward module into pseudo-preference pairs; S5: Closed-loop update steps: Use the merged total reward value and the transformed pseudo-preference pair data to update the main model parameters.

2. The method of claim 1, wherein: The formula for the total reward value in step S3 is as follows: ； The above formula: R total The total reward value, This represents the unified reward model score, which primarily measures the matching degree between candidate outputs and tasks, as well as cross-modal consistency. The score represents the self-reward and confidence calibration module score, which reflects the robot's confidence estimate and logical consistency of its own output; The critical reward score contains both positive reinforcement signals and negative penalties based on criticism, used to correct potential biases in the model. parameter The validation set can be automatically tuned, or adjustments can be made during operation using a dynamic weighting mechanism.

3. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 2, characterized in that; The dynamic adjustment method for the weight coefficients α, β, γ is as follows: based on the reliability score of each reward source on the short-term validation set, the weights are calculated using the Softmax function, so that reward sources with high reliability have higher weights in the fusion.

4. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 1, characterized in that; The criticism transformation steps specifically include: Use a critical reward module to diagnose defects in candidate outputs; Generate corresponding correction suggestions based on the defects; Improved candidate outputs are automatically generated based on the suggested corrections. ; Compare the original candidate output y with the improved candidate output Constructing pseudo-preference pairs (y, ).

5. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 1, characterized in that; S5: The closed-loop update step adopts the Direct Preference Optimization (DPO) algorithm, which uses the fused reward value and pseudo preferences to calculate the loss function on the data and update the parameters of the main model.

6. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 1, characterized in that; The training method for the unified reward model includes: Construct multimodal contrastive learning samples and encode the input state x and candidate output y into semantic vectors; Define the InfoNCE loss function to shorten the vector distance between positive sample pairs and widen the vector distance between negative sample pairs; The cross-modal language consistency score is calculated using a cosine similarity function.

7. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 1, characterized in that; The workflow of the self-reward and confidence calibration module includes: Temperature scaling calibration is applied to the raw Logits output by the master generator model, and the calibrated probability distribution is calculated. The highest probability value is used as the confidence score; When the confidence score is lower than a preset threshold, a negative penalty is introduced into the total reward value, and a manual review mechanism is triggered.

8. The closed-loop reward optimization method based on multimodal human preference alignment according to claim 1, characterized in that; The critical reward module is implemented using a fine-tuned large language model, and its input prompts include: a structured description of the multimodal input state, the specific content of the candidate output, and predefined evaluation dimensions. The critical reward model outputs a structured JSON format containing scores and natural language feedback.

9. A closed-loop reward optimization system based on multimodal human preference alignment, characterized in that; The closed-loop reward optimization method based on multimodal human preference alignment as described in any one of claims 1-8 includes a perception and candidate generation layer, a reward evaluation and criticism layer, and a closed-loop optimization layer; The perception and candidate generation layer receives inputs from multiple sources and generates candidate outputs through the main model. The reward evaluation and criticism layer consists of three parts: the Unified Reward Model (URM), the Critical Reward Module (CBR), and the Self-Reward and Confidence Calibration Module (CSR), which respectively provide cross-modal consistency evaluation, intra-model confidence correction, and criticism feedback generation. The closed-loop optimization layer achieves automatic expansion of training data and model iteration through reward function fusion and pseudo-preference pair generation mechanism.

Images