Perception consistency constraint and gated fusion multi-modal reasoning reinforcement learning method
By integrating perceptual consistency constraints with gating into a multimodal reasoning reinforcement learning method, the problem of insufficient visual understanding in multimodal reasoning models during reinforcement learning is solved. This method effectively constrains and optimizes visual evidence, thereby improving the reliability and stability of reasoning results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA ELECTRONICS CYBERSPACE RESEARCH INSTITUTE CO LTD
- Filing Date
- 2026-04-13
- Publication Date
- 2026-07-07
AI Technical Summary
Existing multimodal reasoning models struggle to effectively constrain whether they truly understand and utilize image evidence during post-reinforcement learning training, leading to visual illusions and erroneous inferences, which impacts reliability and generalization ability.
A multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating is introduced. By generating candidate reasoning trajectories, perceptual consistency rewards are evaluated based on the consistency between visual description information and reference visual descriptions. These perceptual consistency rewards are then gating and fused with answer consistency rewards to construct intra-group advantage values and optimize model parameters.
It significantly reduces visual illusions, improves the credibility and robustness of multimodal reasoning, and enhances the model's generalization ability in complex scenarios and out-of-distribution data.
Smart Images
Figure CN122347191A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimodal large model technology, specifically a multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating. Background Technology
[0002] Multimodal reasoning generally refers to a type of intelligent ability where a model can understand, associate, deduce, and make decisions when given input from multiple sources of information, such as images and text. Unlike pure text reasoning, multimodal reasoning requires not only linguistic logical deduction capabilities but also that its reasoning process be supported by visual evidence: the model needs to identify objects, attributes, spatial relationships, quantitative information, or diagrammatic structures related to the question from the image, and then combine this with the textual question to deduce and output the answer. Multimodal reasoning tasks are widely used in applications such as visual question answering, diagrammatic understanding, visual mathematics, visual commonsense inference, and multi-step reasoning. The reliability of its output largely depends on the model's true understanding of the image content and its correct use of visual evidence.
[0003] Existing multimodal large-scale models typically employ a "pre-training + supervised alignment / instructional fine-tuning" approach. Pre-training with large-scale image and text data enables alignment of visual and linguistic representations, followed by supervised fine-tuning to adapt the model to downstream tasks such as question answering and reasoning. However, this approach relies on manually labeled reference answers or texts, and its training signals are largely focused on the final output, making it difficult to constrain the correct use of visual evidence during the reasoning process. Furthermore, high-quality annotation of fine-grained visual elements (such as differences in small object attributes, occlusion relationships, spatial layout, counting, OCR text, and chart coordinate reading) is costly, leading to insufficient training data coverage. This makes the model more prone to visual illusions or incorrect inferences in complex scenes or with out-of-distribution data.
[0004] To further enhance reasoning capabilities, the field of multimodal reasoning has recently introduced reinforcement learning-based post-training paradigms. Common approaches can be categorized as reinforcement learning based on verifiable signals (RLVR) optimization methods, such as Group Relative Policy Optimization (GRPO). This paradigm typically involves the model first generating candidate answers, then calculating rewards based on automatic evaluation rules or review mechanisms, and finally optimizing the policy to increase the probability of high-reward outputs. Compared to purely supervised learning, this method can, to some extent, strengthen the model's preference for task objectives and improve answer accuracy, instruction compliance, and output structure stability.
[0005] However, in multimodal reasoning scenarios, current reinforcement learning post-training still faces significant bottlenecks. Many tasks rely primarily on answer-level correctness judgments for rewards, such as whether a multiple-choice question was answered correctly or whether numerical values match. These rewards are outcome-oriented and fail to characterize whether the model truly understands and utilizes visual evidence. The model may "guess correctly" due to data bias or linguistic priors, and even if the final answer is correct, its understanding of key visual objects, attributes, or relationships may still be flawed. As scene complexity increases or out-of-distribution images are encountered, the model's generated reasoning chain may be self-consistent but inconsistent with the visual facts, thus affecting the reliability and generalization ability of multimodal reasoning. Summary of the Invention
[0006] This invention addresses the key technical problem of how to introduce a scalable reward mechanism that is directly related to the quality of visual understanding during the post-training process of reinforcement learning, thereby reducing visual illusions and false attribution phenomena and improving the reliability and generalization ability of multimodal reasoning.
[0007] To address the aforementioned issues, this invention provides a multimodal reasoning reinforcement learning method, learning system, electronic device, and storage medium that integrates perceptual consistency constraints and gating.
[0008] In a first aspect, the present invention provides a multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating, comprising the following steps: Obtain multimodal training samples containing images, text questions, and standard answers; The multimodal training samples are input into the multimodal base model to be optimized to generate multiple candidate reasoning trajectories. Each candidate reasoning trajectory includes visual description information, reasoning process and final answer. A consistency reward is determined based on the consistency between the final answer and the standard answer; Based on the consistency between the visual description information and the preset reference visual description, a perceptual consistency reward is determined. The perceptual consistency reward is gating the answer consistency reward, and the perceptual consistency reward and the gating answer consistency reward are fused to obtain the final reward of the candidate reasoning trajectory. Based on the final reward, an intra-group advantage value is constructed, and the parameters of the multimodal base model are optimized using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
[0009] Optionally, the candidate reasoning trajectory is generated according to a preset output template, which at least requires the visual description information, the reasoning process, and the final answer to be presented in a parsable fixed paragraph structure; when parsing fails, the candidate reasoning trajectory is marked as invalid and resampling is triggered.
[0010] Optionally, determining the answer consistency reward includes: using a rule-based determination method to compare the final answer with the standard answer for consistency; the rule-based determination method includes one or more of exact matching, numerical matching, or set matching.
[0011] Optionally, determining the reward for perceptual consistency includes: Construct for the image Each reference visual description contains at least one structured factual information, including object, attribute, spatial relationship, or quantity information; A judgment model is used to evaluate the consistency between the visual description information and each reference visual description, resulting in... Individual perception score; Regarding the The perceptual consistency reward is obtained by truncating and aggregating the individual perception scores.
[0012] Optionally, the truncated mean aggregation includes: setting a truncation ratio. Remove the minimum and maximum values from the score sequence. The average of the remaining intermediate scores is taken as the perceptual consistency reward, where... .
[0013] Optionally, gating the answer consistency reward based on the perceived consistency reward includes: Define a perception gating function, which is used when the perception consistency reward is lower than a preset threshold. When the gated output is zero, the perceptual consistency reward is not lower than a preset threshold. At that time, the gated output increases monotonically as the reward for perceptual consistency increases.
[0014] Optionally, the answer consistency reward fused with the perceived consistency reward and the gating constraint includes: The adaptive fusion weight is dynamically calculated based on the perceived consistency reward. , to make the fusion weight The reward for perceptual consistency shows a monotonically increasing relationship. The final reward is represented as: ; in Rewards are given for consistent answers. For perceived consistency rewards For sensing gate functions.
[0015] Optionally, constructing intra-group dominance values includes: Generated from the same multimodal training samples The final reward set of the candidate inference trajectories is filtered to ensure that there is at least one higher reward sample and one lower reward sample in the set; Calculate the quantile baseline of the reward set: , Indicates quantiles; The advantage value of each candidate inference trajectory is defined as: .
[0016] Optionally, optimizing the parameters of the multimodal base model using a reinforcement learning algorithm includes: updating the model parameters by using a group relative strategy to optimize the objective function, which is optimized based on the within-group dominance value, probability ratio pruning term, and sampling strategy.
[0017] Secondly, embodiments of the present invention provide a structural block diagram of a multimodal reasoning reinforcement learning system that integrates perceptual consistency constraints and gating, the system comprising: The sample acquisition module 101 is used to acquire multimodal training samples containing images, text questions, and standard answers; The trajectory generation module 102 is used to input the multimodal training samples into the multimodal base model to be optimized and generate multiple candidate inference trajectories. Each candidate inference trajectory includes visual description information, inference process and final answer. The answer reward determination module 103 is used to determine the answer consistency reward based on the consistency between the final answer and the standard answer; The perceptual reward determination module 104 is used to determine the perceptual consistency reward based on the consistency between the visual description information and the preset reference visual description. The reward fusion module 105 is used to perform gating constraints on the answer consistency reward based on the perceptual consistency reward, and fuse the perceptual consistency reward and the gating constraint-constrained answer consistency reward to obtain the final reward of the candidate reasoning trajectory; The model optimization module 106 is used to construct intra-group advantage values based on the final reward and to optimize the parameters of the multimodal base model using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
[0018] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.
[0019] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal reasoning reinforcement learning method of perceptual consistency constraint and gating fusion as described in the first aspect.
[0020] The beneficial effects of the multimodal reasoning reinforcement learning method that fuses perceptual consistency constraints and gating in this invention are as follows: This invention introduces perceptual consistency rewards and fused them with answer rewards through gating. This achieves explicit constraints and priority optimization of the model's visual understanding ability during reinforcement learning optimization, significantly reducing the phenomenon of "the answer happens to be correct but the visual fact is wrong" caused by visual perception errors in multimodal reasoning, and effectively improving the credibility and robustness of the reasoning results. At the same time, by adopting an adaptive fusion weight and quantile baseline advantage construction mechanism based on perceptual consistency rewards, it can dynamically balance the optimization priority of perceptual alignment and answer correctness during training, suppress the interference of abnormal samples on gradient updates, enhance the stability and convergence efficiency of the training process, thereby achieving better performance on multiple multimodal reasoning benchmarks and improving the model's generalization ability on complex scenarios and out-of-distribution data. Attached Figure Description
[0021] Figure 1 This is a flowchart of the multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating in an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating in an embodiment of the present invention. Figure 3 This is a diagram of the model optimization framework in an embodiment of the present invention; Figure 4 This is a structural diagram of the reward model in an embodiment of the present invention; Figure 5 This is a structural block diagram of the multimodal reasoning reinforcement learning system that integrates perceptual consistency constraints and gating in an embodiment of the present invention; Figure 6 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation
[0022] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.
[0023] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.
[0024] like Figure 1 As shown in the figure, a multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating, provided in this embodiment of the invention, includes the following steps: S1. Obtain multimodal training samples containing images, text questions, and standard answers; For example, from a multimodal dataset Get from There are 1 input sample, each containing an image. Text problem and the standard answer .
[0025] S2. Input the multimodal training samples into the multimodal base model to be optimized to generate multiple candidate reasoning trajectories. Each candidate reasoning trajectory includes visual description information, reasoning process and final answer. Specifically, image-text pairs Input to the base model Generate using a preset decoding strategy candidate outputs ; in This represents the strategy parameters used for sampling in the current iteration. In this embodiment, Take 10.
[0026] Candidate output The output is generated according to a preset output template, which requires that the candidate outputs at least include: Visual description information (Used to express verifiable facts in the image that are relevant to the question); Reasoning process based on visual description and questions; Final Answer .
[0027] In this embodiment, the output template adopts a structured format (fixed-label paragraphs) to facilitate subsequent parsing and consistency evaluation; when parsing fails, the candidate output is regarded as an invalid sample and resampling is triggered during the sampling phase.
[0028] S3. Determine the answer consistency reward based on the consistency between the final answer and the standard answer; The consistency reward for determining the answer includes: comparing the final answer with the standard answer using a rule-based determination method; the rule-based determination method includes one or more of exact matching, numerical matching, or set matching.
[0029] This embodiment uses a rule-based judgment method to perform verifiable consistency judgment on the final answer of the candidate output and allocate a binary answer reward.
[0030] Specifically, from candidate outputs Extract the final answer In this embodiment, parsing is performed based on the answer tags of the output template; if parsing fails, ... This answer is considered invalid.
[0031] A rule-based decision-making method is used to compare candidate answers. Compared with the standard answer Consistency of answers, and allocate consistency rewards accordingly: ; in, The consistency determination function can be implemented using different rules depending on the task type, including: Exact matching: Comparison after standardization for tasks such as classification and multiple selection; Numerical matching: For tasks involving numerical values, counting, and reading, within a preset tolerance error range. Internal determination is consistent; Set matching: For tasks with multiple answer sets, determine whether they are consistent or inconsistent according to the set consistency rule.
[0032] In this embodiment, Consistency determination is performed using exact matching rules.
[0033] S4. Determine the perceptual consistency reward based on the consistency between the visual description information and the preset reference visual description; The rewards for perceptual consistency include: Construct for the image Each reference visual description contains at least one structured factual information, including object, attribute, spatial relationship, or quantity information; A judgment model is used to evaluate the consistency between the visual description information and each reference visual description, resulting in... Individual perception score; Regarding the The perceptual consistency reward is obtained by truncating and aggregating the individual perception scores.
[0034] Furthermore, the truncated mean aggregation includes: setting a truncation ratio. Remove the minimum and maximum values from the score sequence. The average of the remaining intermediate scores is taken as the perceptual consistency reward, where... .
[0035] Gating constraints on the answer consistency reward based on the perceived consistency reward include: Define a perception gating function, which is used when the perception consistency reward is lower than a preset threshold. When the gated output is zero, the perceptual consistency reward is not lower than a preset threshold. At that time, the gated output increases monotonically as the reward for perceptual consistency increases.
[0036] For each input sample image , build Reference visual description Each reference visual description It should at least include descriptive information about the facts of the image, including structured factual information such as objects, attributes, spatial relationships, quantity information, and local text or chart readings. In this embodiment, the reference visual description is generated by a higher-performance multimodal model.
[0037] Adopting a judgment model Candidate visual description Compared with reference visual description A consistency assessment was conducted to obtain a single-perception score: ; in, The larger the value, the higher the consistency between the candidate visual description and the reference description.
[0038] To reduce the impact of extreme ratings on perceived reward, a robust aggregation of multiple single-perceptual ratings from the same sample is performed using the truncated mean to obtain a perceptual consistency reward. .
[0039] Specifically, first sort the multiple single-perception scores from smallest to largest: ; Let the truncation ratio be... ,make Remove the smallest Each rating and the largest After each rating, the middle The average of the scores is used to obtain the perceptual consistency reward: ; In this embodiment, take .
[0040] S5. Gating constraints are applied to the answer consistency reward based on the perceptual consistency reward, and the perceptual consistency reward and the gating constraint-constrained answer consistency reward are merged to obtain the final reward of the candidate reasoning trajectory; The answer consistency reward, which combines the perceptual consistency reward with the gating constraint, includes: The adaptive fusion weight is dynamically calculated based on the perceived consistency reward. , to make the fusion weight The reward for perceptual consistency shows a monotonically increasing relationship.
[0041] Specifically, define the perception gate function. This is used to suppress the contribution of answer reward when perceptual consistency is insufficient: ; in This is the perception gating threshold. Therefore, when perception consistency is rewarded... When the value is less than the threshold, the gated output is 0; when the value is perceptually consistent, the reward is... When the threshold is not less than 1, the gated output is affected by the perceptual consistency reward. It increases monotonically. In this embodiment, .
[0042] The fusion weight coefficients are set to adapt to perceptual reward to achieve dynamic weighting of candidate outputs. Specifically, candidate outputs are defined. The corresponding fusion weight coefficients are: ; in, and , It is a monotonically increasing mapping function. In this embodiment, , , .
[0043] Answer-based rewards Perceptual reward Sensing gate function and adaptive fusion weight coefficients Calculate candidate outputs The final reward for: ; in, Rewards are given for consistent answers. For perceived consistency rewards For sensing gate functions. , ,and Therefore, when hour, Even if the answer reward is 1, the answer choice does not affect the final reward, thus suppressing unreliable updates of "perceptually inconsistent but correct answers". (Fusing weight coefficients) Perceptual reward based on candidate output Obtained through adaptive calculation, and It exhibits a monotonically increasing relationship. When When the weight is low, the system reduces the weight of the answer channel to prioritize correcting perceptual bias; when When the value is high, the system increases the weight of the answer channel to enhance the optimization of the correctness of the final answer, thereby realizing dynamic reward allocation under the perception priority constraint.
[0044] S6. Construct intra-group advantage values based on the final reward, and optimize the parameters of the multimodal base model using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
[0045] The dominant values within the construction group include: Generated from the same multimodal training samples The final reward set of the candidate inference trajectories is filtered to ensure that there is at least one higher reward sample and one lower reward sample in the set; Calculate the quantile baseline of the reward set; The advantage value of each candidate inference trajectory is defined as: .
[0046] Optimizing the parameters of the multimodal base model using reinforcement learning algorithms includes: updating the model parameters by using a group relative strategy to optimize the objective function, which is optimized based on the within-group dominance value, probability ratio pruning term, and sampling strategy.
[0047] Specifically, for the same input The generated candidate output set Filtering is performed to ensure that there are valid differences in rewards within the set (there is at least one sample with a higher reward and at least one sample with a lower reward) to guarantee that the relative advantage construction is effective.
[0048] Final reward set for candidate outputs Calculate the quantile baseline: ; in, The preset quantile is set to 0.5 (median) in this embodiment.
[0049] Define the within-group advantage for each candidate output. .
[0050] Then, policy optimization is performed based on reinforcement learning (RLVR) of verifiable signals.
[0051] The target is updated using a group-relative strategy to optimize the model parameters: ; ; In this embodiment, It's a probability ratio. It is the importance sampling clipping range. , .
[0052] Finally, iterative training is performed, repeating the above steps until the stopping conditions are met. The stopping conditions include the number of training rounds reaching a threshold, convergence of the validation set metrics, and stability of the reward statistics.
[0053] After obtaining the optimized model, multimodal inference is performed on the input image and text to obtain the inference result. Compared with the original model, this embodiment improves the accuracy and reliability of inference by suppressing the positive impact of unreliable samples on model updates through perceptual gating and adaptive fusion weights.
[0054] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. In the reinforcement learning optimization framework based on verifiable answer signals, this invention further introduces a perceptual consistency evaluation for visual description fragments in candidate outputs and controls the reward contribution through a perceptual gating function. This ensures that when the candidate output does not fully understand the image facts, even if the final answer is coincidentally consistent, its positive learning signal will be suppressed or ineffective. This significantly reduces the problems of "the answer happens to be correct but the visual facts are wrong" and visual illusions caused by inconsistencies between the reasoning process and image evidence, thereby improving the credibility and stability of multimodal reasoning outputs.
[0055] 2. This invention transforms the consistency constraints of fine-grained visual facts such as objects, attributes, spatial relationships, quantity information, and local text or chart readings into continuous and computable perceptual reward signals. It also supports robust aggregation of multiple reference descriptions (e.g., truncated mean). Without the need for large-scale manual annotation of each inference chain, it achieves effective supervision of the model's "use of visual evidence," reduces the cost of training data construction, and enhances the transferability and scalability of the method to different tasks and data sources.
[0056] 3. This invention introduces adaptive fusion weights in the final reward calculation, enabling the reward to dynamically adjust according to the perceptual consistency of candidate outputs during training: when perceptual consistency is low, the reward is more inclined to promote the alignment of the model's visual description with the image facts; as perceptual consistency gradually increases, the contribution of the answer reward gradually strengthens, thus forming an optimization path that smoothly migrates from "perceptual alignment priority" to "answer correctness enhancement". This mechanism can reduce misleading updates caused by noisy scores or accidental consistency, improve the stability and effectiveness of RLVR training, and enhance the model's generalization performance in complex scenes and out-of-distribution samples.
[0057] 4. This invention employs a quantile baseline to construct relative advantages within groups and combines this with a group relative strategy to optimize the objective for updates. Compared to the mean baseline, this method better suppresses the interference of extreme samples on the learning signal, improving the noise resistance and convergence stability of the training process. The optimized model obtained based on this training method achieves superior or more competitive performance on multiple multimodal inference evaluation datasets. Experimental results are shown in Table 1.
[0058] Table 1: Accuracy (%) metrics across multiple datasets
[0059] In summary, this embodiment provides a perceptual consistency-constrained multimodal reasoning reinforcement learning optimization method applicable to multimodal reasoning tasks involving both images and text. The method first feeds the input multimodal samples into a base model to generate multiple candidate reasoning trajectories. Then, a rule-based judgment method is used to determine the consistency between the final answer of the candidate output and the standard answer, obtaining an answer reward. Simultaneously, multiple reference visual descriptions for the input image are constructed, and a judgment model is used to score the consistency between the visual descriptions in the candidate output and the reference visual descriptions, ultimately obtaining a perceptual reward through the truncated mean.
[0060] Building upon this foundation, this embodiment introduces a perceptual gating function and adaptive fusion weights to fuse and calculate the answer reward and the perceptual reward. This ensures that when the perceptual consistency of candidate outputs is insufficient, the contribution of the answer reward to the final reward is suppressed or ineffective; as perceptual consistency gradually improves, the contribution of the answer reward gradually increases, thereby achieving a training orientation of "first perceptual alignment, then reinforcing correct answer." Furthermore, this embodiment constructs intra-group relative advantages based on the candidate output set corresponding to the same input and uses a quantile baseline instead of a mean baseline to reduce the interference of extreme reward samples on advantage estimation and improve the robustness and stability of the training process.
[0061] Finally, this embodiment employs a reinforcement learning (RLVR) training method based on verifiable signals to optimize the base model, resulting in an optimized multimodal reasoning model. The obtained model can be used for reasoning outputs in similar multimodal reasoning tasks, improving the consistency between the model output and image facts in various scenarios, enhancing the reliability and stability of reasoning results, and to some extent improving reasoning accuracy.
[0062] This invention also provides a multimodal reasoning reinforcement learning system that integrates perceptual consistency constraints and gating. This system is used to implement the above-described method embodiments, and details already described will not be repeated. The terms "module," "unit," and "subunit" used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.
[0063] like Figure 5 As shown, Figure 5 This is a block diagram of the multimodal reasoning reinforcement learning system that integrates perceptual consistency constraints and gating in this invention. The system includes: The sample acquisition module 101 is used to acquire multimodal training samples containing images, text questions, and standard answers; The trajectory generation module 102 is used to input the multimodal training samples into the multimodal base model to be optimized and generate multiple candidate inference trajectories. Each candidate inference trajectory includes visual description information, inference process and final answer. The answer reward determination module 103 is used to determine the answer consistency reward based on the consistency between the final answer and the standard answer; The perceptual reward determination module 104 is used to determine the perceptual consistency reward based on the consistency between the visual description information and the preset reference visual description. The reward fusion module 105 is used to perform gating constraints on the answer consistency reward based on the perceptual consistency reward, and fuse the perceptual consistency reward and the gating constraint-constrained answer consistency reward to obtain the final reward of the candidate reasoning trajectory; The model optimization module 106 is used to construct intra-group advantage values based on the final reward and to optimize the parameters of the multimodal base model using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
[0064] like Figure 6 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: S1. Obtain multimodal training samples containing images, text questions, and standard answers; S2. Input the multimodal training samples into the multimodal base model to be optimized to generate multiple candidate reasoning trajectories. Each candidate reasoning trajectory includes visual description information, reasoning process and final answer. S3. Determine the answer consistency reward based on the consistency between the final answer and the standard answer; S4. Determine the perceptual consistency reward based on the consistency between the visual description information and the preset reference visual description; S5. Gating constraints are applied to the answer consistency reward based on the perceptual consistency reward, and the perceptual consistency reward and the gating constraint-constrained answer consistency reward are merged to obtain the final reward of the candidate reasoning trajectory; S6. Construct intra-group advantage values based on the final reward, and optimize the parameters of the multimodal base model using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
[0065] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0066] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.
[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal reasoning reinforcement learning method that integrates perceptual consistency constraints and gating, characterized in that, Includes the following steps: Obtain multimodal training samples containing images, text questions, and standard answers; The multimodal training samples are input into the multimodal base model to be optimized to generate multiple candidate reasoning trajectories. Each candidate reasoning trajectory includes visual description information, reasoning process and final answer. A consistency reward is determined based on the consistency between the final answer and the standard answer; Based on the consistency between the visual description information and the preset reference visual description, a perceptual consistency reward is determined. The perceptual consistency reward is gating the answer consistency reward, and the perceptual consistency reward and the gating answer consistency reward are fused to obtain the final reward of the candidate reasoning trajectory. Based on the final reward, an intra-group advantage value is constructed, and the parameters of the multimodal base model are optimized using a reinforcement learning algorithm to obtain an optimized multimodal inference model.
2. The method according to claim 1, characterized in that, The candidate reasoning trajectory is generated according to a preset output template. The output template requires that the visual description information, the reasoning process, and the final answer be presented in a parsable fixed paragraph structure. When parsing fails, the candidate reasoning trajectory is marked as invalid and resampling is triggered.
3. The method according to claim 1, characterized in that, The consistency reward for determining the answer includes: comparing the final answer with the standard answer using a rule-based determination method; the rule-based determination method includes one or more of exact matching, numerical matching, or set matching.
4. The method according to claim 1, characterized in that, The rewards for perceptual consistency include: Construct for the image Each reference visual description contains at least one structured factual information, including object, attribute, spatial relationship, or quantity information; A judgment model is used to evaluate the consistency between the visual description information and each reference visual description, resulting in... Individual perception score; Regarding the The perceptual consistency reward is obtained by truncating and aggregating the individual perception scores.
5. The method according to claim 4, characterized in that, The truncated mean aggregation includes: setting a truncation ratio. Remove the minimum and maximum values from the score sequence. The average of the remaining intermediate scores is taken as the perceptual consistency reward, where... .
6. The method according to claim 1, characterized in that, Gating constraints on the answer consistency reward based on the perceived consistency reward include: Define a perception gating function, which is used when the perception consistency reward is lower than a preset threshold. When the perceived consistency reward is not lower than a preset threshold, the gated output is zero. At that time, the gated output increases monotonically as the reward for perceptual consistency increases.
7. The method according to claim 1 or 6, characterized in that, The answer consistency reward, which combines the perceptual consistency reward with the gating constraint, includes: The adaptive fusion weight is dynamically calculated based on the perceived consistency reward. ,make The reward for perceptual consistency shows a monotonically increasing relationship. The final reward is represented as: ； in Rewards are given for consistent answers. For perceived consistency rewards For sensing gate functions.
8. The method according to claim 1, characterized in that, The dominant values within the construction group include: Generated from the same multimodal training samples The final reward set of the candidate inference trajectories is filtered to ensure that there is at least one higher reward sample and one lower reward sample in the set; Calculate the quantile baseline of the reward set: , Indicates quantiles; The advantage value of each candidate inference trajectory is defined as: .
9. The method according to claim 1, characterized in that, Optimizing the parameters of the multimodal base model using reinforcement learning algorithms includes: updating the model parameters by using a group relative strategy to optimize the objective function, which is optimized based on the within-group dominance value, probability ratio pruning term, and sampling strategy.
10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the multimodal reasoning reinforcement learning method that combines perceptual consistency constraints and gating as described in any one of claims 1 to 9.