Multi-modal large model reasoning enhancement method and system

Abstract

The invention provides a reasoning enhancement method and system for a multi-modal large model. The method comprises the following steps: generating thinking chain reasoning data through self-sampling of the multi-modal large model; the thinking chain reasoning data comprises a correct answer sample and a wrong answer sample for the question; based on the thinking chain reasoning data, basic reasoning enhancement training is carried out on the multi-modal large model, and a basic reasoning enhanced model is obtained; the basic reasoning is used for enhancing the accuracy of correct answer generation of the multi-modal large model; performing process reasoning enhancement training on the model after basic reasoning enhancement by adopting a group relative strategy optimization algorithm to obtain a final reasoning enhanced multi-modal large model; the process reasoning enhancement is to enhance the thinking chain quality of the multi-modal large model; the group relative strategy optimization algorithm optimizes the model by considering thinking chain quality and intra-group relative rewards with correct answers. According to the method, the model reasoning accuracy and the thinking chain generation quality can be improved at the same time.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for enhancing reasoning in a multimodal large model. Background Technology

[0002] With the continuous advancement of large-scale language models, multimodal large-scale language models have demonstrated outstanding performance in various open multimodal understanding tasks in recent years. These models possess the ability to simultaneously parse image and text information, opening up new avenues for solving complex image-text question answering problems. Nevertheless, existing question answering systems still face some challenges when dealing with complex problems requiring multi-step reasoning. They often struggle to generate both clear and accurate reasoning paths and answers. Summary of the Invention

[0003] The present invention aims to solve the problems existing in the prior art and provide a reasoning enhancement method and system for multimodal large models, which can simultaneously improve the accuracy of model reasoning and the quality of thought chain generation.

[0004] This invention provides a reasoning enhancement method for a multimodal large model, comprising: generating thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data including correct answer samples and incorrect answer samples for a question; performing basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data to obtain a model with basic reasoning enhancement; the basic reasoning enhancement aims to improve the accuracy of correct answer generation of the multimodal large model; employing a group relative policy optimization algorithm to perform process reasoning enhancement training on the model with basic reasoning enhancement to obtain a multimodal large model with final reasoning enhancement; the process reasoning enhancement aims to improve the quality of the thought chain of the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers.

[0005] According to the present invention, a method for enhancing the reasoning of a multimodal large model includes the following steps: First, based on the thought chain reasoning data, the multimodal large model undergoes basic reasoning enhancement training. This includes: based on the correct answer samples, using a supervised fine-tuning algorithm, gradually correcting errors in the multimodal large model's reasoning through gradient updates to generate correct answers using thought chain reasoning; second, based on the incorrect answer samples, using a contrastive learning method, enabling the multimodal large model to identify and learn from incorrect answers during the simultaneous learning of correct and incorrect reasoning, thereby deriving correct answers.

[0006] According to the present invention, a method for enhancing the reasoning of a multimodal large model includes training the basic reasoning-enhanced model with process reasoning enhancement using a group relative policy optimization algorithm to obtain a final reasoning-enhanced multimodal large model. The method comprises: sampling a set of answers for a question using a sampling model; the sampling model being the basic reasoning-enhanced model from the previous time step; evaluating each sampled answer using a reward function to obtain a corresponding reward value; normalizing the reward value of each sampled answer to obtain a relative advantage; updating the parameters of the basic reasoning-enhanced model at the current time step based on the relative advantage; and repeatedly iteratively executing the sampling step, the evaluation step, the relative advantage step, and the parameter update step until a preset number of iterations are reached to obtain the final reasoning-enhanced multimodal large model.

[0007] According to the inference enhancement method for a multimodal large model provided by the present invention, before updating the parameters of the basic inference-enhanced model at the current moment based on the relative advantage, the method further includes: controlling the parameter update of the basic inference-enhanced model through KL divergence constraints.

[0008] According to the present invention, a reasoning enhancement method for a multimodal large model is provided, wherein the reward function includes an answer correctness function, a format consistency function, and a reasoning step correctness function; the answer correctness function is used to measure the consistency between the model's output answer and the standard answer; the format consistency function is used to measure the consistency between the format of the model's output result and the preset format; and the reasoning step correctness function is used to measure the correctness of the thought chain reasoning process.

[0009] According to the inference enhancement method for a multimodal large model provided by the present invention, the objective function of the group relative policy optimization algorithm is: , The formula for calculating the relative advantage is: , in, Let the objective function of the group relative strategy optimization algorithm be . For model parameters, To balance the hyperparameters of the KL divergence term, To control the hyperparameters of the model update magnitude, For the clipping function, The advantage value is calculated based solely on the relative reward output from samples within each group. For sampling models, The model is an enhancement of the basic reasoning at the current moment. Number of samples per group The average reward for all candidate outputs. Let the standard deviation of the reward for all candidate outputs be . This is the score given by the reward function to each sample within the group. As a preset reference model, To calculate the KL divergence between the base inference-enhanced model and the preset reference model at the current time, For the question, For the first i One answer, The importance sampling coefficient.

[0010] This invention also provides a reasoning enhancement system for a multimodal large model, comprising: a reasoning data generation module, used to generate thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a question; a basic enhancement module, used to perform basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data, to obtain a model with basic reasoning enhancement; the basic reasoning enhancement is to improve the accuracy of correct answer generation of the multimodal large model; and a process enhancement module, used to perform process reasoning enhancement training on the model with basic reasoning enhancement using a group relative policy optimization algorithm, to obtain a multimodal large model with final reasoning enhancement; the process reasoning enhancement is to improve the quality of the thought chain of the multimodal large model; the group relative policy optimization algorithm optimizes the model by taking into account both the quality of the thought chain and the intra-group relative reward for correct answers.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the inference enhancement method for multimodal large models as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the inference enhancement method for multimodal large models as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the inference enhancement method for multimodal large models as described above.

[0014] This invention provides a reasoning enhancement method and system for a multimodal large model. The method includes: generating thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a given question; performing basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data to obtain a model with basic reasoning enhancement; basic reasoning enhancement aims to improve the accuracy of correct answer generation in the multimodal large model; employing a group relative policy optimization algorithm to perform process reasoning enhancement training on the model with basic reasoning enhancement to obtain a final reasoning-enhanced multimodal large model; process reasoning enhancement aims to improve the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers. The method of this invention can simultaneously improve the model's reasoning accuracy and the quality of thought chain generation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a method for enhancing inference in a multimodal large model provided by the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the principle of a multimodal large model inference enhancement method provided by the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of a multimodal large-scale model inference enhancement system provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Multimodal large models have made significant progress in the field of artificial intelligence and have shown broad application prospects. These models can process and fuse information from different modalities, such as text, images, and sound, thereby exhibiting higher-level intelligence in various tasks. However, existing question-answering systems still have limitations when dealing with complex questions requiring multi-step reasoning, often failing to provide clear reasoning processes and accurate answers. Specific problems include: for complex questions requiring multi-step reasoning, existing question-answering systems often struggle to provide reasonable reasoning processes; due to the lack of clear reasoning steps, the generated answers may contain logical errors or be irrelevant to the question.

[0022] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for enhancing inference in a multimodal large model provided by the present invention.

[0023] To address the technical problems existing in the prior art, this invention provides a method for enhancing inference in multimodal large models, comprising: 101: Generate thought chain reasoning data through multimodal large model self-sampling; the thought chain reasoning data includes correct answer samples and incorrect answer samples for the question; 102: Based on the reasoning data of the thinking chain, the basic reasoning enhancement training of the multimodal large model is carried out to obtain the model with basic reasoning enhancement; the basic reasoning enhancement is to improve the accuracy of the correct answer generation of the multimodal large model; 103: The group relative strategy optimization algorithm is used to train the model with enhanced process reasoning after basic reasoning enhancement, resulting in a multimodal large model with enhanced reasoning. Process reasoning enhancement is used to improve the quality of the thought chain of the multimodal large model. The group relative strategy optimization algorithm optimizes the model by taking into account both the quality of the thought chain and the relative reward within the group for correct answers.

[0024] This invention provides a reasoning enhancement method for large multimodal models, aiming to improve their performance in complex reasoning tasks. This method achieves efficient training without relying on additional manually labeled multimodal thought chain data and significantly reduces the perceptual error rate in multimodal tasks. Addressing the two core problems faced by large multimodal models—the scarcity of high-quality training data and the high perceptual error rate during reasoning—this invention first generates thought chain reasoning data through model self-sampling. By fully utilizing correct and incorrect answer samples, a self-learning method is employed to perform basic reasoning enhancement self-training on the large multimodal model. This effectively improves the model's reasoning ability, allowing the model to further improve the accuracy of generated answers even under noisy data conditions, and laying the foundation for subsequent reinforcement learning optimization processes.

[0025] Considering that generating data using multimodal models struggles to overcome the limitations of the models themselves, and that generated data often contains numerous illusions (i.e., incorrect reasoning processes leading to correct results), this restricts the upper limit of improving reasoning ability when training models with synthetic data. This invention addresses this by employing a group relative policy optimization algorithm to enhance the process reasoning of a model after basic reasoning enhancement, resulting in a final, enhanced multimodal model. The group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers, simultaneously improving the model's reasoning accuracy and the quality of thought chain generation during reinforcement learning, effectively mitigating the illusion problem. This invention significantly reduces data annotation requirements, substantially improves training efficiency, and is universally applicable to various multimodal task scenarios where data is scarce.

[0026] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of a multimodal large-scale model inference enhancement method provided by the present invention.

[0027] As a preferred embodiment, the multimodal large model is subjected to basic reasoning enhancement training based on the thought chain reasoning data, including: based on the correct answer samples, using a supervised fine-tuning algorithm, the errors in the reasoning of the multimodal large model are gradually corrected through gradient updates to generate correct answers using thought chain reasoning; based on the incorrect answer samples, using a contrastive learning method, the multimodal large model is able to identify incorrect answers and learn from incorrect answers during the process of learning both correct and incorrect reasoning to deduce the correct answer.

[0028] In this embodiment, the model is trained using correct answer samples under supervised supervision. Specifically, a set of samples containing correct answers is selected from the thought chain reasoning data; these samples have been validated through the thought chain reasoning process. Then, using a supervised fine-tuning algorithm, the model parameters are updated by calculating the gradient of the loss function to minimize the difference between the predicted and true answers. This process involves the backpropagation algorithm, where the loss function is typically chosen as cross-entropy loss to evaluate the inconsistency between the probability distribution of the model's output and the true labels. In this way, after the model training converges, it can generate accurate answers using thought chain reasoning.

[0029] Furthermore, this embodiment also includes comparative learning based on incorrect answer samples. A set of samples containing incorrect answers is selected from the thought chain reasoning data; these samples represent errors the model may make during reasoning. By mimicking the A / B comparison learning method used in human learning, the model learns both correct and incorrect reasoning steps simultaneously. This can be achieved by constructing a loss function that considers not only the difference between the correct answer and the model's prediction, but also the difference between the incorrect answer and the model's prediction. In this way, the model is given the ability to identify and learn from errors. The model is trained to identify which reasoning steps lead to incorrect answers and learn from them to avoid repeating the same mistakes in future reasoning. During reasoning, the model can discover the correct result autonomously through multiple attempts without additional human intervention.

[0030] As a preferred embodiment, a group relative strategy optimization algorithm is used to train the basic reasoning-enhanced model for process reasoning enhancement, resulting in a final reasoning-enhanced multimodal large model. This includes: sampling a set of answers for the question using a sampling model; the sampling model being the basic reasoning-enhanced model from the previous time step; evaluating each sampled answer using a reward function to obtain a corresponding reward value; normalizing the reward value of each sampled answer to obtain a relative advantage; updating the parameters of the basic reasoning-enhanced model at the current time step based on the relative advantage; and repeatedly iterating through the sampling, evaluation, relative advantage, and parameter update steps until a preset number of iterations is reached to obtain the final reasoning-enhanced multimodal large model.

[0031] As a preferred embodiment, before updating the parameters of the basic inference-enhanced model at the current moment based on relative advantage, the method further includes: controlling the parameter update of the basic inference-enhanced model through KL divergence constraints.

[0032] As a preferred embodiment, the objective function of the group-relative strategy optimization algorithm is: , The formula for calculating relative advantage is: , in, Let the objective function be the group relative policy optimization algorithm. For model parameters, To balance the hyperparameters of the KL divergence term, To control the hyperparameters of the model update magnitude, For the clipping function, The advantage value is calculated based solely on the relative reward output from samples within each group. For sampling models, The model is an enhancement of the basic reasoning at the current moment. Number of samples per group The average reward for all candidate outputs. Let the standard deviation of the reward for all candidate outputs be . This is the score given by the reward function to each sample within the group. As a preset reference model, To calculate the KL divergence between the base inference-enhanced model and the preset reference model at the current time, For the question, For the first i One answer, The importance sampling coefficient.

[0033] After improving the model's basic reasoning ability, a thought chain reasoning evaluation index was further designed, and the GRPO (Group Relative Policy Optimization) algorithm was used to optimize and train the model. To maintain the stability and efficiency of policy updates, the core idea of ​​the group relative policy optimization algorithm in this invention is to optimize the policy model (the model enhanced by basic reasoning) through relative rewards within groups. The goal of GRPO is to maximize the expected cumulative reward while maintaining the stability of policy updates. Specifically, GRPO samples a set of actions in each state and then adjusts the policy based on the relative performance of these actions. The GRPO algorithm process can be divided into several key steps, which work together to achieve efficient optimization of the policy model.

[0034] In each round of GRPO updates, the sampled model first considers the problem (state). Sample the answers (actions) of a group. The sampling of these answers was based on the probability distribution of the sampling model, ensuring diversity.

[0035] Each sampled answer is evaluated using a reward function to obtain a corresponding reward value. The reward value of each sampled answer is then normalized to obtain a relative advantage (through the first...). i The relative advantage of each sample is calculated using the formula. GRPO calculates the average reward value of the candidate set for the same problem, and uses the difference between this average and the reward value of the current data as the advantage value estimate for the current data, simplifying the advantage value calculation steps and thus improving training efficiency.

[0036] Update the parameters of the base inference-enhanced model at the current time step based on the calculated relative advantage. ,then As the sampling model for the next round New data is sampled for training. The goal of the update is to increase the probability of answers with a positive relative advantage while decreasing the probability of answers with a negative relative advantage.

[0037] To prevent overly drastic policy updates, GRPO introduces a KL divergence constraint during the update process. By limiting the KL divergence between the old and new policies (the sampled model and the model enhanced from the base inference at the current time step), the change in policy distribution is ensured to remain within a controllable range. The introduction of the KL divergence term makes GRPO more robust, preventing overfitting in complex tasks and maintaining good generalization ability. This reinforcement learning process of multiple rounds of sampling, optimization, resampling, and re-optimization can significantly improve the model's performance on downstream tasks.

[0038] in, Indicates the current training model and sampling model The probability difference for the current sample. and These are used to limit model differences in importance sampling and adjust the proportion of KL divergence, respectively, to improve the stability of model training.

[0039] This embodiment significantly reduces memory usage and computational cost during training. GRPO estimates the advantage function through within-group comparisons, reducing the variance of policy updates and thus ensuring a more stable learning process. GRPO introduces KL divergence constraints to prevent overly drastic policy updates, thereby maintaining the stability of the policy distribution.

[0040] In a preferred embodiment, the reward function includes an answer correctness function, a format consistency function, and a reasoning step correctness function; the answer correctness function measures the consistency between the model's output answer and the standard answer; the format consistency function measures the consistency between the format of the model's output result and the preset format; and the reasoning step correctness function measures the correctness of the thought chain reasoning process.

[0041] In this embodiment, GPRO estimates the reward using population samples, avoiding additional computational overhead. The reward function, designed to balance process correctness and answer correctness in multimodal model inference, consists of three parts: The correctness function is : , The correctness function measures whether the model's output answer is consistent with the standard answer.

[0042] Format consistency function : To enhance the robustness of the model's reasoning ability, a format consistency reward mechanism was designed to determine whether the model should reason before outputting the answer. Specifically, the model is required to output the reasoning process first and then output the answer in... The system will output the result after the reasoning section. If the result conforms to the preset format, a reward will be given.

[0043] , Reasoning Step Correctness Function : To measure the correctness of the reasoning steps, for a single model output... ,in The first part represents the reasoning section. i Step 1. Based on the factual descriptions of the corresponding questions in the training dataset, determine whether the reasoning generated by the model contains factual conflicts. If no conflicts exist, a reward is given. This metric aims to improve the correctness of the model's reasoning process and reduce logical errors and factual inconsistencies generated during model inference.

[0044] .

[0045] The reward function used in the GRPO process is a linear combination of the three functions mentioned above: , in, , is the hyperparameter that is set.

[0046] The inference enhancement system for multimodal large models provided by this invention will be described below. The inference enhancement system for multimodal large models described below can be referred to in correspondence with the inference enhancement method for multimodal large models described above.

[0047] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a multimodal large-scale model inference enhancement system provided by the present invention.

[0048] This invention also provides a reasoning enhancement system for a multimodal large model, comprising: a reasoning data generation module 301, used to generate thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a question; a basic enhancement module 302, used to perform basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data, to obtain a model with basic reasoning enhancement; the basic reasoning enhancement aims to improve the accuracy of correct answer generation in the multimodal large model; and a process enhancement module 303, used to perform process reasoning enhancement training on the model with basic reasoning enhancement using a group relative policy optimization algorithm, to obtain a multimodal large model with final reasoning enhancement; the process reasoning enhancement aims to improve the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers.

[0049] This invention can be widely applied to various data-scarce multimodal task scenarios, such as image understanding and reasoning in specialized fields. It can significantly improve model performance in specific domains without requiring manually labeled additional training data. This invention is expected to become a standard method for deploying general-purpose multimodal large-scale models in specialized fields, with significant economic benefits and market application prospects.

[0050] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions in the memory 403 to execute a reasoning enhancement method for a multimodal large model. This method includes: generating thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a question; performing basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data to obtain a model with basic reasoning enhancement; basic reasoning enhancement aims to improve the accuracy of correct answer generation in the multimodal large model; employing a group relative policy optimization algorithm to perform process reasoning enhancement training on the model with basic reasoning enhancement to obtain a final reasoning-enhanced multimodal large model; process reasoning enhancement aims to improve the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers.

[0051] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the reasoning enhancement method for the multimodal large model provided by the above methods. The method includes: generating thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a question; performing basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data to obtain a model with basic reasoning enhancement; the basic reasoning enhancement is to enhance the accuracy of correct answer generation of the multimodal large model; using a group relative policy optimization algorithm to perform process reasoning enhancement training on the model with basic reasoning enhancement to obtain a final reasoning-enhanced multimodal large model; the process reasoning enhancement is to enhance the quality of the thought chain of the multimodal large model; the group relative policy optimization algorithm optimizes the model by taking into account both the quality of the thought chain and the intra-group relative reward for correct answers.

[0053] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a reasoning enhancement method for a multimodal large model provided by the methods described above. This method includes: generating thought chain reasoning data through self-sampling of the multimodal large model; the thought chain reasoning data includes correct answer samples and incorrect answer samples for a question; performing basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data to obtain a model with basic reasoning enhancement; basic reasoning enhancement aims to improve the accuracy of correct answer generation in the multimodal large model; employing a group relative policy optimization algorithm to perform process reasoning enhancement training on the model with basic reasoning enhancement to obtain a final reasoning-enhanced multimodal large model; process reasoning enhancement aims to improve the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the intra-group relative reward for correct answers.

[0054] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing inference in a multimodal large model, characterized in that, include: Mind chain reasoning data is generated through self-sampling of a multimodal large model; The thought chain reasoning data includes samples of correct and incorrect answers to the questions. Based on the thought chain reasoning data, the multimodal large model is subjected to basic reasoning enhancement training to obtain a model with enhanced basic reasoning. The basic reasoning enhancement aims to improve the accuracy of generating correct answers for the multimodal large model; The basic reasoning-enhanced model is trained using a group relative strategy optimization algorithm to obtain a final reasoning-enhanced multimodal large model. The process reasoning enhancement improves the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the relative reward within the group for correct answers.

2. The inference enhancement method for multimodal large models according to claim 1, characterized in that, The basic reasoning enhancement training of the multimodal large model based on the thought chain reasoning data includes: Based on the correct answer samples, a supervised fine-tuning algorithm is used to gradually correct errors in the multimodal large model reasoning through gradient updates, so as to generate correct answers using thought chain reasoning; Based on the aforementioned incorrect answer samples, the multimodal large model is enabled to identify and learn from incorrect answers during the process of simultaneously learning correct and incorrect reasoning, in order to deduce the correct answer.

3. The inference enhancement method for multimodal large models according to claim 1 or 2, characterized in that, The process reasoning enhancement training is performed on the basic reasoning-enhanced model using a group relative strategy optimization algorithm to obtain the final reasoning-enhanced multimodal large model, including: A set of answers is sampled from the question using a sampling model; the sampling model is an enhanced version of the basic inference from the previous time step. Each sampled answer is evaluated using a reward function to obtain a corresponding reward value; The reward value for each sampled answer is normalized to obtain the relative advantage; Based on the relative advantage, update the parameters of the base inference-enhanced model at the current moment; The sampling step, the evaluation step, the relative advantage step, and the parameter update step are repeatedly and iteratively executed until a preset number of iterations are reached to obtain the final inference-enhanced multimodal large model.

4. The inference enhancement method for multimodal large models according to claim 3, characterized in that, Before updating the parameters of the base inference-enhanced model at the current moment based on the relative advantage, the method further includes: The parameter updates of the enhanced model based on the basic inference are controlled by KL divergence constraints.

5. The inference enhancement method for multimodal large models according to claim 3, characterized in that, The reward function includes an answer correctness function, a format consistency function, and a reasoning step correctness function; the answer correctness function measures the consistency between the model's output answer and the standard answer; the format consistency function measures the consistency between the format of the model's output result and the preset format; and the reasoning step correctness function measures the correctness of the thought chain reasoning process.

6. The inference enhancement method for multimodal large models according to claim 4, characterized in that, The objective function of the group relative strategy optimization algorithm is: ， The formula for calculating the relative advantage is: ， in, Let the objective function of the group relative strategy optimization algorithm be . For model parameters, To balance the hyperparameters of the KL divergence term, To control the hyperparameters of the model update magnitude, For the clipping function, The advantage value is calculated based solely on the relative reward output from samples within each group. For sampling models, The model is an enhancement of the basic reasoning at the current moment. Number of samples per group The average reward for all candidate outputs. Let the standard deviation of the reward for all candidate outputs be . The reward function scores each sample within the group. As a preset reference model, To calculate the KL divergence between the base inference-enhanced model and the preset reference model at the current time, For the question, For the first i One answer, The importance sampling coefficient.

7. A multimodal large-scale model inference enhancement system, characterized in that, include: The inference data generation module is used to generate thought chain inference data through self-sampling of a multimodal large model; The thought chain reasoning data includes samples of correct and incorrect answers to the questions. The basic enhancement module is used to perform basic reasoning enhancement training on the multimodal large model based on the thought chain reasoning data, so as to obtain a model with basic reasoning enhancement. The basic reasoning enhancement aims to improve the accuracy of generating correct answers for the multimodal large model; The process enhancement module is used to perform process reasoning enhancement training on the basic reasoning enhanced model using a group relative strategy optimization algorithm to obtain the final reasoning-enhanced multimodal large model. The process reasoning enhancement improves the quality of the thought chain in the multimodal large model; the group relative policy optimization algorithm optimizes the model by balancing the quality of the thought chain and the relative reward within the group for correct answers.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the inference enhancement method for multimodal large models as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the inference enhancement method for multimodal large models as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the inference enhancement method for multimodal large models as described in any one of claims 1 to 6.

Images