Semi-offline strategy reinforcement learning method oriented to visual language slow-thinking reasoning

Abstract

The invention relates to a visual language slow thinking reasoning-oriented semi-offline strategy reinforcement learning method, aims to solve the problem of insufficient reasoning ability of a current large-scale visual language model (LVLM) in a complex multi-modal task, and relates to an extensible semi-offline strategy reinforcement learning (SOPHIA) framework. Comprising the steps of constructing a semi-off strategy behavior model combining online strategy visual understanding and off-line strategy reasoning, designing a feedback and distribution mechanism of visual and reasoning rewards and designing an off-line strategy optimization method based on the visual and reasoning rewards. Compared with the prior art, the visual slow-thinking reasoning ability of the LVLM is systematically improved, meanwhile, the defects of an existing method in the aspects of visual understanding consistency and reasoning generalization ability are overcome, and the visual slow-thinking reasoning ability of the LVLM is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a semi-offline policy reinforcement learning method for visual language slow thinking reasoning. BACKGROUND

[0002] In recent years, large language models (LLM) such as OpenAI o series and deep search R1 model have made breakthroughs in the field of text reasoning due to their strong "slow thinking" ability (i.e. modeling of complex reasoning processes). This progress has prompted academia to attempt to transfer the "slow thinking" ability to visual-linguistic models (LVLM) to solve complex problems such as mathematical geometry and multi-disciplinary multi-modal problems.

[0003] The shortcomings of the prior art mainly manifest in three aspects. First, the previous distillation strategy mainly guides the model to remember specific reasoning patterns, making it difficult to learn real "slow thinking" reasoning ability. Second, online policy reinforcement learning methods are limited by the initial policy distribution of existing LVLM, making it difficult to break through the original behavior pattern. Finally, directly using external model trajectories, LVLM lacks the ability to perceive visual features involved in them, leading to serious degradation phenomena such as optimization direction conflict and visual "hallucination". SUMMARY

[0004] The present application is to overcome the shortcomings of the prior art and provide a semi-offline policy reinforcement learning method for visual language slow thinking reasoning to solve or partially solve the problems of visual language model hallucination and difficulty in learning new reasoning methods during reasoning.

[0005] The object of the present application can be achieved by the following technical solutions:

[0006] In one aspect of the present application, a semi-offline policy reinforcement learning method for visual language slow thinking reasoning is provided, comprising the following steps:

[0007] Obtain a multi-modal training data set comprising a plurality of samples, wherein each sample comprises image information, text problem information and standard answer information;

[0008] Use the image information of the sample as the input of the trainable visual language model, and use the online policy to perceive and represent the input picture, and extract the corresponding visual understanding information;

[0009] Concatenate the visual understanding information with the text problem information of the sample to form a composite input comprising image structure description and task requirements to be answered, and use the reasoning large language model to generate a multi-step logical reasoning trajectory in an offline policy manner;

[0010] Based on the logical reasoning track, a result-based reward signal is calculated, and by sampling multiple logical reasoning tracks, the track reward of obtaining the correct answer is averagely distributed in the visual description layer as the reward label of the visual understanding information.

[0011] Based on the visual understanding information with the reward label, the logical reasoning track and its reward signal, the visual language model is optimized by policy gradient, realizing semi-offline policy reinforcement learning.

[0012] As a preferred technical solution, the visual understanding information includes visual descriptions of the overall structure, local elements, object attributes and their mutual relationships of the image.

[0013] As a preferred technical solution, the logical reasoning track includes restatement and understanding of the question, logical reasoning and formula derivation of local conclusions, correction of errors and presentation of the final answer.

[0014] As a preferred technical solution, the process of calculating the result-based reward signal based on the logical reasoning track includes the following steps:

[0015] The position of the answer in the logical reasoning track is extracted using regular expressions, and the difference between the logical reasoning track and the standard answer on the result is evaluated using the equivalence of mathematical expressions. If the reasoning track and the standard answer are consistent, the reward is assigned as 1, otherwise as 0.

[0016] As a preferred technical solution, the track reward of obtaining the correct answer is averagely distributed in the visual description layer as the reward label of the visual understanding information, including the following steps:

[0017] Based on the visual understanding information, multiple logical reasoning tracks are sampled;

[0018] The reward signal corresponding to each logical reasoning track is calculated;

[0019] The average value of the reward signal corresponding to each logical reasoning track is calculated as the reward of the visual understanding information.

[0020] As a preferred technical solution, the process of optimizing the visual language model by policy gradient includes the following steps:

[0021] Based on the multiple logical reasoning tracks corresponding to the visual understanding information and the reward signal corresponding to the logical reasoning track, a preset visual reward threshold is used to filter samples with high-quality visual understanding and reasoning accuracy, and an offline policy dataset is constructed;

[0022] Based on the offline policy dataset, the visual language model is optimized by policy gradient using a policy gradient optimization algorithm with a reward signal.

[0023] As a preferred technical solution, the multi-modal training data set comprises samples in the disciplines of higher mathematics, physics, engineering and financial management.

[0024] As a preferred technical solution, the visual language model is frozen during the strategy gradient optimization process.

[0025] Another aspect of the present application provides an electronic device comprising one or more processors and a memory, the memory having stored therein one or more programs, the one or more programs comprising instructions for executing the aforementioned semi-offline strategy reinforcement learning method for visual language slow thinking reasoning.

[0026] Another aspect of the present application provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for executing the aforementioned semi-offline strategy reinforcement learning method for visual language slow thinking reasoning.

[0027] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0028] (1) Reducing visual hallucination problems: Based on logical reasoning trajectories, the present application calculates result-based reward signals, and by sampling multiple logical reasoning trajectories, the trajectory reward for obtaining the correct answer is evenly distributed in the visual description layer as a reward label for visual understanding information. The visual language model is optimized by strategy gradient, and the reasoning reward is evenly returned to the visual description for reward distribution through the establishment of the above-mentioned visual description reward return mechanism, which strengthens the influence of visual understanding in the reasoning process and significantly reduces the visual hallucination problem.

[0029] (2) Strong learning ability: The present application takes image information of samples as input of a trainable visual language model, perceives and represents input pictures in an online strategy, extracts corresponding visual understanding information, splices the visual understanding information with text problem information of samples to form a composite input including image structure description and task requirements to be answered, and uses a reasoning large language model to generate a multi-step logical reasoning trajectory in an offline strategy. Through the combination of online strategy visual understanding of LVLM and offline strategy reasoning trajectory generation of external LLM, the visual language model can learn new reasoning methods beyond the current ability.

[0030] (3) Strong generalization ability and high data utilization rate: The present application has achieved leading effect on multiple public benchmark tasks through experimental verification, and has surpassed closed-source large models such as GPT-4.1 on difficult competition mathematics and competition physics data sets, with strong generalization ability and high data utilization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 Flowchart for the semi-offline policy reinforcement learning method for visual language slow thinking reasoning in the embodiment;

[0032] Figure 2 Schematic diagram of semi-offline policy reinforcement learning for visual language slow thinking reasoning in the embodiment;

[0033] Figure 3 Prompt words for visual understanding sampling in the embodiment;

[0034] Figure 4 Prompt words for reasoning sampling in the embodiment;

[0035] Figure 5 Schematic diagram of an electronic device in the embodiment. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0037] Embodiment 1

[0038] To solve the problems of the prior art, the embodiment provides a semi-offline policy reinforcement learning method for visual language slow thinking reasoning, which aims to solve the problem of insufficient reasoning ability of current large-scale visual language models (LVLM) in complex multi-modal tasks. The method mainly involves a scalable semi-offline policy reinforcement learning (SOPHIA) framework, which aims to systematically improve the visual slow thinking reasoning ability of LVLM and overcome the shortcomings of existing methods in visual understanding consistency and reasoning generalization ability.

[0039] Referring to Figure 1 and Figure 2 , the core technical process includes four steps of visual understanding adoption, reasoning sampling, reward propagation and policy optimization. The whole process is driven by an online policy visual understanding model and an offline policy reasoning large language model, which are responsible for visual perception and slow thinking reasoning process respectively, and efficiently optimize the reasoning ability of the large language model LVLM through reward transmission and distribution mechanism. The method mainly includes the following steps:

[0040] Step S1, dataset construction: obtaining a multi-modal training dataset including a plurality of samples.

[0041] A multi-modal training dataset is prepared containing images, text questions, and their standard answers, covering multiple disciplines such as higher mathematics, physics, engineering, financial management, etc., to reflect the integration of complex visual scenes and high-order reasoning tasks.

[0042] To ensure the authenticity and diversity of the data, SOPHIA supports both private high-quality datasets and public datasets such as MathV360K. In practice, data sampling not only ensures that each question is closely related to a specific picture, but also prevents overlap with benchmark data. Keyword filtering and large model assisted screening are used to prevent evaluation bias caused by data leakage.

[0043] Step S2, visual understanding sampling: taking the image information of the sample as the input of the trainable visual language model, the input picture is perceived and represented in an online strategy, and the corresponding visual understanding information is extracted.

[0044] The input picture is perceived and represented in detail by the trainable LVLM in an online strategy. Through careful design of the prompt engineering, the model is prompted to describe the spatial layout, semantic relationship between objects, and fine-grained low-level visual details in the picture as comprehensively as possible (prompt words such as Figure 3 The visual description generated by the model usually contains information such as the overall structure of the image, local elements, object attributes, and their mutual relationships.

[0045] For example, for a picture containing multiple geometric shapes, the model will not only identify the location and size of each shape, but also describe their relative arrangement, overlap, and background color and texture characteristics.

[0046] This visual understanding process uses a completely online strategy for sampling, with the LVLM itself outputting at the current training round, ensuring that the visual description is bound to the actual perception ability of the model and reducing the interference of "hallucinations" or irrelevant content on the reasoning result.

[0047] Step S3, reasoning sampling: the visual understanding information is spliced with the text question information of the sample to form a composite input including image structure description and task requirements to be answered, and a large language model is used to generate a multi-step logical reasoning track in an offline strategy.

[0048] In the reasoning sampling stage, SOPHIA first splices the picture visual description obtained in step S2 with the original question and formats it into a composite input suitable for the language model to understand through prompt engineering. This input not only includes a detailed description of the image structure, but also embeds the specific task requirements to be answered (such as Figure 4 Subsequently, this combined input is fed into a reasoning large language model that has been fine-tuned with instructions to generate a slow thinking, multi-step logical reasoning track in an offline strategy.

[0049] After obtaining the prompt word, the reasoning model can simulate the process of "looking at the picture" and perform human-like problem solving, including phenomenon analysis, fact induction, intermediate calculation, logical deduction, etc. The sampled reasoning track usually includes several paragraphs, formula derivation, local conclusion and final answer. This offline strategy reasoning method can break through the ceiling of the LVLM's self-sampling ability. It should be emphasized that since the visual description is generated by the current LVLM, the reasoning track is always consistent with the actual visual perception level of the model, reducing the conflict of optimization direction caused by external track perception mismatch.

[0050] It should be noted that the reasoning track sampler (i.e. the reasoning model) can be replaced by other accessible LLMs, including other open source models or closed source models. At the same time, the reward allocation method can be extended to segmented / hierarchical rewards or a weighted mechanism based on multi-modal content. The framework of the present application can be extended to cross-modal retrieval, visual recognition and other fields.

[0051] Step S4, reward propagation: based on the logical reasoning track, the result-based reward signal is calculated, and the track reward of the correct answer is allocated in the visual description layer as the reward label of visual understanding information by sampling multiple logical reasoning tracks.

[0052] SOPHIA strengthens the causal relationship between visual understanding and reasoning track through reward propagation. Specifically, first, the final output of each reasoning track is automatically evaluated. Considering that most reasoning tracks are long and complex, SOPHIA uses "result-based rewards", i.e. only the final answer is given a reward signal of 0 or 1.

[0053] Specifically, the position of the answer in the reasoning track is extracted using regular expressions, and the difference between the reasoning track and the standard answer on the result is evaluated using the equivalence of mathematical expressions. If it is consistent with the standard answer, the reward is allocated as 1, otherwise as 0.

[0054] After that, a reward propagation mechanism based on reasoning results is also designed for each visual description: for the same set of pictures and questions, multiple reasoning tracks are sampled, and the track reward of all correct answers is allocated in the visual description layer as the reward label of visual understanding.

[0055] Specifically, based on each visual understanding description, several reasoning tracks are generated, and these reasoning tracks are allocated rewards according to the previous method. After obtaining these rewards, the reward of this visual understanding is considered to be the average value of the rewards of the tracks generated by this visual understanding.

[0056] In the screening of subsequent trajectories, by setting a suitable visual reward threshold (such as a correct rate greater than α), SOPHIA can further screen samples with both high-quality visual understanding and reasoning accuracy, and prioritize them for offline policy optimization. This reward mechanism not only guarantees the correctness of the reasoning result, but also estimates the rationality of the visual information.

[0057] Step S5, policy optimization: based on the visual understanding information with reward annotation, logical reasoning trajectory and its reward signal, the visual language model is optimized by policy gradient, realizing semi-offline policy reinforcement learning

[0058] First, SOPHIA organizes the visual description with reward annotation, reasoning trajectory and its corresponding reward signal obtained in step S4 into a standard offline policy dataset. Then, the model uses a policy gradient optimization algorithm with reward signal to efficiently update the visual-language model parameters. In the actual optimization process, SOPHIA also combines techniques such as freezing the visual encoder weights and prioritizing the shortest trajectory to improve model stability and training efficiency. Experimental results show that SOPHIA has achieved significant performance improvement on multiple public and private multi-modal reasoning benchmarks, especially in challenging scenarios such as mathematics and geometry, far exceeding similar open-source and closed-source large models.

[0059] Specifically, the policy gradient optimization algorithm with reward signal is implemented using the method in “Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA.”

[0060] For the semi-offline policy behavior policy model after training, it can be used to process input images and text questions, and output answers to text questions.

[0061] To illustrate the effectiveness of the method, experiments were conducted on multiple mainstream public multi-modal reasoning benchmarks (such as MMMU Pro, MathVision, OlympiadBench, etc.) and mathematical reasoning datasets, covering 8 billion and 38 billion scale LVLM. See Table 1 for experimental results:

[0062] Table 1 Experimental results

[0063]

[0064] The experimental results show that SOPHIA can significantly improve the reasoning performance under mainstream tasks, especially the model of InternVL3.0-38 type after SOPHIA training, which has achieved the best performance of open source models on the whole. SOPHIA has achieved leading effect on multiple public benchmark tasks, with an average improvement of 8.5%, and has surpassed closed-source large models such as GPT-4.1 on difficult datasets such as MathVision and OlympiadBench, with strong generalization ability and high data utilization efficiency.

[0065] In summary, the method mainly makes the following improvements:

[0066] (1) Combination mechanism of semi-offline strategy behavior model: combining the online strategy visual understanding of LVLM and the offline strategy reasoning trajectory generation method of external LLM, a semi-offline strategy behavior model combining online strategy visual understanding and offline strategy reasoning is constructed. Through offline strategy trajectory sampling, LVLM can learn new reasoning methods beyond current capabilities.

[0067] (2) Visual-reasoning reward feedback mechanism: a reward distribution method that averages the reasoning reward back to the visual description, which strengthens the influence of visual understanding in the reasoning process and significantly reduces the problem of visual "hallucination".

[0068] (3) Offline strategy optimization method based on visual and reasoning rewards: improve the visual slow thinking reasoning ability of LVLM and ensure efficient and low-biased strategy update.

[0069] (4) No need for manual annotation or closed-source model support, easy to apply on a large scale, and strong method scalability.

[0070] This embodiment uses a semi-offline strategy method to integrate strategy visual understanding and offline strategy reasoning trajectory, overcome the "hallucination" problem caused by perception mismatch, enhance the visual and reasoning linkage, improve the multi-modal reasoning generalization ability, and ultimately realize the scalable, data-efficient LVLM slow thinking reasoning training paradigm.

[0071] Embodiment 2

[0072] Based on embodiment 1, the embodiment provides an electronic device, comprising: one or more processors and a memory, the memory storing one or more programs, the one or more programs including instructions for executing the semi-offline strategy reinforcement learning method for visual language slow thinking reasoning as described in embodiment 1.

[0073] As Figure 5The electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the above-mentioned Figure 1 The method. Of course, in addition to the software implementation, the present application does not exclude other implementations, such as logic devices or software and hardware combined manner, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but also hardware or logic device.

[0074] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of a computer readable medium.

[0075] Embodiment 3

[0076] The embodiment provides a computer readable storage medium including one or more programs for one or more processors of an electronic device to execute, the one or more programs including instructions for performing the semi-offline policy reinforcement learning method for visual language slow thinking reasoning as described in embodiment 1.

[0077] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, the computer readable medium does not include transitory computer readable media, such as modulated data signals and carriers.

[0078] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A semi-offline policy reinforcement learning method for slow thinking and reasoning of visual language, characterized by: The steps include: Obtain a multimodal training dataset comprising a plurality of samples, wherein each sample comprises image information, text question information, and standard answer information; The sample image information is used as the input of the trainable visual language model, and the input image is perceived and represented in an online strategy to extract the corresponding visual understanding information; The visual understanding information is combined with the textual question information of the sample to form a composite input including the image structure description and the task requirements to be answered. The large language model is used to generate a multi-step logical reasoning trajectory in an offline strategy manner. Based on the logical reasoning trajectory, a reward signal based on the result is calculated. By sampling multiple logical reasoning trajectories, the rewards of the trajectories that obtain the correct answer are evenly distributed in the visual description layer as reward annotations for visual understanding information; Based on the visual understanding information with reward annotations, logical reasoning trajectories and their reward signals, the visual language model is optimized by policy gradient to achieve semi-offline policy reinforcement learning.

2. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The visual understanding information includes a visual description of the overall structure, local elements, object attributes and their mutual relationships of the image.

3. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The logical reasoning trajectory includes restatement and understanding of the problem, logical reasoning and formula derivation of local conclusions, correction of errors and presentation of the final answer.

4. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The process of calculating the reward signal based on the result based on the logical reasoning trajectory includes the following steps: Regular expressions are used to extract the location of the answer from the logical reasoning trajectory, and the equivalence of mathematical expressions is used to evaluate the difference between the results of the logical reasoning trajectory and the standard answer. If the reasoning trajectory is consistent with the standard answer, the reward is assigned to 1, otherwise it is assigned to 0.

5. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The above-mentioned reward for obtaining the correct answer is evenly distributed in the visual description layer as a reward annotation of visual understanding information, including the following steps: Sampling multiple logical reasoning trajectories based on visual understanding information; Calculate the reward signal corresponding to each logical reasoning trajectory; The average value of the reward signals corresponding to each logical reasoning trajectory is calculated as the reward for visual understanding information.

6. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The process of performing policy gradient optimization on the visual language model includes the following steps: Based on multiple logical reasoning trajectories corresponding to visual understanding information and the reward signals corresponding to the logical reasoning trajectories, a preset visual reward threshold is used to screen samples with high-quality visual understanding and reasoning accuracy to construct an offline strategy dataset. Based on the offline policy dataset, the visual language model is optimized using a policy gradient optimization algorithm with a reward signal.

7. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: The multimodal training dataset includes samples from the fields of advanced mathematics, physics, engineering and financial management.

8. The semi-offline strategy reinforcement learning method for slow thinking and reasoning of visual language according to claim 1 is characterized in that: During the policy gradient optimization process of the visual language model, the visual encoder weights are frozen.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the semi-offline policy reinforcement learning method for visual language slow thinking reasoning as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that It includes one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the semi-offline policy reinforcement learning method for visual language slow thinking reasoning as described in any one of claims 1-8.

Images