A visual language reasoning method, device and storage medium based on model decoupling

Abstract

The present invention relates to a visual language reasoning method, device, and storage medium based on model decoupling. The method comprises: obtaining a question description and a visual image to be reasoned, and loading them into a pre-built and trained visual language reasoning framework to obtain reasoning results; the visual language reasoning framework comprises a perception module and a reasoning module, the perception module being used to extract and express the visual image in text form to obtain textual information; the reasoning module being used to perform textual reasoning based on the question description and the textual information output by the perception module to generate an answer result. Compared with the existing technology, the present invention can explicitly decouple the perception and reasoning capabilities of the model. Evaluations based on this framework are closer to the model's actual capabilities, facilitating optimization and iteration, and enabling the model to achieve better reasoning results.

Description

Technical Field

[0001] The present invention relates to the field of visual language reasoning technology, and in particular to a visual language reasoning method, device and storage medium based on model decoupling. Background Art

[0002] Thanks to significant advances in large language models, visual language models (VLMs) have also demonstrated impressive results in multimodal applications, robustly solving tasks such as image-to-text conversation and optical character recognition. In the open-source space, most of these models employ the same strategy: for image-to-text input, the image is encoded into visual features by the visual encoder, which are then aligned and concatenated with the text features encoded by the text encoder before being fed into the LLM for processing to produce a text response. The cognitive process of visual information can be considered perception, while the subsequent process of solving complex problems is reasoning, typically conducted in the text feature space. Evaluating the perception and reasoning capabilities of the VLM plays a crucial role in further model iteration.

[0003] When analyzing the perception and reasoning capabilities of VLMs, the most comparable approaches mostly come from VLM benchmarks, including RealWorldQA and OCRVQA. These efforts collect VQA (visual-language question-answering) data, manually identify the basic capabilities required for each question, and construct benchmark datasets for evaluating model perception or reasoning performance. For low-resource visual-language tasks, existing approaches, such as MiniCPM, have achieved promising results on benchmarks by selecting small-scale language models for end-to-end training. However, the VQA performance of these benchmarks is mostly limited to specific domains. Other equally important scenarios, such as medical imaging and graphical user interfaces, lack similar evaluation benchmarks. Furthermore, existing reasoning-related benchmarks often require the VLM to identify key content before reasoning, while some problems in perception-related benchmarks also require some basic reasoning. The evaluation results on these benchmarks deviate from the actual capabilities of the models.

[0004] Training VLMs for solving visual-language tasks requires a lot of resources and data. For low-resource scenarios, existing models use small-scale language models, which often make them perform poorly on reasoning tasks.

[0005] In addition, because existing VLMs are either closed-source black boxes or end-to-end training models, their perception and reasoning capabilities are intertwined in the latent space, making it difficult for researchers to effectively and accurately estimate the model's individual capabilities, limiting the model's optimization iterations. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art, that is, the existing VLM is either a closed-source black box or an end-to-end training model, whose perception and reasoning capabilities are intertwined in the latent space, making it difficult for researchers to effectively and accurately estimate the single capability of the model, and to provide a visual language reasoning method, device and storage medium based on model decoupling.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A visual language reasoning method based on model decoupling includes the following steps:

[0009] Obtain the question description and visual image to be inferred, and load them into a pre-built and trained visual language reasoning framework to obtain the reasoning result; the visual language reasoning framework includes a perception module and a reasoning module, the perception module is used to extract and express the visual image in text form to obtain text information; the reasoning module is used to perform text reasoning based on the question description and the text information output by the perception module to generate an answer result.

[0010] Furthermore, the perception module extracts and expresses the visual image in text form under the guidance of instructions, and the instructions include general instructions and / or question-related instructions.

[0011] Furthermore, the problem-related instructions are formed by adding guidance of problem-related parts on the basis of the general instructions, and the problem-related parts are generated according to the description of the problem to be inferred.

[0012] Furthermore, the problem-related part is generated by the reasoning module through context learning according to the description of the problem to be reasoned.

[0013] Furthermore, the reasoning module integrates the question description and the text information output by the perception module into the input of the reasoning model through the reasoning template, thereby performing text reasoning and generating an answer result.

[0014] Furthermore, during the training process of the visual language reasoning framework, the reasoning ability of the reasoning module and the impact of different reasoning abilities on the performance of the visual language reasoning framework are evaluated by fixing the structure of the perception module; and the perception ability of the perception module is evaluated by fixing the structure of the reasoning module.

[0015] Furthermore, the perception module and the reasoning module in the visual language reasoning framework both use a combination of one or more network models to perform visual perception or text reasoning.

[0016] Furthermore, the network model is an API model, a locally deployed open source model, or a self-trained self-built model.

[0017] The present invention also provides a visual language reasoning system based on model decoupling, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the above method.

[0018] The present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program is used by a processor to execute the above method.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] (1) The visual language reasoning framework proposed in this paper decouples the perception and reasoning of the visual language model, using a perception module to extract visual information and an external reasoning module to complete the reasoning and answering. Since the perception module is only responsible for the perception task, the evaluation based on this framework is closer to the actual capability of the model, and the framework can estimate the perception capability of the model on visual question-answering data in any scenario. At the same time, by combining lightweight perception modules and reasoning modules, better reasoning capability and general performance can be obtained in low-resource scenarios.

[0021] (2) In terms of evaluating the capabilities of visual language models, existing benchmarks for visual question answering technologies are mostly limited to scenarios in specific fields. Other equally important scenarios lack such evaluation benchmarks. This framework provides a solution for evaluating the perception capabilities of visual language models on any VQA dataset. By fixing the reasoning module, the perception capabilities of existing perception modules can be evaluated. At the same time, by comparing the end-to-end performance of the visual language model with that of the framework, the impact of reasoning capabilities on the visual language model can be explored.

[0022] On the other hand, the evaluation of existing benchmarks deviates from the actual capabilities of the model; in contrast, the evaluation using the solution of the present invention is closer to the actual perception and reasoning of the model.

[0023] (3) Based on this framework, it is possible to combine lightweight perception modules with reasoning modules to handle visual language tasks. This solution trained a visual perception module with 2 billion parameters and integrated it into the framework, achieving performance comparable to that of a perception module ten times larger. In addition, it is possible to try splicing the description text input reasoning module of different perception modules, and achieved better results than the original perception module. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the architecture of a visual language reasoning framework provided in an embodiment of the present invention;

[0025] Figure 2This is an optional instruction set provided in an embodiment of the present invention for guiding the VLM to extract and express visual information during the perception phase. It is divided into: general instructions and question-related instructions. General instructions are fixed and independent of the question. Question-related instructions are general instructions with guidance related to the question. The question-related part is generated by the reasoning module through context learning based on the question generation.

[0026] Figure 3 An example diagram of context learning for generating question-related parts provided in an embodiment of the present invention;

[0027] Figure 4 This is an example diagram of a reasoning template provided in an embodiment of the present invention. The reasoning template combines the extracted visual information of the VLM with the original question as input to the LLM for reasoning and answering. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0031] Example 1

[0032] like Figure 1 As shown, this embodiment provides a visual language reasoning method based on model decoupling, including the following steps:

[0033] Obtain the question description and visual image to be inferred, and load them into a pre-built and trained visual language reasoning framework to obtain the reasoning result; the visual language reasoning framework includes a perception module and a reasoning module. The perception module is used to extract and express the visual image in text form to obtain text information; the reasoning module is used to perform text reasoning based on the question description and the text information output by the perception module to generate the answer result.

[0034] During the training process of the above-mentioned visual language reasoning framework, the reasoning ability of the reasoning module and the impact of different reasoning abilities on the performance of the visual language reasoning framework are evaluated by fixing the structure of the perception module; the perception ability of the perception module is evaluated by fixing the structure of the reasoning module.

[0035] In other words, this solution proposes a visual language reasoning framework, Prism, which decouples perception and reasoning in the VLM (visual language model). This framework is divided into two stages: perception and reasoning. In the perception stage, the VLM, as a perception module, follows the corresponding instructions to extract and express the visual information in the image, while the LLM (reasoning module) will complete text reasoning and solve the task as a reasoning module. By fixing the reasoning module in the framework, the performance obtained by the VLM as a perception module can be used as an approximation of its perception ability. At the same time, by fixing the VLM in the perception module, the impact of different reasoning capabilities on the model can also be explored. In addition, our framework can combine lightweight VLM with external LLM to achieve quite good performance in benchmark tests. This is expected to become a more efficient visual language solution in low-resource scenarios.

[0036] The specific data processing process of the above perception module and reasoning module is as follows:

[0037] The perception module is used to extract and express visual images in text form to obtain text information;

[0038] Preferably, the perception module extracts and expresses the visual image in text form under the guidance of instructions, and the instructions include general instructions and / or question-related instructions.

[0039] The problem-related instructions are formed by adding the guidance of the problem-related part on the basis of the general instructions. The problem-related part is generated according to the description of the problem to be inferred.

[0040] In essence, the VLM is solely responsible for extracting and expressing visual information. Specifically, given an image-question pair, instructions are first used to guide the VLM in extracting and expressing visual information from the image. These instructions can be fixed, general, or user-defined, depending on the question.

[0041] Preferably, the problem-related part is generated by the reasoning module through context learning based on the description of the problem to be reasoned.

[0042] In this embodiment, after ablation, the fixed general instruction with the best guiding effect is selected, and the problem-related part is added to the fixed instruction to form the problem-related instruction. Figure 2 ,The general instructions are fixed and problem-independent, which in this embodiment include: describing the fine-grained content of the image, including scenes, objects, relations, instance locations, and any text.

[0043] The question-related instructions are general instructions with the guidance of the question-related part. The question-related part of the instruction is generated by LLM through context learning based on the question. In this embodiment, the question-related instructions include: describing the fine-grained content of the image, including scenes, objects, relationships, instance locations, and any text. In particular, note the <question-related part>. For a specific example of context learning of the question-related part, see Figure 3 .

[0044] The reasoning module is used to perform text reasoning based on the question description and the text information output by the perception module to generate an answer.

[0045] Preferably, the reasoning module integrates the question description and the text information output by the perception module into the input of the reasoning model through the reasoning template, thereby performing text reasoning and generating an answer result.

[0046] In other words, LLM is responsible for reasoning and answering questions based on visual information and questions. Specifically, the textual visual information generated by VLM and the questions in VQA will be integrated into the input of LLM through a reasoning template. The reasoning template here can also be specified by the user. For the reasoning template used in this embodiment, please refer to Figure 4 , mainly including general text, description and questions. The output of LLM will be used as the response for VQA (Visual Question Answering).

[0047] In the above solution, VLM, LLM, and instructions can all be customized, providing great flexibility. The VLM and LLM here can be commercial API models, locally deployed open source models, or self-trained models.

[0048] It should be noted that:

[0049] The above scheme can be modified as follows:

[0050] 1. Use VLM and LLM other than those used in the present invention to replace framework components to obtain better evaluation or VQA results;

[0051] 2. Change the perception phase instructions and reasoning templates to achieve better results or other purposes;

[0052] 3. Achieve stronger VQA performance by combining multiple VLMs or LLMs in the framework;

[0053] Other uses:

[0054] 1. Use this framework to handle other multimodal tasks besides VQA;

[0055] 2. Use the training strategy in this solution to train an efficient visual captioner.

[0056] Based on the above framework, this example proposes a solution for VQA that combines lightweight VLM and LLM. This example uses high-quality image description pairs generated by GPT4-v and text question-answer pairs generated by GPT4 from the ALLaVA dataset for training. Regarding the model, the SigLip-SO400M model was selected as the visual backbone, and the InternLM2-[1.8B / 7B] model as the language backbone. The trained model was named PrismCaptioner-[1.8B / 7B]. These models were combined with ChatGPT and Llama3, respectively, to verify the feasibility of this solution.

[0057] Specifically, experiments were conducted to verify the feasibility of this framework in two aspects: 1) evaluating model capabilities; 2) solving visual-language tasks.

[0058] Regarding model capability evaluation, this embodiment comprehensively evaluates the perception capabilities of existing VLMs through a reasoning module (ChatGPT) in a fixed framework. The results show that on benchmark tests that rely heavily on visual input, GPT-4o has perception performance far superior to all other models. At the same time, some models in the open source field have achieved performance comparable to GPT-4v, but overall there is still a gap with closed-source models. In addition, it was found that the scale of the language model has a relatively weak impact on the perception capabilities of the model, which inspired reducing the scale of the perception module to achieve higher efficiency. On the other hand, by comparing the end-to-end performance of the VLM with the perception performance under the framework, it was found that small-scale models achieved better results when connected to ChatGPT, especially in tasks related to real reasoning. This shows that the performance of existing small-scale VLMs is limited by their language models.

[0059] Regarding visual language tasks, this framework provides the feasibility of combining lightweight LLM and VLM. This embodiment trains a VLM with 2 billion parameters for visual information extraction and combines it with LLM (ChatGPT) under this framework. The results of the benchmark test show that PrismCaptioner-7B exhibits very competitive performance under this framework, and the performance of PrismCaptioner-2B under this framework is comparable to the end-to-end VLM performance of more than 10 times its scale, while consuming fewer training resources. In addition, in the context of using the LLM API, this PrismCaptioner-2B solution significantly reduces the memory overhead of inference.

[0060] This embodiment also provides a visual language reasoning system based on model decoupling, including a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the above visual language reasoning method based on model decoupling.

[0061] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement the above-mentioned visual language reasoning method based on model decoupling.

[0062] The computer program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such computer program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0063] In the context of the present invention, a machine-readable storage medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable storage medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0064] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A visual language reasoning method based on model decoupling, characterized by: The following steps are involved: Obtain a question description and a visual image to be inferred, and load them into a pre-built and trained visual language inference framework to obtain an inference result; the visual language inference framework includes a perception module and a reasoning module. The perception module is used to extract and express the visual image in text form to obtain text information; the reasoning module is used to perform text reasoning based on the question description and the text information output by the perception module to generate an answer result; The perception module extracts and expresses the visual image in text form under the guidance of instructions, wherein the instructions include general instructions and / or question-related instructions; The reasoning module integrates the question description and the text information output by the perception module into the input of the reasoning model through the reasoning template, thereby performing text reasoning and generating an answer result; The perception module and the reasoning module in the visual language reasoning framework both use a combination of one or more network models to perform visual perception or text reasoning.

2. A visual language reasoning method based on model decoupling according to claim 1, characterized in that: The problem-related instructions are formed by adding guidance of problem-related parts on the basis of the general instructions, and the problem-related parts are generated according to the description of the problem to be inferred.

3. A visual language reasoning method based on model decoupling according to claim 2, characterized in that: The problem-related part is generated by the reasoning module through context learning according to the description of the problem to be reasoned.

4. The visual language reasoning method based on model decoupling according to claim 1 is characterized in that: During the training process of the visual language reasoning framework, the reasoning ability of the reasoning module and the impact of different reasoning abilities on the performance of the visual language reasoning framework are evaluated by fixing the structure of the perception module; the perception ability of the perception module is evaluated by fixing the structure of the reasoning module.

5. The visual language reasoning method based on model decoupling according to claim 1 is characterized in that: The network model is an API model, a locally deployed open source model, or a self-trained self-built model.

6. A visual language reasoning system based on model decoupling, characterized by: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of any one of the methods according to claims 1 to 5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is used by a processor to execute the method according to any one of claims 1 to 5.

Images