Multimodal general reasoning methods, devices, equipment and media

Abstract

This invention discloses a multimodal general reasoning method, apparatus, device, and medium. The method includes: constructing a pseudo-image reasoning thought chain dataset based on a preset first dataset and a dataset construction strategy; supervising and fine-tuning a preset benchmark visual language model using the pseudo-image reasoning thought chain dataset and a preset large model fine-tuning framework to obtain an initial multimodal reasoning model; training the initial multimodal reasoning model using a preset second dataset, reward rules, and a group relative policy optimization algorithm to obtain a target multimodal reasoning model; and inputting multimodal question information sent by a client into the target multimodal reasoning model to obtain the corresponding reasoning thought chain. This invention enhances the deep reasoning capabilities of the benchmark visual language model, improving its complex logical reasoning and generalization abilities in multimodal reasoning tasks, and satisfying the interpretability and accuracy requirements of complex reasoning tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal general reasoning method, apparatus, computer device, and storage medium. Background Technology

[0002] In recent years, large language models, such as GPT-4 and Claude-3, have demonstrated remarkable capabilities in text reasoning. Current text reasoning models, through mechanisms like chain thinking, have shown near-human expert-level abilities in complex reasoning tasks such as mathematical proofs. However, in complex real-world scenarios, reasoning systems relying solely on text modalities cannot solve complex tasks in real-world environments.

[0003] Unlike text-based reasoning, multimodal reasoning tasks require models to directly extract information from images, necessitating robust visual recognition and understanding capabilities. While existing multimodal reasoning models can accurately describe elements within images and understand the scenes they depict, they still lack the capacity for in-depth and accurate visual reasoning. Recent experimental reports indicate that in visual question-answering tasks requiring multi-step reasoning, current mainstream multimodal reasoning models often lack explicit and interpretable intermediate reasoning steps. The models tend to directly output the final answer, resulting in an unobservable and uncontrollable reasoning process. This "black box" approach, lacking intermediate reasoning steps, severely restricts the interpretability and reliability of answers from multimodal reasoning models.

[0004] Existing text-based reasoning models have achieved near-human-level complex reasoning capabilities by constructing rigorous logical thought chains through step-by-step derivation. For example, the DeepSeek-R1 model has evolved a human-like, step-by-step complex reasoning pattern through thought chain mechanisms and reinforcement learning frameworks. The core of thought chain technology lies in simulating the human cognitive process of progressive problem-solving by explicitly generating intermediate reasoning steps. This technology breaks through the "black box" mode of models directly outputting results, improving the model's performance in complex tasks such as mathematical reasoning and logical analysis through interpretable intermediate derivation paths. However, these text-based reasoning models themselves lack the ability to process raw visual information and cannot directly perceive and understand images. Currently, thought chain technology is mainly focused on the field of text-based modal reasoning, and its application in multimodal large language models remains an area for further research.

[0005] When applying text reasoning models to multimodal tasks, a common approach is to rely on an external visual model to generate textual descriptions of images, and then input these descriptions into the text reasoning model. Textual descriptions generated in this way are often incomplete, inaccurate, or lack key details, especially those involving spatial relationships, abstract concepts, or information requiring reasoning. This information loss directly leads to low-quality visual information obtained by the subsequent text reasoning model, severely impairing its accuracy and reliability, and failing to meet the needs of complex visual reasoning.

[0006] Therefore, existing technologies in multimodal reasoning suffer from insufficient interpretability, accuracy, and reliability due to the lack of interpretable intermediate reasoning steps, the inability of text reasoning models to directly process visual information, and the information loss that occurs when relying on external visual models to convert visual information into text information. Summary of the Invention

[0007] This invention provides a multimodal general reasoning method, apparatus, computer device, and storage medium, aiming to solve the problems in the prior art where the lack of interpretable intermediate reasoning steps, the inability of text reasoning models to directly process visual information, and the information loss when relying on external visual models to convert visual information into text information lead to insufficient interpretability, accuracy, and reliability of complex reasoning tasks.

[0008] In a first aspect, embodiments of the present invention provide a multimodal general inference method applied to an inference server, wherein the inference server establishes a communication connection with a client to achieve data information transmission; the method includes:

[0009] The pseudo-image reasoning thought chain dataset is constructed based on the preset first dataset and dataset construction strategy;

[0010] The pseudo-image reasoning thought chain dataset and the preset large model fine-tuning framework are used to supervise the fine-tuning training of the preset benchmark visual language model to obtain the initial multimodal reasoning model.

[0011] The initial multimodal reasoning model is trained using a pre-defined second dataset, reward rules, and a group relative policy optimization algorithm to obtain the target multimodal reasoning model.

[0012] If a multimodal question is received from the client, the multimodal question is input into the target multimodal reasoning model to obtain the reasoning thought chain corresponding to the multimodal question.

[0013] Secondly, embodiments of the present invention provide a multimodal general-purpose inference device applied to an inference server, wherein the inference server establishes a communication connection with a client to realize the transmission of data information; the device includes:

[0014] The dataset construction unit is used to construct a pseudo-image reasoning thought chain dataset based on a preset first dataset and dataset construction strategy.

[0015] The supervised fine-tuning training unit is used to perform supervised fine-tuning training on the preset benchmark visual language model using the pseudo-image reasoning thought chain dataset and the preset large model fine-tuning framework to obtain the initial multimodal reasoning model.

[0016] The reinforcement learning training unit is used to perform reinforcement learning training on the initial multimodal reasoning model using a preset second dataset, reward rules, and group relative policy optimization algorithm to obtain the target multimodal reasoning model.

[0017] The multimodal reasoning unit is used to input the multimodal question information sent by the client into the target multimodal reasoning model if it receives the multimodal question information, so as to obtain the reasoning thought chain corresponding to the multimodal question information.

[0018] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal general reasoning method of the first aspect described above.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to execute the multimodal general reasoning method of the first aspect described above.

[0020] This invention provides a multimodal general reasoning method, apparatus, device, and medium. The method includes: constructing a pseudo-image reasoning thought chain dataset based on a preset first dataset and a dataset construction strategy; supervising and fine-tuning a preset benchmark visual language model using the pseudo-image reasoning thought chain dataset and a preset large model fine-tuning framework to obtain an initial multimodal reasoning model; training the initial multimodal reasoning model using a preset second dataset, reward rules, and a group relative policy optimization algorithm to obtain a target multimodal reasoning model; and inputting multimodal question information sent by a client into the target multimodal reasoning model to obtain the corresponding reasoning thought chain. This invention enhances the deep reasoning capabilities of the benchmark visual language model, improving its complex logical reasoning and generalization capabilities in multimodal reasoning tasks, and meeting the interpretability, accuracy, and reliability requirements of complex reasoning tasks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a multimodal general reasoning method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a sub-process of a multimodal general reasoning method provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a sub-process of a multimodal general reasoning method provided in another embodiment of the present invention;

[0025] Figure 4 This is a schematic block diagram of a multimodal universal inference device provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multimodal general inference method provided in an embodiment of the present invention. The multimodal general inference method provided in this embodiment is applied to an inference server, which establishes a communication connection with a client to achieve data transmission, such as... Figure 1 As shown, the multimodal general reasoning method provided in this embodiment of the invention includes steps S11 to S14.

[0032] S11. Based on the preset first dataset and dataset construction strategy, construct the pseudo-image reasoning thought chain dataset.

[0033] In this embodiment, a pseudo-image reasoning thought chain dataset containing complex cognitive steps and self-reflection processes is constructed according to a preset first dataset and dataset construction strategy. This pseudo-image reasoning thought chain dataset is used for supervised fine-tuning training of the benchmark visual language model, thereby giving the benchmark visual language model the ability to generate high-quality thought chains in multimodal tasks.

[0034] In one embodiment, such as Figure 2 As shown, step S11 includes:

[0035] S111. Obtain the first dataset; wherein the first dataset includes several first data subsets, and each first data item in each first data subset includes corresponding question information and correct answer information, wherein the question information includes corresponding question text information and question image;

[0036] S112. For each piece of first data in each first data subset, the problem image in the first data is converted into corresponding formal language information using the data modality conversion module in the dataset construction strategy.

[0037] S113. Combine the question text information in the first data with the formal language information to form new question information;

[0038] S114. Input the new question information and the preset thought chain generation prompt words into the pre-trained text reasoning model in the dataset construction strategy to obtain the first thought chain;

[0039] S115. Adjust the first thought chain according to the role-playing strategy in the dataset construction strategy to obtain the second thought chain;

[0040] S116. Based on the correct answer information in the first data and the answer filtering rules in the dataset construction strategy, the second thinking chain is filtered and judged to obtain the filtering judgment result;

[0041] S117. If the filtering judgment result is determined to be a valid answer result, the question information in the first data and the second thinking chain are combined to form pseudo-image reasoning thinking chain data and stored in a preset database.

[0042] S118. The pseudo-image reasoning thought chain data stored in the database are combined to form the pseudo-image reasoning thought chain dataset.

[0043] In this embodiment, the generation process of the pseudo-image reasoning thought chain dataset mainly includes six core stages: data preparation, formal description, thought chain generation, thought chain adjustment, answer filtering, and dataset construction. First, the data preparation stage is entered, i.e., step S111 is executed to obtain the first dataset. The first dataset includes several first data subsets supporting structured reasoning and multimodal reasoning. These first data subsets cover fields such as common sense question answering, document reading, graph understanding, and mathematical reasoning, with each first data subset focusing on one field. Each piece of first data in each first data subset includes corresponding question information and correct answer information. The question information includes corresponding question text information and question image. These question information contain deep coupling between visual elements such as graphs and text and the text, requiring the model to simultaneously understand the logical conditions in the visual and text information and deduce the correct answer. By filtering the first dataset in this way, it is ensured that the question information processed by the text reasoning model meets the basic conditions for image reasoning and step-by-step reasoning, thus ensuring the quality of the generated thought chain. Preferably, the first dataset can be filtered from the LLaVA-OneVision dataset.

[0044] After obtaining the first dataset, the formal description stage begins, i.e., step S112 is executed. In this stage, the data modality conversion module is used to convert the image information of the problem image in each data point of the first dataset into formal language information that the text reasoning model can understand and process in a structured and semantically complete manner. That is, the formal language information is structured text that the text reasoning model can understand and process. Thus, the data modality conversion module realizes the modality conversion of information from image to text, enabling the text reasoning model to perceive and use the key information in the image based on the formal language information to perform reasoning, ensuring that the image content can be fully expressed and understood during the reasoning process.

[0045] Next, the thought chain generation stage begins, specifically steps S113 and S114. The question text information and formalized language information from the first data are combined to form new question information. This new question information, along with preset thought chain generation prompts, are input into a pre-trained text reasoning model. The text reasoning model performs pseudo-image reasoning to obtain the first thought chain. During this process, the thought chains generated by the text reasoning model, containing complex cognitive steps and self-reflection processes, are systematically collected—that is, the first thought chains are collected. This captures the powerful chain reasoning ability of the text reasoning model in simulating visual information processing. Preferably, the text reasoning model is the DeepSeek-R1 model. The thought chain generation prompts guide the text reasoning model in generating thought chains containing complex cognitive steps and self-reflection processes during the pseudo-image reasoning task. For example, the thought chain generation prompts can be set as: "Please answer the question (i.e., the question corresponding to the question text information) and provide your reasoning process," which should include the following requirements: 1. Simulate image reasoning: Treat the formalized language information as the image itself, simulate reasoning by imagining you are looking at the image, and act as if you can see the image, but avoid using visualization as a reasoning step. 2. Direct visual language: Describe content from the perspective of directly observing the image (e.g., "The image shows..."), avoiding inferences based on image descriptions or textual explanations. 3. Avoid using phrases such as "according to the description," "based on the image description," or "visualized image."

[0046] Since the new problem information is text-based, the text reasoning model still utilizes text-based information during pseudo-image reasoning. However, the pseudo-image reasoning process, which simulates visual information, still deviates from the real image reasoning process. The first thought chain output by the text reasoning model is a structured reasoning process directly based on text-based information, but this reasoning process lacks the direct perception and interpretation of images that a multimodal reasoning model should have. In real image reasoning tasks, image information should be directly perceived and understood by the multimodal reasoning model from the image. To address this issue, after obtaining the first thought chain output by the text reasoning model, a thought chain adjustment phase is initiated, i.e., step S115. This phase uses a role-playing strategy that simulates human visual understanding to adjust the style and perspective of the first thought chain, thereby ensuring the authenticity and coherence of the multimodal reasoning process within the thought chain.

[0047] The thought chain is the process by which a text reasoning model deduces a final answer to a specific problem step by step. Therefore, the second thought chain includes two parts: the final answer obtained by the text reasoning model and the reasoning process. Due to the loss in the modality transformation process of the problem image and the performance limitations of the text reasoning model itself, the final answer obtained by the model may be incorrect, or even if the final answer obtained by the model is correct, the reasoning process reflected in the model's thought chain may be vague or imprecise. Therefore, it is necessary to set answer filtering rules to further remove second thought chains with incorrect final answers or vague and imprecise reasoning processes, and to select second thought chains with correct final answers and correct reasoning processes (i.e., clear and rigorous reasoning processes). Specifically, after adjusting the first thought chain output by the text reasoning model into the second thought chain, the answer filtering stage is entered, that is, steps S116 and S117 are executed. The second thought chain is filtered and judged according to the correct answer information in the first data and the answer filtering rules, thereby obtaining the filtering judgment result. The filtering judgment result is used to indicate whether the second thought chain is valid. If the filtering result indicates a valid answer, it means the second thought chain is valid. This can also be understood as the final answer and reasoning process contained in the second thought chain being correct. Therefore, this second thought chain is a high-quality processed thought chain, and it is retained. The question information from the first data and the second thought chain are combined to form high-quality pseudo-image reasoning thought chain data, which is then stored in a pre-defined database. This effectively improves the quality of the pseudo-image reasoning thought chain dataset. Conversely, if the filtering result indicates an invalid answer, it means the second thought chain is not valid. This can also be understood as the final answer contained in the second thought chain being incorrect or the reasoning process being vague or imprecise. In this case, the second thought chain is removed.

[0048] Finally, the dataset construction stage begins, which involves executing step S118. The database stores the constructed high-quality pseudo-image reasoning thought chain data, and these pseudo-image reasoning thought chain data are combined into a pseudo-image reasoning thought chain dataset.

[0049] In one embodiment, the data modality conversion module includes a pre-trained multimodal language model, an object detection model, and a text recognition model; the data modality conversion module in step S112, which utilizes the dataset to construct a strategy, converts the problem image in the first data into corresponding formal language information, including:

[0050] If the image type of the problem image in the first data is determined to be a first preset image type, then the multimodal language model and the object detection model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding first image description text, the first object type list and the bounding box of each object in the first object type list, and to form the formal language information.

[0051] If the image type of the problem image in the first data is determined to be the second preset image type, then the text recognition model and the multimodal language model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding first restored text and use it as the formal language information;

[0052] If the image type of the problem image in the first data is determined to be a third preset image type, then the multimodal language model, the object detection model and the text recognition model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding second image description text, the second object type list, the bounding box of each object in the second object type list and the second restored text, and form the formal language information;

[0053] If the image type of the problem image in the first data is determined to be the fourth preset image type, then the multimodal language model is used to perform modal transformation processing on the problem image in the first data to obtain the corresponding chart description text, which is used as the formal language information.

[0054] In this embodiment, different formal language rules and tools are used to achieve modal bridging from image to text for different image types. The image type of the problem image in the first data belongs to one of the first preset image type, the second preset image type, the third preset image type, and the fourth preset image type. The first preset image type is used to indicate that the problem image contains only natural scene visual elements, the second preset image type is used to indicate that the problem image contains only text visual elements, the third preset image type is used to indicate that the problem image contains both the natural scene visual elements and the text visual elements, and the fourth preset image type is used to indicate that the problem image contains only chart visual elements.

[0055] Specifically, if the image type of the problem image in the first data is determined to be a first preset image type, it means that the problem image in the first data only contains natural scene visual elements, or it can be understood as the problem image in the first data only contains objects in nature. Then, a multimodal language model is used to generate a first image description text and a first object category list corresponding to the problem image. The first image description text is a detailed description of the problem image, and the first object category list contains the categories corresponding to all objects appearing in the problem image. Then, an object detection model is used to detect each object in the problem image according to the first object category list, obtaining the bounding box of each object in the first object category list. Finally, the obtained first image description text, the first object category list, and the bounding boxes of each object in the first object category list are combined to form formalized language information. Preferably, the multimodal language model is the GPT-4o model, and the object detection model is the GroundingDINO model.

[0056] If the image type of the problem image in the first data is determined to be the second preset image type, it means that the problem image in the first data only contains visual text elements. Therefore, a text recognition model is first used to identify the text content appearing in the problem image, obtaining discrete, out-of-order text recognition results. Then, in order to restore the original, complete text content, the problem image and the text recognition results are input into a multimodal language model to reconstruct the text recognition results, obtaining the corresponding first restored text, which is then used as formalized language information. Preferably, the text recognition model is the Easy OCR model.

[0057] If the image type of the problem image in the first data is determined to be the third preset image type, it indicates that the problem image in the first data contains both natural scene visual elements and text visual elements, making it a complex image. Therefore, combining the aforementioned methods for processing natural scene visual elements and text visual elements in the problem image, a multimodal language model is used to generate a second image description text and a second object category list corresponding to the problem image. An object detection model is then used to detect each object in the problem image according to the second object category list to obtain the bounding box of each object in the second object category list. Similarly, the second image description text is a detailed description of the problem image, and the second object category list contains the categories corresponding to all objects appearing in the problem image. Simultaneously, a text recognition model is used to recognize the text content appearing in the problem image to obtain discrete, disordered text recognition results. The multimodal language model is then used to reconstruct the text recognition results to obtain the corresponding second restored text. Finally, the second image description text, the second object category list, the bounding boxes of each object in the second object category list, and the second restored text are combined to form formalized language information.

[0058] If the image type of the problem image in the first dataset is determined to be the fourth preset image type, it means that the problem image only contains chart visual elements. The chart categories corresponding to these visual elements include circuit diagrams, flowcharts, musical scores, Sudoku puzzles, UI layout diagrams, tables, and annotated charts. The problem image and preset modality transition prompts are then input into a multimodal language model, which outputs the corresponding chart description text. The modality transition prompts guide the multimodal language model to determine the chart category to which the problem image belongs and represent it according to the formal language rules corresponding to that category. Specifically, the multimodal language model is guided to represent problem images of circuit schematics according to the SPICE standard, flowcharts according to PlantUML text syntax rules, sheet music according to LilyPond syntax rules, Sudoku puzzles according to Python syntax rules, UI layouts according to HTML syntax rules, tables according to CSV or JSON syntax rules, and annotated charts according to Matplotlib syntax rules. Finally, the chart description text is used as formalized circuit schematic language information.

[0059] In one embodiment, step S115 includes:

[0060] The first thought chain and the preset thought chain adjustment prompts are input into a pre-trained large-scale language model to obtain the second thought chain.

[0061] In this embodiment, the thought chain adjustment prompt is used to guide the large language model to simulate human visual understanding and adjust the style and perspective of the first thought chain. The adjusted first thought chain is then used as the second thought chain to ensure the authenticity and coherence of the multimodal reasoning process in the thought chain.

[0062] For example, the thought chain adjustment prompts can be set to revise the current first thought chain according to the following guidelines: 1. Style conversion: Convert all inferences based on image descriptions to direct image inferences. For example, replace phrases such as "based on description" and "based on image description" with "image displayed" or "visible from image"; 2. Remove image visualization steps: If the thought chain contains inference steps based on image visualization, remove those steps and rewrite the thought chain so that it infers directly from the image itself, rather than from inferences based on image visualization; Strictly apply these modifications to ensure that the final thought chain reflects a direct interpretation of the image, unaffected by descriptions, textual descriptions, or image visualization processes.

[0063] In one embodiment, step S116 includes:

[0064] The correct answer information from the first data, the second thought chain, and the preset answer filtering prompts are input into a pre-trained large-scale language model to obtain the filtering judgment result.

[0065] In this embodiment, the large-scale language model outputs a filtering judgment result based on the correct answer information in the first data, the second thought chain, and preset answer filtering prompts. The filtering judgment result is either a valid answer or an invalid answer. A valid answer indicates that the second thought chain is valid, and an invalid answer indicates that the second thought chain is invalid. The answer filtering prompts guide the large-scale language model to determine whether the second thought chain is valid based on the correct answer information.

[0066] For example, the prompt for adjusting the thought chain could be set as: "This provides you with a second thought chain and the correct answer information. This second thought chain is the reasoning steps for solving the problem." Please follow these rules to determine if the second thought chain is valid: Validity Criteria: 1. The second thought chain must contain correct reasoning steps. 2. The second thought chain must have a final answer derived from the reasoning, and this final answer must be consistent in meaning with the correct answer information. 3. The second thought chain must be based on the reasoning process of an image, not on image descriptions or textual explanations. 4. The second thought chain must not contain steps unrelated to reasoning, and each step of reasoning must be closely related.

[0067] S12. Using the pseudo-image reasoning thought chain dataset and the preset large model fine-tuning framework, the preset benchmark visual language model is subjected to supervised fine-tuning training to obtain the initial multimodal reasoning model.

[0068] In this embodiment, the high-quality pseudo-image reasoning thought chain dataset obtained through processing is used as cold-start initialization training data. Based on a preset large model fine-tuning framework, the selected benchmark visual language model is subjected to the first stage of supervised fine-tuning training using the image reasoning thought chain dataset. The benchmark visual language model after supervised fine-tuning is then used as the initial multimodal reasoning model. The core task of this supervised fine-tuning training is to enable the benchmark visual language model to learn and internalize the complex, step-by-step, and interpretable reasoning patterns exhibited by the text reasoning model, thereby endowing it with the ability to generate high-quality thought chains. By training on thought chain data generated by the text reasoning model, which includes complex cognitive self-reflection processes, the benchmark visual language model learns the complex cognitive processes of the text reasoning model, enhancing its complex reasoning ability in multimodal tasks. Supervised fine-tuning training not only standardizes the model output but also lays a solid foundation for model training in the subsequent reinforcement learning training stage. Preferably, the benchmark visual language model is the Qwen2.5-VL model, and the large model fine-tuning framework is the LLaMA-Factory training framework.

[0069] More specifically, each pseudo-image reasoning thought chain in the pseudo-image reasoning thought chain dataset includes corresponding question information and a second thought chain. The question information includes corresponding question text information and question image. Step S12 includes: selecting a pseudo-image reasoning thought chain from the pseudo-image reasoning thought chain dataset; inputting the question information in the pseudo-image reasoning thought chain into the benchmark visual language model to obtain an intermediate thought chain; calculating the loss value between the intermediate thought chain and the second thought chain in the image reasoning thought chain data; adjusting the parameters of the benchmark visual language model using the loss value to obtain an adjusted benchmark visual language model; updating the benchmark visual language model using the adjusted benchmark visual language model, and returning to the process of selecting a pseudo-image reasoning thought chain from the pseudo-image reasoning thought chain dataset until the loss value meets the preset fine-tuning stopping condition, then stopping the adjustment and outputting the initial multimodal reasoning model.

[0070] S13. The initial multimodal reasoning model is trained by reinforcement learning using a preset second dataset, reward rules, and group relative policy optimization algorithm to obtain the target multimodal reasoning model.

[0071] In this embodiment, the initial multimodal reasoning model is trained using reinforcement learning to refine the model's reasoning ability and further improve its stability and generalization ability in multimodal reasoning tasks.

[0072] In one embodiment, such as Figure 3 As shown, step S13 includes:

[0073] S131. Select one second data point from the second dataset;

[0074] S132. The problem information in the second data is repeatedly input into the initial multimodal reasoning model a preset number of times to obtain a predictive thought chain set; wherein, the predictive thought chain set includes the preset number of predictive thought chains;

[0075] S133. Based on the predicted thinking chain set, the correct answer information in the second data, and the reward rule, obtain the overall reward score corresponding to each predicted thinking chain in the predicted thinking chain set;

[0076] S134. The overall loss is calculated based on the overall reward score corresponding to each prediction thought chain in the prediction thought chain set and the group relative strategy optimization algorithm.

[0077] S135. The parameters in the initial multimodal inference model are adjusted using the overall loss to obtain the adjusted initial multimodal inference model;

[0078] S136. Update the initial multimodal inference model using the adjusted initial multimodal inference model, and return to the step of selecting a second data point from the second dataset until the overall loss meets the preset stopping condition. Then stop the adjustment and output the target multimodal inference model.

[0079] In this embodiment, the data structure of each piece of second data in the second dataset is the same as that of each piece of first data in the first dataset. Each piece of second data in the second dataset includes corresponding question information and correct answer information, and the question information includes corresponding question text information and question image. Preferably, the second dataset is the open-source Clevr dataset. Specifically, when training the initial multimodal inference model using reinforcement learning, firstly, a piece of second data is selected from the second dataset, which can be randomly selected or selected sequentially. The question information in the second data is repeatedly input into the initial multimodal inference model for a preset number of times. The initial multimodal inference model outputs a preset number of predicted thought chains, forming a set of predicted thought chains. Then, based on the set of predicted thought chains and the correct answer information in the second data, the correctness of the final answer in each predicted thought chain and the standardization of the format of each predicted thought chain are evaluated through reward rules, thereby obtaining the overall reward score corresponding to each predicted thought chain. Then, the overall loss is calculated based on the total reward score corresponding to each predictive thought chain and the group relative policy optimization algorithm (i.e., GRPO algorithm). This overall loss is then used to adjust the parameters in the initial multimodal inference model, completing this round of reinforcement learning training for the initial multimodal inference model, thus obtaining the adjusted initial multimodal inference model. Finally, the adjusted initial multimodal inference model is used to update the initial multimodal inference model, and execution returns to step S131 to begin the next round of reinforcement learning training for the initial multimodal inference model. This training process is repeated until the overall loss meets the preset stopping condition, at which point the adjustment stops, and the current initial multimodal inference model is output as the final target multimodal inference model.

[0080] In one embodiment, the reward rules include accuracy reward rules and format reward rules, and step S133 includes:

[0081] For each prediction thinking chain in the prediction thinking chain set, prediction answer information is extracted from the prediction thinking chain, and a corresponding accuracy reward score is obtained based on the correct answer information in the second data, the prediction answer information, and the accuracy reward rule.

[0082] The corresponding format reward score is obtained based on the predictive thought chain, the preset format template, and the format reward rules;

[0083] The accuracy reward score is added to the format reward score to obtain the overall reward score corresponding to the predictive thought chain.

[0084] In this embodiment, the reward rules are set with two aspects: accuracy reward rules and format reward rules. The accuracy reward rules are used to evaluate the correctness of the final answer in each predictive thought chain, and the format reward rules are used to evaluate the standardization of the format of each predictive thought chain.

[0085] Specifically, for each predictive thought chain in the predictive thought chain set, the predictive thought chain includes two parts: the final answer obtained from the initial multimodal reasoning model and the reasoning process. The final answer in the predictive thought chain is extracted as the predicted answer information. Based on the correct answer information in the second data, the correctness of the predicted answer information is evaluated using an accuracy reward rule to obtain the accuracy reward score corresponding to the predictive thought chain. For example, in some embodiments, if the answer to the question information in the second data is definite and unique, that is, the correct answer information in the second data is the unique answer to the corresponding question information, then it is determined whether the predicted answer information is consistent with the correct answer information in the second data; if it is determined that the predicted answer information is consistent with the correct answer information in the second data, the accuracy reward score is 1; if it is determined that the predicted answer information is inconsistent with the correct answer information in the second data, the accuracy reward score is 0. If the correct answer information in the second data is the bounding boxes of each object appearing in the question image, then the intersection-union ratio (IUR) between the bounding boxes of each object in the correct answer information and the corresponding predicted bounding boxes in the predicted answer information is calculated to obtain an IUR set. If it is determined that each IUR in the IUR set exceeds a preset threshold, then the accuracy reward score is 1; if it is determined that there is an IUR in the IUR set that does not exceed the preset threshold, then the accuracy reward score is 0.

[0086] In one embodiment, when performing step S134, firstly, the relative advantage score corresponding to each predictive thought chain in the predictive thought chain set is calculated based on the overall reward score corresponding to each predictive thought chain in the predictive thought chain set and the first calculation strategy. The formula corresponding to the first calculation strategy is as follows:

[0087] ;

[0088] Where N represents the total number of predictive thought chains contained in the predictive thought chain set; the value of i ranges from [1, N]; A i r represents the relative advantage score corresponding to the i-th predictive thought chain in the predictive thought chain set; i The mean({r1,r2,⋯,r) represents the total reward score corresponding to the i-th prediction thought chain in the prediction thought chain set; N}) represents the average total reward score corresponding to all predictive thought chains in the predictive thought chain set; std({r1,r2,⋯,r) N}) represents the standard deviation of the total reward score corresponding to all predictive thought chains in the predictive thought chain set.

[0089] Then, the strategy ratio is calculated according to the second calculation strategy. The formula corresponding to the second calculation strategy is as follows:

[0090] ;

[0091] Where S1 represents the strategy ratio; O i π represents the i-th predictive thought chain in the predictive thought chain set; q represents the question information in the second data; θ (O i |q) indicates that when the question information q is input into the current initial multimodal inference model, the model generates a predictive thought chain O. i The probability of; This indicates that when the question information q is input into the initial multimodal reasoning model obtained after the previous round of reinforcement learning training, the model generates a predictive thought chain O. i The probability of.

[0092] Next, the strategy ratio is pruned according to the set pruning hyperparameters to obtain the pruned ratio, in order to prevent the model parameters from being updated too large.

[0093] Finally, the overall loss is calculated using the third calculation strategy, the relative advantage score of each predictive thought chain in the predictive thought chain set, the strategy ratio, and the pruned ratio. The formula for the third calculation strategy is as follows:

[0094]

[0095] ;

[0096] ;

[0097] Among them, J GRPO (θ) represents the total loss; S1 represents the strategy ratio; A i Si represents the relative advantage score corresponding to the i-th predictive thought chain in the predictive thought chain set; S2 represents the pruned ratio; β represents the preset hyperparameter. Represents the Kullback-Leibler divergence penalty; π ref (O i |q) indicates that when the question information q is input into the reference model, the model generates a predictive thought chain O. i The probability is determined by the reference model, which is the initial multimodal inference model obtained after supervised fine-tuning training of the baseline visual language model.

[0098] S14. If the multimodal question information sent by the client is received, the multimodal question information is input into the target multimodal reasoning model to obtain the reasoning thought chain corresponding to the multimodal question information.

[0099] In this embodiment, a target multimodal reasoning model is deployed in the inference server. Users can send multimodal question information to the inference server through a client, and the target multimodal reasoning model in the inference server completes the reasoning task for the multimodal question information. The multimodal question information includes text and images. Specifically, the inference server inputs the multimodal question information into the target multimodal reasoning model, which directly perceives and understands the images in the multimodal question information and uses key information from the images to perform reasoning, thereby obtaining the corresponding reasoning thought chain. The reasoning thought chain includes intermediate reasoning steps, satisfying the interpretability, accuracy, and reliability requirements of complex reasoning tasks.

[0100] The multimodal general reasoning method provided by this invention can be applied to complex reasoning tasks in different application scenarios. For example, in the educational question-and-answer scenario, a problem-solving thought chain can be generated step by step through the target multimodal reasoning model based on the test question image and the question text. In the robot path navigation scenario, a path planning reasoning can be performed through the target multimodal reasoning model based on the task description text and the map image to obtain a navigation thought chain that includes intermediate reasoning steps and recommended paths, thereby optimizing the navigation task.

[0101] This invention discloses a multimodal general reasoning method that enhances the deep reasoning capabilities of a benchmark visual language model, thereby improving the model's ability to perform complex logical reasoning and generalization in multimodal reasoning tasks and satisfying the interpretability, accuracy, and reliability requirements of complex reasoning tasks.

[0102] This invention also provides a multimodal general-purpose inference device, which is used to execute any embodiment of the aforementioned multimodal general-purpose inference method. The multimodal general-purpose inference device provided in this invention is applied to an inference server, which establishes a communication connection with a client to achieve data transmission. Specifically, please refer to... Figure 4 , Figure 4 This is a schematic block diagram of a multimodal general-purpose inference device provided in an embodiment of the present invention. The multimodal general-purpose inference device provided in this embodiment includes a dataset construction unit 11, a supervised fine-tuning training unit 12, a reinforcement learning training unit 13, and a multimodal inference unit 14. Detailed descriptions of each functional unit are as follows:

[0103] Data set construction unit 11 is used to construct a pseudo-image reasoning thought chain dataset based on a preset first dataset and dataset construction strategy.

[0104] The supervised fine-tuning training unit 12 is used to perform supervised fine-tuning training on the preset benchmark visual language model using the pseudo-image reasoning thought chain dataset and the preset large model fine-tuning framework to obtain the initial multimodal reasoning model.

[0105] The reinforcement learning training unit 13 is used to perform reinforcement learning training on the initial multimodal reasoning model using a preset second dataset, reward rules, and group relative policy optimization algorithm to obtain the target multimodal reasoning model.

[0106] The multimodal reasoning unit 14 is used to input the multimodal question information sent by the client into the target multimodal reasoning model if it receives the multimodal question information, so as to obtain the reasoning thought chain corresponding to the multimodal question information.

[0107] In one embodiment, the dataset construction unit 11 is specifically used for:

[0108] Obtain the first dataset; wherein the first dataset includes several first data subsets, and each first data item in each first data subset includes corresponding question information and correct answer information, wherein the question information includes corresponding question text information and question image;

[0109] For each piece of first data in each first data subset, the problem image in the first data is converted into corresponding formal language information using the data modality conversion module in the dataset construction strategy;

[0110] Combine the question text information in the first data with the formal language information to form new question information;

[0111] The new question information and the preset thought chain generation prompts are input into the pre-trained text reasoning model in the dataset construction strategy to obtain the first thought chain;

[0112] The first thought chain is adjusted based on the role-playing strategy in the dataset construction strategy to obtain the second thought chain;

[0113] The second thought chain is filtered and judged based on the correct answer information in the first data and the answer filtering rules in the dataset construction strategy to obtain the filtering judgment result;

[0114] If the filtering judgment result is determined to be a valid answer, the question information in the first data and the second thinking chain are combined to form pseudo-image reasoning thinking chain data and stored in a preset database.

[0115] The pseudo-image reasoning thought chain data stored in the database are used to form the pseudo-image reasoning thought chain dataset.

[0116] In one embodiment, the data modality conversion module includes a pre-trained multimodal language model, an object detection model, and a text recognition model; when the dataset construction unit 11 executes the step of converting the problem image in the first data into corresponding formal language information using the data modality conversion module in the dataset construction strategy, it is specifically used for:

[0117] If the image type of the problem image in the first data is determined to be a first preset image type, then the multimodal language model and the object detection model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding first image description text, the first object type list and the bounding box of each object in the first object type list, and to form the formal language information.

[0118] If the image type of the problem image in the first data is determined to be the second preset image type, then the text recognition model and the multimodal language model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding first restored text and use it as the formal language information;

[0119] If the image type of the problem image in the first data is determined to be a third preset image type, then the multimodal language model, the object detection model and the text recognition model are used to perform modal conversion processing on the problem image in the first data to obtain the corresponding second image description text, the second object type list, the bounding box of each object in the second object type list and the second restored text, and form the formal language information;

[0120] If the image type of the problem image in the first data is determined to be the fourth preset image type, then the multimodal language model is used to perform modal transformation processing on the problem image in the first data to obtain the corresponding chart description text, which is used as the formal language information.

[0121] In one embodiment, the dataset construction unit 11 executes the adjustment of the first thought chain according to the role-playing strategy in the dataset construction strategy to obtain a second thought chain, specifically for:

[0122] The first thought chain and the preset thought chain adjustment prompts are input into a pre-trained large-scale language model to obtain the second thought chain.

[0123] In one embodiment, the dataset construction unit 11 performs the filtering judgment on the second thought chain based on the correct answer information in the first data and the answer filtering rules in the dataset construction strategy, and obtains the filtering judgment result, specifically for:

[0124] The correct answer information from the first data, the second thought chain, and the preset answer filtering prompts are input into a pre-trained large-scale language model to obtain the filtering judgment result.

[0125] In one embodiment, the reinforcement learning training unit 13 is specifically used for:

[0126] Select one second data point from the second dataset;

[0127] The question information in the second data is repeatedly input into the initial multimodal reasoning model a preset number of times to obtain a predictive thought chain set; wherein, the predictive thought chain set includes the preset number of predictive thought chains;

[0128] Based on the predicted thinking chain set, the correct answer information in the second data, and the reward rules, the overall reward score corresponding to each predicted thinking chain in the predicted thinking chain set is obtained;

[0129] The overall loss is calculated based on the total reward score corresponding to each predictive thought chain in the predictive thought chain set and the group relative strategy optimization algorithm.

[0130] The parameters in the initial multimodal inference model are adjusted using the overall loss to obtain the adjusted initial multimodal inference model;

[0131] The initial multimodal inference model is updated using the adjusted initial multimodal inference model, and the step of selecting a second data point from the second dataset is returned until the overall loss meets the preset stopping condition. Then the adjustment is stopped and the target multimodal inference model is output.

[0132] In one embodiment, the reward rules include an accuracy reward rule and a format reward rule. When the reinforcement learning training unit 13 executes the step of obtaining the overall reward score corresponding to each predictive thought chain in the predictive thought chain set based on the predicted thought chain set, the correct answer information in the second data, and the reward rules, it is specifically used for:

[0133] For each prediction thinking chain in the prediction thinking chain set, prediction answer information is extracted from the prediction thinking chain, and a corresponding accuracy reward score is obtained based on the correct answer information in the second data, the prediction answer information, and the accuracy reward rule.

[0134] The corresponding format reward score is obtained based on the predictive thought chain, the preset format template, and the format reward rules;

[0135] The accuracy reward score is added to the format reward score to obtain the overall reward score corresponding to the predictive thought chain.

[0136] The present invention discloses a multimodal general reasoning device for executing any embodiment of the aforementioned multimodal general reasoning method. By empowering the deep reasoning capability of the benchmark visual language model, it improves the model's complex logical reasoning and generalization capabilities in multimodal reasoning tasks, and satisfies the interpretability, accuracy and reliability of complex reasoning tasks.

[0137] The aforementioned multimodal general reasoning method can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0138] Please see Figure 5 , Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a device bus 501, wherein the memory may include a storage medium 503 and internal memory 504.

[0139] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute a multimodal general inference method.

[0140] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0141] The internal memory 504 provides an environment for the execution of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multimodal general reasoning method.

[0142] This network interface 505 is used for network communication, such as providing data transmission. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0143] The processor 502 is used to run the computer program 5032 stored in the memory to implement the multimodal general reasoning method disclosed in the embodiments of the present invention.

[0144] Those skilled in the art will understand that Figure 5The embodiments of the computer device shown do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only memory and a processor. In such embodiments, the structure and function of the memory and processor are different from those shown. Figure 5 The embodiments shown are consistent and will not be repeated here.

[0145] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0146] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the multimodal general reasoning method disclosed in the embodiments of the present invention.

[0147] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0148] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or may be electrical, mechanical, or other forms of connection.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0150] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a backend server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0152] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal general inference method, characterized in that, The application is applied to an inference server which establishes a communication connection with a client to realize transmission of data information; the method comprises: According to a preset first data set and a data set construction strategy, a pseudo-image inference thought chain data set is constructed; An initial multi-modal inference model is obtained by supervising and fine-tuning a preset benchmark visual language model by using the pseudo-image inference thought chain data set and a preset large model fine-tuning framework; A target multi-modal inference model is obtained by performing reinforcement learning training on the initial multi-modal inference model by using a preset second data set, a reward rule and a group relative strategy optimization algorithm; If multi-modal question information sent by the client is received, the multi-modal question information is input into the target multi-modal inference model to obtain an inference thought chain corresponding to the multi-modal question information; The pseudo-image inference thought chain data set is constructed according to the preset first data set and the data set construction strategy, comprising: The first data set is obtained; wherein the first data set comprises a plurality of first data subsets, and each first data in each first data subset comprises corresponding question information and correct answer information, and the question information comprises corresponding question text information and question images; For each first data in each first data subset, the question image in the first data is converted into corresponding formal language information by using a data modality conversion module in the data set construction strategy; wherein the data modality conversion module comprises a pre-trained multi-modal language model, a target detection model and a character recognition model; The question text information in the first data and the formal language information are combined to form new question information; The new question information and a preset thought chain generation prompt word are input into a pre-trained text inference model in the data set construction strategy to obtain a first thought chain; The first thought chain is adjusted according to a role-playing strategy in the data set construction strategy to obtain a second thought chain; The second thought chain is filtered and judged according to the correct answer information in the first data and a response filtering rule in the data set construction strategy to obtain a filtering judgment result; If it is determined that the filtering judgment result is a valid response result, the question information in the first data and the second thought chain are combined to form pseudo-image inference thought chain data, which is stored in a preset database; The pseudo-image inference thought chain data stored in the database is combined to form the pseudo-image inference thought chain data set.

2. The multi-modal general reasoning method of claim 1, wherein, The question image in the first data is converted into corresponding formal language information by using the data modality conversion module in the data set construction strategy, comprising: If it is determined that the image type of the problem image in the first data is a first preset image type, the problem image in the first data is subjected to modal conversion processing by using the multi-modal language model and the target detection model, to obtain corresponding first picture description text, a first object category list, and a bounding box of each object in the first object category list, and to form the formalized language information; wherein the first preset image type is used to indicate that the problem image only contains natural scene visual elements; If it is determined that the image type of the problem image in the first data is a second preset image type, the problem image in the first data is subjected to modal conversion processing by using the character recognition model and the multi-modal language model, to obtain corresponding first restored text and use it as the formalized language information; wherein the second preset image type is used to indicate that the problem image only contains character visual elements; If it is determined that the image type of the problem image in the first data is a third preset image type, the problem image in the first data is subjected to modal conversion processing by using the multi-modal language model, the target detection model, and the character recognition model, to obtain corresponding second picture description text, a second object category list, a bounding box of each object in the second object category list, and second restored text, and to form the formalized language information; wherein the third preset image type is used to indicate that the problem image contains the natural scene visual elements and the character visual elements; If it is determined that the image type of the problem image in the first data is a fourth preset image type, the problem image in the first data is subjected to modal conversion processing by using the multi-modal language model, to obtain corresponding chart description text and use it as the formalized language information; wherein the fourth preset image type is used to indicate that the problem image only contains chart visual elements.

3. The multi-modal general reasoning method of claim 1, wherein, The role-playing strategy in the data set construction strategy is used to adjust the first thought chain to obtain a second thought chain, including: The first thought chain and a preset thought chain adjustment prompt word are input into a pre-trained large language model to obtain the second thought chain.

4. The multi-modal generalization method of claim 1, wherein, The correct answer information in the first data and the answer filtering rule in the data set construction strategy are used to filter and judge the second thought chain to obtain a filtering judgment result, including: The correct answer information in the first data, the second thought chain, and a preset answer filtering prompt word are input into a pre-trained large language model to obtain the filtering judgment result.

5. The multi-modal generalization method of claim 1, wherein, A preset second data set, a reward rule, and a group relative strategy optimization algorithm are used to perform reinforcement learning training on the initial multi-modal reasoning model to obtain a target multi-modal reasoning model, including: A second data is selected from the second data set; Problem information in the second data is repeatedly input into the initial multi-modal reasoning model for a preset number of times to obtain a set of predicted thought chains; wherein the set of predicted thought chains includes the preset number of predicted thought chains; According to the prediction thought chain set, the correct answer information in the second data, and the reward rule, a total reward score corresponding to each prediction thought chain in the prediction thought chain set is obtained; According to the total reward score corresponding to each prediction thought chain in the prediction thought chain set and the group relative strategy optimization algorithm, a total loss is calculated; The parameters in the initial multi-modal reasoning model are adjusted by using the total loss, and an adjusted initial multi-modal reasoning model is obtained; The initial multi-modal reasoning model is updated by using the adjusted initial multi-modal reasoning model, and the step of selecting a second data from the second data set is performed again until the total loss meets a preset stop condition, and then the adjustment is stopped and the target multi-modal reasoning model is output.

6. The multi-modal generalization method of claim 5, wherein, The reward rule includes an accuracy reward rule and a format reward rule, and the total reward score corresponding to each prediction thought chain in the prediction thought chain set is obtained according to the prediction thought chain set, the correct answer information in the second data, and the reward rule, including: For each prediction thought chain in the prediction thought chain set, prediction answer information is extracted from the prediction thought chain, and an accuracy reward score corresponding to the prediction thought chain is obtained according to the correct answer information in the second data, the prediction answer information, and the accuracy reward rule; A format reward score corresponding to the prediction thought chain is obtained according to the prediction thought chain, a preset format template, and the format reward rule; The accuracy reward score and the format reward score are added to obtain the total reward score corresponding to the prediction thought chain.

7. A multi-modal general-purpose inference device, comprising: The device is applied to a reasoning server, and the reasoning server establishes a communication connection with a client to realize transmission of data information; the device includes: A data set construction unit is configured to construct a pseudo-image reasoning thought chain data set according to a preset first data set and a data set construction strategy; A supervision fine-tuning training unit is configured to perform supervision fine-tuning training on a preset benchmark visual language model by using the pseudo-image reasoning thought chain data set and a preset large model fine-tuning framework, to obtain an initial multi-modal reasoning model; A reinforcement learning training unit is configured to perform reinforcement learning training on the initial multi-modal reasoning model by using a preset second data set, a reward rule, and a group relative strategy optimization algorithm, to obtain a target multi-modal reasoning model; A multi-modal reasoning unit is configured to, if receiving multi-modal question information sent by the client, input the multi-modal question information into the target multi-modal reasoning model, to obtain a reasoning thought chain corresponding to the multi-modal question information; The data set construction unit is specifically configured to: obtain the first data set; wherein the first data set includes a plurality of first data subsets, and each first data in each first data subset includes corresponding question information and correct answer information, and the question information includes corresponding question text information and question images; For each first data in each first data subset, a data modality conversion module in the data set construction strategy is used to convert a question image in the first data into corresponding formal language information; wherein the data modality conversion module includes a pre-trained multi-modal language model, a target detection model and a character recognition model; The question text information in the first data and the formal language information are combined to form new question information; The new question information and a preset thinking chain generation prompt word are input into a pre-trained text reasoning model in the data set construction strategy to obtain a first thinking chain; The first thinking chain is adjusted according to a role-playing strategy in the data set construction strategy to obtain a second thinking chain; The second thinking chain is filtered and judged according to correct answer information in the first data and a reply filtering rule in the data set construction strategy to obtain a filtering judgment result; If it is determined that the filtering judgment result is a valid reply result, the question information in the first data and the second thinking chain are combined to form pseudo-image reasoning thinking chain data, and the pseudo-image reasoning thinking chain data is stored in a preset database; The pseudo-image reasoning thinking chain data stored in the database is combined to form the pseudo-image reasoning thinking chain data set.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the multi-modal general reasoning method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program causes the processor to execute the multi-modal general reasoning method of any one of claims 1-6 when executed by the processor.

Images