Multi-modal context learning method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, is applied to the field of financial science and technology and the field of medical health, and discloses a multi-modal context learning method, device and equipment and a storage medium, and the learning method comprises the steps: obtaining a support set related to a task; based on the image and text problem of the multi-modal example, generating a reasoning text of the multi-modal example; organizing all the enhancement examples according to a preset format to form a context prompt part containing N enhancement examples; receiving query input containing a target image and a target text problem; matching a context prompt part corresponding to the target image and the target text question; and combining the context prompt part and the query input into a complete prompt, and inputting the complete prompt into a visual language reasoning model to obtain an output result which corresponds to the query input and contains a reasoning process. According to the method, the technical problems of low reasoning accuracy, weak generalization ability and insufficient capability of adapting to complex tasks of a multi-modal context learning scheme in the prior art are solved.

Description

Multimodal context learning methods, devices, equipment, and storage media Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and is applied in the fields of financial technology and healthcare. In particular, it relates to a multimodal context learning method, device, equipment and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models have shown great potential in the field of multimodal context learning. They can quickly adapt to new multimodal tasks (such as image diagnosis question answering in the medical field and defect analysis querying in the industrial field) without updating model parameters, simply by providing a small number of multimodal examples (support set) in the prompts. They are widely used in general fields such as education, healthcare, industrial quality inspection, and daily interaction.

[0003] However, existing multimodal context learning techniques still have core flaws that severely limit their reasoning accuracy and generalization ability, as shown below.

[0004] The inconsistency between the support set and the query format leads to a break in the core logic. Existing support set examples in the multimodal context learning field generally only contain static triple structures of image-text question-real answer, without any reasoning process. However, in actual multimodal reasoning tasks, user queries require the model to output a structured result that includes the reasoning process and the final answer. This mismatch between the input example lacking reasoning and the output requiring reasoning prevents the model from learning the derivation logic from the question to the answer. It can only passively imitate the example answer and cannot form transferable reasoning capabilities.

[0005] Furthermore, the model relies on shallow heuristics and lacks true task understanding. This is because examples only provide input-output mappings and do not reveal the intermediate reasoning links. When performing multimodal context learning, the model often relies on shallow heuristics rather than truly understanding the essence of the task. For example, it might copy answer keywords from similar examples or perform majority voting on example answers, rather than deriving conclusions based on the logical connections between image features and textual questions. This learning pattern leads to the model performing reasonably well in simple scenarios, but its accuracy is extremely low in complex reasoning tasks, and it is prone to misclassifying semantically unrelated but similar answers.

[0006] Furthermore, when the query and support set examples come from different data distributions, the performance improvement of existing technologies stagnates or even declines sharply. This is because the model does not learn transferable reasoning methods from the examples; it only memorizes the input-answer mapping under a specific distribution and cannot transfer the reasoning logic to new distribution scenarios, which violates the core purpose of multimodal context learning to adapt to new tasks with a small number of examples.

[0007] In addition, existing technologies lack a standardized way of organizing multimodal examples that support a set of rules. The image-question-answer format of different examples is chaotic. Some examples contain redundant descriptions and some are missing key information, making it difficult for the model to quickly capture the core logical connections between examples, which further exacerbates inference bias.

[0008] In summary, existing multimodal context learning techniques suffer from inconsistent formats, missing inference chains, and disordered example organization, failing to guide models to truly learn task reasoning logic. This results in low inference accuracy, weak generalization ability, and insufficient adaptability to complex tasks, making it difficult to meet the practical needs of various fields for precise, interpretable, and highly generalized multimodal reasoning. Therefore, there is an urgent need for a multimodal context learning solution that can overcome these shortcomings and fundamentally address the dual pain points of inconsistent formats and insufficient reasoning ability. Summary of the Invention

[0009] This invention provides a multimodal context learning method, apparatus, device, and storage medium, which can solve the technical problems of low inference accuracy, weak generalization ability, and insufficient adaptability to complex tasks in existing multimodal context learning schemes.

[0010] In a first aspect, the present invention provides a multimodal context learning method, comprising: acquiring a support set related to a task; wherein the support set contains N multimodal examples, the multimodal examples including an image, a text question, and a true answer corresponding to the image and the text question; generating inference text for the multimodal examples based on the image and the text question, so as to form an enhanced example together with the inference text and the true answer; organizing all the enhanced examples according to a preset format to form a context hint portion containing N enhanced examples; wherein the organization format of each enhanced example is the same; receiving a query input containing a target image and a target text question; matching the context hint portion corresponding to the target image and the target text question; combining the context hint portion with the query input to form a complete hint, inputting it into a visual language inference model, and obtaining an output result containing the inference process corresponding to the query input.

[0011] Secondly, the present invention provides a multimodal contextual learning device, comprising: a support set acquisition module for acquiring a support set related to a task; wherein the support set contains N multimodal examples, the multimodal examples including an image, a text question, and the true answer corresponding to the image and the text question; a text generation module for generating inference text for the multimodal examples based on the image and the text question, so as to form an enhanced example together with the inference text and the true answer; a prompt generation module for organizing all the enhanced examples according to a preset format to form a contextual prompt portion containing N enhanced examples; wherein the organization format of each enhanced example is the same; a query input module for receiving a query input containing a target image and a target text question; a prompt matching module for matching the contextual prompt portion corresponding to the target image and the target text question; and a result output module for combining the contextual prompt portion with the query input to form a complete prompt, inputting it into a visual language inference model, and obtaining an output result containing the inference process corresponding to the query input.

[0012] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal context learning method.

[0013] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described multimodal context learning method.

[0014] This invention generates inference text based on multimodal example images and text questions. This inference text, together with the real answer, forms an enhanced example containing the image, text question, inference text, and real answer. This ensures that the format of the support set examples is fully aligned with the query output requirements, upgrading the examples from a traditional static input-output mapping to a complete logical chain of input-inference-output. The model can directly see how to deduce the answer from the question in contextual hints, rather than simply memorizing the result, thus resolving the problem of broken reasoning logic caused by inconsistent formatting at its source.

[0015] Furthermore, the reasoning text in the enhanced examples of this invention provides step-by-step derivation logic, and the contextual hints organize these examples in a uniform format, providing the model with a reasoning template to emulate. The model learns the core solution to the task by studying the reasoning chains in the examples, rather than simply memorizing answers to specific examples.

[0016] Furthermore, after receiving the query input, the present invention matches the contextual hints corresponding to the target image and the target text question to ensure that the example is highly relevant to the query task logic and domain scenario.

[0017] The matching mechanism filters out examples with transferable reasoning logic. The model learns a general reasoning framework, rather than a domain-specific answer mapping. Even if the query and example data distributions differ, the model can still analyze new scenarios based on general reasoning logic.

[0018] Furthermore, the present invention organizes all enhanced examples into a contextual hint section according to a unified preset format, and then combines it with the query input to form a complete hint. This unified format allows the model to quickly identify the fixed relationship between input, inference, and output, eliminating the need to expend computational resources parsing format differences. This enables the model to efficiently focus on core inference logic, avoids misunderstandings caused by format confusion, and improves inference efficiency and accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a flowchart illustrating a multimodal context learning method according to an embodiment of the present invention.

[0021] Figure 2 is a flowchart of step S120 in Figure 1.

[0022] Figure 3 is another flowchart of step S120 in Figure 1.

[0023] Figure 4 is another flowchart of step S160 in Figure 1.

[0024] Figure 5 is a schematic diagram of a multimodal context learning device according to an embodiment of the present invention.

[0025] Figure 6 is a structural schematic diagram of a computer device according to an embodiment of the present invention.

[0026] Figure 7 is another structural schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Figure 1 shows a flowchart of a multimodal context learning method provided by an embodiment of the present invention. As shown in Figure 1, the multimodal context learning method includes the following steps.

[0029] Step S110: Obtain the support set related to the task.

[0030] The support set contains N multimodal examples, which include images, text questions, and the actual answers to the images and text questions.

[0031] Specifically, the core objective of this step is to collect multimodal example data to provide a foundation for subsequent enhanced example construction and contextual hint generation.

[0032] Specifically, the support set can be sourced from an appropriate dataset chosen based on the target domain, covering typical scenarios in general domains. For example, in the field of medical imaging, it can be extended based on the ChestX-ray14 dataset, so that the support set includes images of lung lesions, diagnostic questions (such as whether there are pneumonia lesions in the image), and real answers annotated by professional doctors; in the field of industrial quality inspection, the support set can be a part defect detection dataset, which can include images of part surfaces, defect query questions (such as whether there are cracks or defects in the part in the image), and real answers to quality inspection standards.

[0033] As a concrete example, in the property insurance scenario, multimodal examples of auto insurance claims can be collected, which may include accident scene images (collision / scratches / water damage), text questions (such as whether the accident falls within the scope of vehicle damage insurance coverage), and actual answers (if so, the compensation ratio is 80%), forming a support set. As another concrete example, in the health insurance scenario, multimodal examples of critical illness insurance can be collected, including lung CT images (lung cancer lesions), text questions (whether the situation meets the conditions for critical illness insurance compensation), and actual answers (if so, it falls within the scope of malignant tumor compensation), forming a support set.

[0034] Step S120: Based on the image and text question of the multimodal example, generate inference text for the multimodal example, so as to form an enhanced example together with the inference text and the real answer.

[0035] As a concrete example, in the property insurance scenario, inference text can be generated based on accident images and questions (e.g., "1. The image shows a dent in the right door of the vehicle, caused by a collision; 2. The vehicle is insured for vehicle damage, and the collision is covered by insurance; 3. There are no deductible circumstances, so the compensation ratio is 80%), forming an enhanced example with the actual answer. As another concrete example, in the health insurance scenario, inference text can be generated based on CT images and questions (e.g., "1. The image shows a space-occupying lesion in the lung with blurred boundaries; 2. The pathology report indicates a malignant tumor; 3. The critical illness insurance terms clearly state that malignant tumors are covered by compensation - the conclusion is consistent"), forming an enhanced example with the actual answer.

[0036] In some embodiments of the present invention, as shown in FIG2, step S120 includes the following steps.

[0037] Step S1211: Input the image and text question of the multimodal example into a preset visual language model.

[0038] Specifically, in this step, the visual language model can initially use a general visual language model (such as LLaVA-1.5), and can be adapted and optimized according to the scenario (such as using a fine-tuned version of a medical-specific visual language model in the medical field); specifically, the input format of the visual language model can be the preprocessed multimodal example images and text questions packaged according to the model requirements. Images can be passed in Base64 encoded form, and text questions can be concatenated in string form.

[0039] Step S1212: Based on the reasoning prompt template corresponding to the reasoning text, guide the visual language model to generate the reasoning text of the multimodal example and the predicted answer corresponding to the reasoning text.

[0040] Specifically, in this step, when designing the reasoning prompt template, a general reasoning prompt template can be designed for different domain scenarios to ensure clear guidance logic. For example, a general template could be: "Based on the provided image information and text question, please reason step-by-step to arrive at the answer." The reasoning process should clearly explain the basis for each step (such as features observed in the image, relevant knowledge principles), and finally, the final conclusion should be clearly given with "Answer:".

[0041] In the field of industrial quality inspection, a dedicated template can be set up: Please observe the surface features and dimensions of the parts in the image, compare them with the quality inspection standards, and determine step by step whether the target defect exists. The reasoning process should explain the basis for the defect judgment (e.g., "The continuous black lines on the edge of the part in the image meet the characteristics of a crack defect"). Finally, the conclusion should be given with "Answer:".

[0042] Finally, the reasoning prompt template, along with the image and text question, is input into the visual language model. The model outputs a structured result containing step-by-step reasoning text and predicted answers.

[0043] Understandably, the above embodiment breaks through the limitation of traditional multimodal context learning examples containing only a static "image-question-answer" structure, injecting a reasoning process into the examples for the first time, thus solving the core pain point of inconsistency between the support set and the query format from the source. Compared to models that can only see the input-output mapping relationship and are forced to rely on shallow heuristics (such as copying similar answers), this embodiment guides the model to generate step-by-step reasoning text through reasoning prompt templates, making the examples a complete logical chain of image-question-reasoning-answer, enabling the model to clearly understand how to deduce the answer from the question, rather than just memorizing the result.

[0044] As a concrete example, in a property insurance scenario, when a car insurance surveyor uploads images of an accident scene (such as a video of a vehicle submerged in water), they can input the question, "Does damage to electronic devices caused by water submersion fall under the scope of water damage insurance coverage?" If the image and question are input into the LLaVA-CoT model, the model will be guided to generate reasoning text ("Please combine the accident image features and car insurance terms to reason step by step whether it falls under the corresponding insurance liability and finally give the answer") through a property insurance-specific reasoning prompt template ("1. The image shows water damage to the vehicle's wheel hubs, indicating water ingress damage to electronic devices; 2. Water damage insurance covers vehicle damage caused by water submersion; 3. There are no deductible situations such as secondary ignition - predicted answer: It falls under the scope of water damage insurance coverage").

[0045] As another concrete example, in a health insurance scenario, the policyholder uploads an image of the hospital bill and a screenshot of their medical record, and enters the question, "I was hospitalized for surgery due to acute appendicitis. After medical insurance reimbursement, can the remaining costs be reimbursed by medical insurance?" If the image and question are input into the VL-Rethinker model, the model will be guided to generate reasoning text ("Please combine the medical record and bill, compare with the medical insurance reimbursement terms, reason step by step whether it meets the reimbursement conditions, and finally give the answer") through a health insurance-specific reasoning prompt template ("1. The medical record shows that acute appendicitis is a sudden illness, which falls within the scope of medical insurance coverage; 2. The bill includes surgery fees and hospitalization fees, which are reimbursable items; 3. It does not exceed the reimbursement limit - predicted answer: Medical insurance reimbursement is possible").

[0046] In some embodiments of the present invention, after step S1212, the method further includes: step S1213, comparing the predicted answer with the actual answer; if the two are consistent, retaining the reasoning text corresponding to the predicted answer to construct an enhanced example through the reasoning text; if the two are inconsistent, deleting the multimodal example corresponding to the predicted answer from the support set.

[0047] Specifically, in this step, the predicted answer is precisely matched with the real answer in the support set examples. This can be done using string similarity calculation or domain-specific dictionary matching (e.g., using a medical terminology dictionary for verification in the medical field). Further, if the predicted answer matches the real answer (e.g., both in the industrial field indicate the presence of cracks or defects), the inference text of that example is retained for building enhanced examples. If they do not match, the multimodal example is completely removed from the support set to avoid misleading the model's learning. Further, the number of remaining valid examples after filtering can be counted to ensure the support set size meets the N-shot learning requirement (ensuring N is greater than or equal to 3, default N=5). If the number of examples in the support set is insufficient, steps S1211 to S1213 can be repeated to supplement examples from the initial support set.

[0048] Understandably, compared to directly using all support set examples, even if the examples contain erroneous logic or misleading reasoning, they will still be learned by the model, leading to a performance degradation; this embodiment filters out examples with credible reasoning processes (the predicted answer is consistent with the real answer, indicating that the reasoning logic is likely correct) through answer consistency verification, and removes examples with loopholes in the reasoning text.

[0049] As a specific example, in the property insurance scenario, when constructing a support set for property insurance fire damage assessment, the model-generated predicted answer (70% compensation ratio) for some examples is inconsistent with the actual damage assessment conclusion (50% compensation ratio). The system automatically compares the two, finds the conflicting answers, and removes the example (fire scene image + compensation ratio + incorrect reasoning text) from the support set to prevent the model from learning incorrect damage assessment logic.

[0050] As another concrete example, in the health insurance scenario, when constructing a support set for cardiovascular and cerebrovascular disease claims under critical illness insurance, a certain example model generates a predicted answer (qualifies for critical illness compensation) that is inconsistent with the doctor's marked answer (does not meet the severity level of critical illness, therefore does not qualify). If the system filters out this conflicting example, it will be directly removed to ensure that the reasoning text in the support set is based on the correct conclusion and to avoid the model misleading the policyholder.

[0051] In some embodiments of the present invention, as shown in FIG3, step S120 includes the following steps.

[0052] Step S1221: Based on the image and text question of the multimodal example, generate the first inference text of the multimodal example; Step S1222: When the multimodal example contains manually annotated second inference text, unify the format of the first inference text to the format of the first inference text; Specifically, in this step, the source of manually annotated text can be a dataset with professional annotation resources, from which the corresponding second inference text (the real inference process annotated manually) is extracted.

[0053] Specifically, if the first and second reasoning texts are logically consistent but differ in expression, both should be retained and supplementary reasoning should be noted; if there is a logical conflict, the second reasoning text should prevail and the first reasoning text should be deleted.

[0054] Step S1223: Combine the real answer with the first and second reasoning texts after standardization to form the enhanced example.

[0055] Specifically, in this step, the standardized reasoning text can be bound to the real answer to form a complete enhanced example, with the format being "image + text question + reasoning text + real answer".

[0056] Understandably, this embodiment balances the quality of reasoning text with the cost of annotation, achieving the dual advantages of machine efficiency and human accuracy. Compared to either relying solely on manually annotated reasoning text (high cost and difficult to scale) or solely on machine generation (prone to logical loopholes), this embodiment achieves large-scale example enhancement through dual reasoning text fusion, utilizing both machine generation for large-scale example enhancement and human-annotated golden reasoning to correct machine logic biases (such as rigorous reasoning steps in professional fields).

[0057] As a concrete example, in the property insurance scenario, when building a support set for engineering insurance, it is necessary to quickly and scalably generate examples while ensuring professionalism. For example, 1. The model automatically generates the first inference text (based on the image of equipment damage + whether it belongs to engineering insurance compensation, generating basic inference); 2. The second inference text format is standardized for annotation by insurance company loss assessment experts; 3. The two are combined to form enhanced examples, which reduces annotation costs and ensures the professionalism of inference.

[0058] As another concrete example, in the health insurance scenario, when constructing a support set for chronic disease reimbursement under medical insurance, it is necessary to balance efficiency and medical professionalism; 1. The model generates the first inference text (based on the image of the medication list + whether chronic disease medications can be reimbursed, generating basic inference); 2. The format of the second inference text labeled by doctors is unified; 3. Combine real answers to form enhanced examples, balancing machine efficiency and medical accuracy.

[0059] Step S130: Organize all the enhancement examples according to a preset format to form a contextual hint section containing N enhancement examples.

[0060] The organization format of each of the enhanced examples is the same.

[0061] As a specific example, in the property insurance scenario, enhanced examples can be arranged in a uniform format (e.g., "image + whether it belongs to vehicle damage insurance compensation + inference text + yes, compensation of 80%"); as another specific example, in the health insurance scenario, contextual prompts can be organized: enhanced examples can be arranged in a uniform format (e.g., "CT image + whether it meets the criteria for critical illness compensation + inference text + yes, malignant tumor compensation"); in some embodiments of the present invention, step S130 includes the following steps.

[0062] Step S131: Arrange the images and text questions contained in each of the enhanced examples as a first information group.

[0063] Specifically, in this step, the "image + text question" in each enhanced example is taken as the first information group, and arranged in the order after the examples are filtered. A separator is added between each first information group to ensure that the model can distinguish between different examples.

[0064] Step S132: Arrange the reasoning text and the real answer of the enhanced example as a second information group, wherein the second information group is arranged after the first information group; specifically, in this step, "reasoning text + real answer" in each enhanced example is arranged as a second information group, immediately following the corresponding first information group, with the arrangement format being consistent with the first information group, and the reasoning text and real answer are distinguished by clear labels.

[0065] Step S133: All the enhanced examples are arranged sequentially according to the structure of the first information group and the second information group to form the context prompt portion.

[0066] Specifically, in this step, all enhanced examples are combined in the order of "Example 1 (first information group + second information group) - Example 2 (first information group + second information group) - ... - Example N (first information group + second information group)" to form a contextual hint section containing N enhanced examples. The overall format is uniform, the structure is clear, and there is no redundant information.

[0067] Understandably, this embodiment achieves structured and highly logical contextual hints, allowing the model to quickly capture the mapping relationship between input, reasoning, and output. Compared to examples with no fixed rules and images, questions, and answers presented in a scattered manner, where the model cannot clearly understand the logical connection between the three, this embodiment forces the model to focus on the reasoning path from input to output through a fixed structure of the first information group (input) - the second information group (reasoning + output). This improves the accuracy of the model in imitating the reasoning process and effectively suppresses shallow learning behavior that merely copies the answer.

[0068] As a concrete example, in the property insurance scenario, organizing contextual prompts for car insurance glass claims facilitates the model's rapid learning of the claims logic; for example: 1. First information group arrangement: image (glass breakage image) + text question (Can glass breakage alone be covered by compulsory traffic accident liability insurance); 2. Second information group arrangement: reasoning text (1. Compulsory traffic accident liability insurance only covers third-party losses; 2. Glass breakage alone is a supplementary liability under vehicle damage insurance; 3. No supplementary insurance → no compensation) + actual answer (No, glass breakage insurance is required); 3. Combined in the format of "Example 1 - First Information Group - Second Information Group" to form contextual prompts.

[0069] As another concrete example, in the health insurance scenario, contextual prompts for uremia claims under critical illness insurance can be organized to adapt to the model's reasoning needs; 1. First information group arrangement: image (dialysis treatment record video) + text question (Does dialysis for uremia qualify for critical illness insurance compensation); 2. Second information group arrangement: reasoning text (1. Uremia is a covered disease under critical illness insurance; 2. Dialysis is a necessary treatment; 3. The waiting period has passed - qualifying for compensation) + actual answer (qualified, basic sum insured will be paid according to the contract); 3. Combined into contextual prompts in a unified format.

[0070] Step S140: Receive query input containing the target image and the target text question.

[0071] Specifically, the core objective of this step is to obtain the user's multimodal query requests, providing target objects for subsequent inference.

[0072] Specifically, this step supports multiple input channels, including API calls (such as integration with industrial quality inspection systems), web-based visual uploads (users upload images and input text questions), and mobile application-based interactive input. Furthermore, input format validation can be performed to ensure that the query input strictly includes "target image + target text question," and if either element is missing, a prompt "Please complete the image and question information" will be returned. Additionally, the same preprocessing operations (image normalization, text standardization) as those in the support set examples can be performed on the target image and target text question to ensure data format consistency.

[0073] As a concrete example, in the property insurance scenario, car owners can upload new accident images and enter the question: "My car was rear-ended and the rear bumper is damaged. Can I claim compensation under the vehicle damage insurance?" As another concrete example, in the health insurance scenario, policyholders can upload new lung CT images and enter the question: "I have been diagnosed with a malignant tumor in my lungs. Can I apply for critical illness insurance compensation?"

[0074] Step S150: Match the contextual hint portion corresponding to the target image and the target text question.

[0075] As a specific example, in the property insurance scenario, contextual prompts for car insurance collision claims can be matched from the prompt library; as another specific example, in the health insurance scenario, contextual prompts for "critical illness insurance malignant tumor claims" can be matched from the prompt library.

[0076] Step S160: Combine the contextual hint with the query input to form a complete hint, input it into the visual language reasoning model, and obtain the output result containing the reasoning process corresponding to the query input.

[0077] It is understood that the solution of this invention generates inference text based on multimodal example image and text questions, which, together with the real answer, form an enhanced example containing the image, text question, inference text, and real answer. This ensures that the format of the support set examples is fully aligned with the query output requirements, upgrading the examples from a traditional static input-output mapping to a complete logical chain of input-inference-output. The model can directly see how to deduce the answer from the question in contextual hints, rather than simply memorizing the result, thus resolving the problem of broken reasoning logic caused by inconsistent formatting at its source.

[0078] Furthermore, the reasoning text in the enhanced examples of the embodiments of the present invention provides step-by-step derivation logic, and the contextual hints organize these examples in a uniform format, providing the model with a reasoning template to emulate. The model learns the core solution to the task by studying the reasoning chains in the examples, rather than memorizing answers to specific examples.

[0079] Furthermore, after receiving the query input, the embodiment of the present invention matches the contextual hints corresponding to the target image and the target text question, ensuring that the example is highly relevant to the query's task logic and domain scenario. The matching mechanism filters out examples with transferable reasoning logic, and the model learns a general reasoning framework rather than a domain-specific answer mapping. Even if the query and example data distributions differ, the model can still analyze new scenarios based on general reasoning logic.

[0080] Furthermore, in this embodiment of the invention, all enhanced examples are organized into a contextual hint section according to a unified preset format, and then combined with the query input to form a complete hint. This unified format allows the model to quickly identify the fixed relationship between input, inference, and output, eliminating the need to expend computational resources to resolve format differences. This enables the model to efficiently focus on the core inference logic, avoids misunderstandings caused by format confusion, and improves inference efficiency and accuracy.

[0081] As a specific example, in the property insurance scenario, the model outputs the reasoning process and the answer (such as "Reasoning: 1. The image shows that the rear bumper is dented, which is caused by a rear-end collision; 2. The vehicle damage insurance covers the collision liability; 3. There is no deductible - Answer: It falls within the scope of vehicle damage insurance, with a compensation ratio of 80%").

[0082] As another specific example, in the health insurance scenario, the model outputs the reasoning process and the answer (such as "Reasoning: 1. The image shows the characteristics of a malignant lung tumor; 2. It meets the definition of a malignant tumor under critical illness insurance; 3. The waiting period has passed - Answer: It meets the conditions for critical illness insurance compensation and a claim can be submitted").

[0083] In some embodiments of the present invention, the visual language reasoning model is either LLaVA-CoT or VL-Rethinker.

[0084] Specifically, in this embodiment, in general scenarios (such as educational science question answering and daily image understanding), the LLaVA-CoT model can be selected. This model supports step-by-step reasoning and is suitable for basic reasoning needs in multiple fields. In complex reasoning scenarios (such as medical image diagnosis and precision industrial quality inspection), the VL-Rethinker model can be selected. It has multi-step reflective reasoning capabilities and can improve the reasoning accuracy of complex problems. Specifically, the model's output results require that the model must output structured results that include the reasoning process and the final answer. The reasoning process must be consistent with the example format (step-by-step explanation, clear logic), and the answer must be clear and unambiguous.

[0085] Understandably, this embodiment allows the design of the reasoning-enhanced MM-ICL to fully realize its value, solving the problem that ordinary visual language models have weak reasoning capabilities and cannot support step-by-step deduction. Ordinary VLMs (such as the basic version LLaVA) are better at image description and simple question answering, but lack the ability to perform complex logical reasoning. Even when receiving structured prompts, they are difficult to generate coherent step-by-step reasoning. In contrast, LLaVA-CoT (integrating thought chain reasoning) and VL-Rethinker (supporting multi-step reflective reasoning) are designed specifically for complex visual language reasoning. They can accurately analyze the correlation logic between image features, text questions, and example reasoning, resulting in improved reasoning accuracy compared to ordinary VLMs. Moreover, the logical coherence and professionalism of the reasoning process are significantly better than traditional models.

[0086] In some embodiments of the present invention, as shown in FIG4, step S160 includes the following steps.

[0087] Step S161: Before the context prompt section, add system prompt text, which defines the role or task description of the visual language reasoning model. Specifically, in this step, the model role and reasoning requirements can be defined based on the target task type to ensure that the model clearly defines the task boundaries. As a specific example, in the field of industrial quality inspection, the system prompt could be: "You are an industrial part defect detection reasoning assistant. You need to refer to the defect judgment logic in the following examples, analyze the features of the parts in the query image, compare them with the quality inspection standards, and reason step by step whether the target defect exists, outputting a detailed reasoning process and a clear conclusion." Specifically, the system prompt text is added before the context prompt section as a pre-modeling guide for the model.

[0088] Step S162: After the context prompt section, add the query input to form a complete prompt.

[0089] The query input is arranged according to the first information group. Specifically, in this step, the preprocessed query input (target image + target text question) can be arranged according to the format of the first information group, immediately following the context prompt section, and clearly labeled "query input". All content is then concatenated in the order of system prompt text - context prompt section - query input to form a complete prompt string, ensuring clear separation of each part, no formatting errors, and adaptation to the input requirements of the visual language model.

[0090] Step S163: Input the complete prompt into the visual language reasoning model to obtain the output result containing the reasoning process corresponding to the query input.

[0091] Understandably, this embodiment constructs a closed loop with consistent format throughout the entire process. Compared to scenarios where the model's role and output format are not clearly defined, even if the example includes a reasoning process, the model may only output the answer and ignore the reasoning. This embodiment defines task requirements through system prompts and ensures that the format of the query input and the first information group of the example are completely consistent, so that the model always follows a consistent format specification throughout the entire process of understanding the task, learning the example, and executing the query.

[0092] As a specific example, in a property insurance scenario, when a car owner has an accident in another location and wants to inquire about the claims process, the model needs to output the reasoning result according to the property insurance rules. 1. Add system prompts (You are a car insurance claims assistant in another location. You need to combine the example reasoning logic, deduce step by step according to the car insurance terms, and output the claims process and conclusion); 2. Concatenate the query input (--- Query input--- Image (video of the accident scene in another location) + text question (How to apply for claims after a minor collision in another location); 3. Form a complete prompt by following "System prompts-Context prompts-Query input" and input it into the model.

[0093] As another concrete example, in the health insurance scenario, after an insured person is hospitalized in another location, they need to check the reimbursement conditions. The model needs to deduce according to the health insurance rules. For example: 1. Add a system prompt (You are a medical insurance reimbursement assistant in another location. You need to combine the example reasoning logic, deduce step by step according to the reimbursement terms, and output the reimbursement conditions and process); 2. Concatenate the query input (---Query input---Image (medical record image of hospitalization in another location) + text question (How to reimburse the cost of hospitalization in another location)); 3. Combine them into a complete prompt and input it into the model.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0095] In one embodiment, a multimodal context learning device is provided, which corresponds one-to-one with the multimodal context learning method in the above embodiments. As shown in FIG5, the processing device includes a support set acquisition module 510, a text generation module 520, a prompt generation module 530, a query input module 540, a prompt matching module 550, and a result output module 560. The functional modules are described in detail below: Support set acquisition module 510, used to acquire a support set related to the task; wherein the support set contains N multimodal examples, the multimodal examples including images, text questions, and the corresponding real answers to the images and text questions; Text generation module 520, used to generate inference text for the multimodal examples based on the images and text questions, so as to form an enhanced example together with the inference text and the real answers; Hint generation module 530, used to organize all the enhanced examples according to a preset format to form a contextual hint portion containing N enhanced examples; wherein the organization format of each enhanced example is the same; Query input module 540, used to receive a query input containing a target image and a target text question; Hint matching module 550, used to match the contextual hint portion corresponding to the target image and the target text question; Result output module 560, used to combine the contextual hint portion with the query input into a complete hint, input it to the visual language inference model, and obtain an output result containing the inference process corresponding to the query input.

[0096] In one embodiment, the text generation module 520 is specifically used to: input the image and text question of the multimodal example into a preset visual language model; and guide the visual language model to generate the inference text of the multimodal example and the predicted answer corresponding to the inference text based on the inference prompt template corresponding to the inference text.

[0097] In one embodiment, the text generation module 520 is further configured to: compare the predicted answer with the actual answer; if the two are consistent, retain the reasoning text corresponding to the predicted answer to construct an enhanced example through the reasoning text; if the two are inconsistent, delete the multimodal example corresponding to the predicted answer from the support set.

[0098] In one embodiment, the text generation module 520 is specifically configured to: generate a first inference text for the multimodal example based on the image and the text question of the multimodal example; when the multimodal example contains manually annotated second inference text, unify the format of the first inference text to the format of the first inference text; and combine the real answer, the formatted first inference text, and the second inference text to form the enhanced example.

[0099] In one embodiment, the prompt generation module 530 is specifically configured to: arrange the image and text question contained in each of the enhanced examples as a first information group; arrange the reasoning text and the true answer of the enhanced examples as a second information group, wherein the second information group is arranged after the first information group; and arrange all the enhanced examples in sequence according to the structure of the first information group and the second information group to form the context prompt portion.

[0100] In one embodiment, the result output module 560 is specifically configured to: add system prompt text before the context prompt section, wherein the system prompt text is used to define the role or task description of the visual language reasoning model; add the query input after the context prompt section to form a complete prompt; wherein the query input is arranged according to the first information group; and input the complete prompt into the visual language reasoning model to obtain an output result containing the reasoning process corresponding to the query input.

[0101] In one embodiment, the visual language reasoning model is either LLaVA-CoT or VL-Rethinker.

[0102] This invention's solution acquires the dialogue text stream in real time and automatically extracts key information related to preset claims information categories using a sequence labeling model, forming a structured set of extracted information. This eliminates the need for customers to actively sort through information or for manual filtering, automatically extracting key information from fragmented conversations, ensuring that no useful information is missed, laying the foundation for subsequent communication, and avoiding ineffective follow-up questions caused by fragmented information.

[0103] It is understood that the solution of this invention generates inference text based on multimodal example image and text questions, which, together with the real answer, form an enhanced example containing the image, text question, inference text, and real answer. This ensures that the format of the support set examples is fully aligned with the query output requirements, upgrading the examples from a traditional static input-output mapping to a complete logical chain of input-inference-output. The model can directly see how to deduce the answer from the question in contextual hints, rather than simply memorizing the result, thus resolving the problem of broken reasoning logic caused by inconsistent formatting at its source.

[0104] Furthermore, the reasoning text in the enhanced examples of the embodiments of the present invention provides step-by-step derivation logic, and the contextual hints organize these examples in a uniform format, providing the model with a reasoning template to emulate. The model learns the core solution to the task by studying the reasoning chains in the examples, rather than memorizing answers to specific examples.

[0105] Furthermore, after receiving the query input, the embodiment of the present invention matches the contextual hints corresponding to the target image and the target text question, ensuring that the example is highly relevant to the query's task logic and domain scenario. The matching mechanism filters out examples with transferable reasoning logic, and the model learns a general reasoning framework rather than a domain-specific answer mapping. Even if the query and example data distributions differ, the model can still analyze new scenarios based on general reasoning logic.

[0106] Furthermore, in this embodiment of the invention, all enhanced examples are organized into a contextual hint section according to a unified preset format, and then combined with the query input to form a complete hint. This unified format allows the model to quickly identify the fixed relationship between input, inference, and output, eliminating the need to expend computational resources to resolve format differences. This enables the model to efficiently focus on the core inference logic, avoids misunderstandings caused by format confusion, and improves inference efficiency and accuracy.

[0107] Based on the above-described multimodal context learning method, as shown in Figure 6, this embodiment of the invention also provides a schematic diagram of the structure of a device for the steps of the multimodal context learning method. The device includes a processor 61 and a memory 62 coupled to the processor 61. The memory 62 stores a computer program, which, when executed by the processor 61, causes the processor 61 to perform the steps of the multimodal context learning method described in the above embodiment.

[0108] For further details regarding the implementation of the above technical solution by the processor 61 in the device for the above multimodal context learning method steps, please refer to the description of the multimodal context learning method provided in the above embodiments of the invention, which will not be repeated here.

[0109] The processor 61 can also be called a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or the processor 61 can be any conventional processor.

[0110] As shown in Figure 7, this embodiment of the invention also provides a schematic diagram of a computer-readable storage medium, on which a readable computer program 71 is stored. The computer program 71 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in various embodiments of the invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks or optical disks, ROM (Read-Only Memory), RAM (Random Access Memory), or terminal devices such as computers, servers, mobile phones, and tablets.

[0111] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0112] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0114] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0115] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., SSD (solid state disk)).

[0116] The technical solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0118] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0121] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multimodal context learning method, characterized in that, include: Obtain a support set related to the task; wherein the support set contains N multimodal examples, the multimodal examples including images, text questions, and the corresponding real answers to the images and text questions; based on the images and text questions of the multimodal examples, generate inference text for the multimodal examples, so as to form an enhanced example together with the inference text and the real answers; organize all the enhanced examples according to a preset format to form a contextual hint part containing N enhanced examples; wherein the organization format of each enhanced example is the same; receive a query input containing a target image and a target text question; match the contextual hint part corresponding to the target image and the target text question; combine the contextual hint part with the query input to form a complete hint, input it into the visual language inference model, and obtain an output result containing the inference process corresponding to the query input.

2. The multimodal context learning method according to claim 1, characterized in that, The step of generating inference text for the multimodal example based on the image and text question of the multimodal example, so as to form an enhanced example together with the inference text and the real answer, includes: inputting the image and text question of the multimodal example into a preset visual language model; and guiding the visual language model to generate inference text for the multimodal example and the predicted answer corresponding to the inference text according to the inference prompt template corresponding to the inference text.

3. The multimodal context learning method according to claim 2, characterized in that, After guiding the visual language model to generate the inference text of the multimodal example and the predicted answer corresponding to the inference text based on the inference prompt template corresponding to the inference text, the method further includes: comparing the predicted answer with the real answer; if the two are consistent, retaining the inference text corresponding to the predicted answer to construct an enhanced example through the inference text; if the two are inconsistent, deleting the multimodal example corresponding to the predicted answer from the support set.

4. The multimodal context learning method according to claim 1, characterized in that, The process of generating inference text for the multimodal example based on the image and text question, and then combining the inference text with the real answer to form an enhanced example, includes: generating first inference text for the multimodal example based on the image and text question; when the multimodal example includes manually annotated second inference text, unifying the format of the first inference text to the format of the first inference text; and combining the real answer with the formatted first inference text and the second inference text to form the enhanced example.

5. The multimodal context learning method according to claim 1, characterized in that, The step of organizing all the enhanced examples according to a preset format to form a contextual prompt section containing N enhanced examples includes: arranging the image and text question contained in each enhanced example as a first information group; arranging the reasoning text and the true answer of the enhanced example as a second information group, wherein the second information group is arranged after the first information group; and arranging all the enhanced examples sequentially according to the structure of the first information group and the second information group to form the contextual prompt section.

6. The multimodal context learning method according to claim 5, characterized in that, The step of combining the contextual hints with the query input to form a complete hint, and inputting it into the visual language reasoning model to obtain an output result containing the reasoning process corresponding to the query input, includes: adding system hint text before the contextual hints, wherein the system hint text is used to define the role or task description of the visual language reasoning model; adding the query input after the contextual hints to form a complete hint; wherein the query input is arranged according to the first information group; and inputting the complete hint into the visual language reasoning model to obtain an output result containing the reasoning process corresponding to the query input.

7. The multimodal context learning method according to claim 1, characterized in that, The visual language reasoning model is either LLaVA-CoT or VL-Rethinker.

8. A multimodal context learning device, characterized in that, include: A support set acquisition module is used to acquire a support set related to the task; wherein the support set contains N multimodal examples, the multimodal examples including an image, a text question, and the corresponding real answers to the image and the text question; a text generation module is used to generate inference text for the multimodal examples based on the image and the text question, so as to form an enhanced example together with the inference text and the real answer; a prompt generation module is used to organize all the enhanced examples according to a preset format to form a context prompt part containing N enhanced examples; wherein the organization format of each enhanced example is the same; a query input module is used to receive a query input containing a target image and a target text question; a prompt matching module is used to match the context prompt part corresponding to the target image and the target text question; a result output module is used to combine the context prompt part with the query input into a complete prompt, input it into the visual language inference model, and obtain an output result containing the inference process corresponding to the query input.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal context learning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal context learning method as described in any one of claims 1 to 7.

Images