Image inference method, device, apparatus and storage medium

Abstract

The application provides an image reasoning method, device and equipment and a storage medium. The method comprises the following steps: inputting a user-inputted static image to be reasoned and a natural language prompt into a trained task routing module, identifying at least one task category corresponding to the natural language prompt, and activating a trained expert agent corresponding to each task category; inputting the static image to be reasoned and the natural language prompt into each trained expert agent for first reasoning, obtaining a shared semantic representation of each task category, and inputting the shared semantic representation into a collaborative memory pool for graph fusion to generate a consensus representation of the static image to be reasoned; and inputting the consensus representation of the static image to be reasoned and the natural language prompt into a trained target agent for second reasoning to obtain a final answer corresponding to each task category of the static image to be reasoned. The application avoids interference between tasks and improves the generality, reasoning efficiency, reliability and stability.

Description

Technical Field

[0001] This application relates to the field of visual understanding technology, and in particular to an image reasoning method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid evolution of deep learning models, image understanding tasks have gradually evolved from early independent tasks such as single target recognition, classification, and detection to a "unified model and unified reasoning" approach, forming a visual reasoning system that integrates multiple tasks such as target detection, instance segmentation, key object counting, visual question answering (VQA), image-text matching, and ideographic expression understanding.

[0003] In this evolution, the introduction of the Transformer architecture has greatly promoted the fusion of visual and linguistic modalities, enabling models to achieve cross-modal semantic alignment. Image-text pre-trained models, such as Contrastive Language-Image Pre-training (CLIP), Bootstrapping Language-Image Pre-training for unified vision-language understanding and generation (BLIP), Florenc's foundational computer vision model, and Generative Image-to-text Transformer (GIT), have demonstrated excellent performance in tasks such as zero-shot image recognition and image-text retrieval through large-scale image-text contrastive learning. However, when faced with complex reasoning tasks, these models often lack structured reasoning capabilities, struggling to achieve deep understanding and logical inference of multiple objects, attributes, and relationships within images.

[0004] Meanwhile, visual reasoning tasks exhibit characteristics of "high diversity, high complexity, and high reliability" in practical applications. For example, in intelligent security scenarios, systems need to simultaneously possess capabilities such as target detection (identifying people / vehicles), attribute recognition (determining age and clothing), behavior understanding (determining whether a fight has occurred / someone has fallen), and visual question answering (answering questions like "Is this person holding a mobile phone?"). In medical image analysis, tasks such as lesion segmentation, quantity statistics, location description, and diagnostic suggestion generation need to be performed in conjunction. In educational image question answering, the system needs to understand charts, text, and symbols in images and answer questions involving logical reasoning. These scenarios place higher demands on the model's multi-task processing capabilities, the controllability of reasoning paths, and the interpretability of results.

[0005] Traditional methods often employ a "single model, multiple task heads" architecture, achieving multi-task learning by sharing a visual encoder and task-specific heads. This architecture assumes all tasks share a single visual backbone network, such as the Vision Transformer (ViT), Swing Transformer (Transformer), and the new era of convolutional neural networks (Convolutional NeXt, ConvNeXt). Dedicated heads are attached at the top for tasks like detection, segmentation, counting, and VQA. During training, the visual backbone is pre-trained visually, and then all parameters are fine-tuned using multi-task data. Inference returns all head results at once. Its advantages include simple structure and easy deployment, but it suffers from severe gradient conflicts between tasks, requiring retraining of the entire model for new tasks, resulting in poor scalability. Therefore, while it improves the model's versatility to some extent, it also introduces problems such as "severe interference between tasks," "rigid structure with difficulty in expansion," and "single inference path."

[0006] Therefore, there is an urgent need to provide an effective technical solution to solve the above-mentioned technical problems. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this application provides an image reasoning method, apparatus, device, and storage medium. This application avoids inter-task interference and improves versatility, reasoning efficiency, reliability, and stability.

[0008] In a first aspect, embodiments of this application provide an image reasoning method, which includes the following steps: The user-input static image to be reasoned and natural language prompts are input into the trained task routing module, which identifies at least one task category corresponding to the natural language prompts and activates the trained expert agent corresponding to each task category. The static image to be reasoned and the natural language prompts are respectively input into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and the shared semantic representations are input into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The consensus representation of the static image to be reasoned and the natural language prompts are input into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0009] According to an embodiment of this application, an image reasoning method is provided, wherein the task routing module is a dual-branch network structure including a visual encoder with frozen parameters, a text encoder with frozen parameters, and a trainable classification head. The step of training a task routing module that combines the user-input static image to be inferred with natural language prompts to identify at least one task category corresponding to the natural language prompts includes: The visual encoder using the frozen parameters performs visual encoding on the static image to be inferred, resulting in a visual encoding vector. The natural language prompt is text-encoded using the text encoder with the frozen parameters to obtain a text-encoded vector; The visual encoding vector and the text encoding vector are input into a trained classification head to identify at least one task category corresponding to the natural language prompt; the task category includes at least one of object detection, instance segmentation, object counting, visual question answering, and image-text matching.

[0010] According to an embodiment of this application, an image reasoning method is provided, wherein each trained expert agent includes a low-rank adapter corresponding to the task category and a specific projection layer of a shared base model; the shared base model adopts a parameter-frozen multimodal large model. The step of inputting the static image to be reasoned and the natural language prompt into each of the trained expert agents for initial reasoning, and obtaining the shared semantic representation of the static image to be reasoned from the output of each of the trained expert agents, includes: For any of the trained expert agents, cross-modal feature encoding is performed on the static image to be reasoned and the natural language prompt through a specific projection layer of the shared base model to obtain a cross-modal feature sequence; By using the parameters of the low-rank adapter corresponding to the task category, the cross-modal feature sequence is subjected to task-oriented linear modulation to generate an initial inference result, which includes the shared semantic representation of the static image to be inferred.

[0011] According to an embodiment of this application, an image reasoning method is provided, wherein the shared semantic representations are input into a collaborative memory pool for graph fusion to generate a consensus representation of the static image to be reasoned, including: Using the collaborative memory pool, an undirected graph is constructed with each semantic unit in the shared semantic representation as a graph node and the semantic similarity between the graph nodes as edges. Perform graph convolution operation on the undirected graph to output the consensus representation of the static image to be inferred.

[0012] According to an embodiment of this application, an image reasoning method is provided, the method further includes: The attention map generated during the secondary reasoning process and the consensus representation of the static image to be reasoned are input into the trained interpretable generator to generate user-oriented structured explanatory text and visual heatmaps. The interpretable generator is a visual-language dual-branch network that includes a shared cross-attention mechanism; the structured interpretable text includes the premises on which the reasoning is based, the operational logic performed, and the conclusions drawn; the visual heatmap is used to highlight key image regions of interest during the reasoning process.

[0013] According to an embodiment of this application, an image reasoning method is provided, the training steps of the task routing module include: Obtain a sample dataset containing multiple images; the sample dataset includes multiple triplet training samples, each triplet training sample including an image, prompt text, and task category label; the prompt text is a prompt text generated for each image corresponding to multiple predefined visual subtask categories; A two-branch network structure is constructed, including a visual encoder branch and a text encoder branch, and a trainable classification head is connected after the fused features of the two-branch network structure; the visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, and the text encoder branch uses a pre-trained language model with frozen parameters. The classification head is trained using the training samples of each triplet to obtain a trained classification head; the classification head is a multilayer perceptron. Based on the pre-trained and frozen parameter visual Transformer model, the pre-trained and frozen parameter language model, and the trained classification head, the trained task routing module is determined.

[0014] According to an embodiment of this application, an image reasoning method is provided, in which multiple agents include expert agents and a target agent, and the collaborative training steps of the multiple agents include: For each training image, multiple sub-task prompts corresponding to the training image are randomly selected and the answers to the multiple sub-tasks corresponding to the training image are organized, and an independent training sample is constructed for each training image. For each independent training sample of the same training image, the expert agent corresponding to each sub-task is activated to perform the first inference, and the output of each expert agent of the training image is obtained; the output of the expert agent includes the thought process of performing the sub-task and the task answer of the sub-task; Extract fixed-length shared semantic representations from the outputs of each expert agent, and input all the corresponding shared semantic representations of the same training image into the collaborative memory pool to generate a consensus representation of the training image; The consensus representation of the training image is concatenated with the multiple sub-task prompts and then fed into the target agent for secondary reasoning to obtain the corrected answers corresponding to each of the sub-tasks of the training image. The reward function is calculated based on the corrected answers corresponding to the sub-tasks of each training image, the outputs of the expert agents of each training image, and the independent training samples of each training image. Based on the reward function, the adjusted multi-agent system is trained until the training termination condition is met, resulting in the trained expert agents and the trained target agent.

[0015] An image reasoning method provided according to an embodiment of this application can explain the training steps of a generator, including: Construct an interpretable generator that includes visual and linguistic branches; Through the visual branch, the attention map generated by the target agent during the reasoning process of each input image is received, and after convolution and upsampling processing, a visual heatmap corresponding to the size of each input image is output. The language branch receives consensus representations and task-related information of the input images and generates structured explanatory texts corresponding to each input image via autoregression. The visual branch shares a cross-attention mechanism with the language branch to align the structured explanatory texts with the regions of each visual heatmap. Based on the structured interpretable texts and the visual heatmaps, the loss function of the interpretable generator is calculated; The interpretable generator is trained based on the loss function of the interpretable generator to obtain a trained interpretable generator.

[0016] According to an embodiment of this application, an image reasoning method is provided, wherein the loss function of the interpretable generator is a weighted sum of language modeling loss and heatmap alignment loss.

[0017] Secondly, embodiments of this application also provide an image reasoning device, which includes the following modules: The task routing unit is used to input the static image to be reasoned by the user and the natural language prompts into the trained task routing module, identify at least one task category corresponding to the natural language prompts, and activate the trained expert agent corresponding to each task category. The first reasoning unit is used to input the static image to be reasoned and the natural language prompt into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and to input the shared semantic representations into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The secondary reasoning unit is used to input the consensus representation of the static image to be reasoned and the natural language prompts into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0018] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image reasoning method as described above.

[0019] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image reasoning method as described above.

[0020] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the image reasoning method as described above.

[0021] This application provides an image reasoning method, apparatus, device, and storage medium. First, a user-input static image to be reasoned, along with natural language prompts, is input into a trained task routing module. This module identifies at least one task category corresponding to the natural language prompts and activates trained expert agents corresponding to each task category. Then, the static image to be reasoned and the natural language prompts are input into each trained expert agent for initial reasoning, yielding a shared semantic representation of the static image output by each expert agent. These shared semantic representations are then input into a collaborative memory pool for graph fusion, generating a consensus representation of the static image to be reasoned. Finally, the consensus representation of the static image to be reasoned and the natural language prompts are input into a trained target agent for secondary reasoning, obtaining the final answer corresponding to each task category of the static image to be reasoned.

[0022] This application breaks down the "single-model-multi-task-head" approach into a pipeline of "task-aware routing → multi-agent parallel reasoning → intra-graph consensus fusion." Each static image to be reasoned requires only one forward inference to complete sub-tasks such as detection, segmentation, counting, VQA, and image-text matching. Using dynamic task routing, it is possible to finely distinguish user instruction intents, achieving fine-grained task recognition and agent scheduling, thereby enabling multi-agent collaborative reasoning, avoiding inter-task interference, and improving versatility, inference efficiency, reliability, and stability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is one of the flowcharts illustrating the image reasoning method provided in the embodiments of this application.

[0025] Figure 2 This is the second flowchart illustrating the image reasoning method provided in the embodiments of this application.

[0026] Figure 3 This is a schematic diagram of the image reasoning device provided in the embodiments of this application.

[0027] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first node can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] To gain a clearer understanding of the various embodiments provided in this application, the technical content involved in this application will first be described as follows: In the field of multi-task image understanding of static images, existing technologies mainly revolve around three technical routes: "single model multi-task", "image-text comparison learning", and "reinforcement learning optimization".

[0031] 1. Single-model multi-task structure. Gradient conflicts between tasks are severe, adding a new task requires retraining the entire model, resulting in poor scalability, a single inference path, and insufficient interpretability.

[0032] 2. CLIP-based Image-Text Comparison Structure. This structure is based on CLIP and first pre-trains an image-text dual-branch model on hundreds of millions of image-text pairs to learn image-text semantic alignment. The downstream image task is transformed into "image-text matching": the image is processed by a visual encoder to obtain global features, and candidate category names or question + answer options are fed into a text encoder to obtain text features. Cosine similarity is calculated, and the highest value is used as the prediction. During the training phase, only the cue vector is fine-tuned or the model is completely frozen. During the inference phase, zero-shot classification, detection, or VQA can be achieved without any task-specific heads. However, the text side of CLIP mainly uses noun phrases and lacks fine-grained expression of verbs, quantities, and spatial relationships, resulting in insufficient discrimination in tasks such as counting, denotation, and complex question answering. At the same time, single-step similarity calculation cannot generate intermediate inference chains, resulting in weak interpretability.

[0033] 3. Reinforcement Learning-Based Inference Chain Structure. To introduce intermediate inference steps, recent work treats the image-language model as a policy network, allowing it to autoregressively output... <think> <answer>The reasoning chain is formatted as (<thought>><answer>), and then fine-tuned using reinforcement learning (RL) through multi-dimensional rewards such as task accuracy, format compliance, and repetition penalties, including Group Relative Policy Optimization (GRPO) / Proximal Policy Optimization (PPO). During the training phase, only low-rank adaptation (LoRA) updates are performed on the policy model; during the inference phase, a single model completes the end-to-end generation of "thought + answer". This method can generate readable reasoning processes, but it is still limited to a single agent, lacking cross-task information fusion and dynamic routing mechanisms. When facing new or complex tasks, retraining or manually designing prompts is required, resulting in insufficient flexibility.

[0034] The existing technology has the following main problems: 1. Strong task coupling, making modular expansion difficult: The single-model multi-task structure shares parameters between tasks, resulting in severe interference between tasks and making it difficult to quickly adapt and replace modules for new tasks.

[0035] 2. Lack of dynamic task scheduling mechanism: Existing methods usually use a fixed structure to handle all tasks during the inference phase, and cannot dynamically select the optimal sub-model or inference path according to the input characteristics, resulting in resource waste and low inference efficiency.

[0036] 3. The reasoning process is black box and has poor interpretability: Although some methods introduce the "thinking chain" mechanism, the reasoning process still lacks structured expression and visualization support, making it difficult to meet the interpretability requirements in high-credibility scenarios (such as supervision, medical care, and security).

[0037] 4. Lack of multi-task collaboration mechanism: Existing methods are mostly single-agent reasoning and lack collaboration mechanisms between multiple models, making it difficult to achieve cross-modal and cross-task information fusion and complementary reasoning in complex tasks.

[0038] This application aims to address the aforementioned issues by proposing a training method for an image reasoning model based on multi-agent collaboration and dynamic task routing. By introducing three modules—"task-aware routing + multi-agent collaborative reasoning + interpretable generator"—it achieves adaptive task scheduling, collaborative reasoning, and interpretable output, thereby improving the model's versatility, efficiency, and reliability.

[0039] This application aims to address the aforementioned issues by proposing an image reasoning and model training method based on multi-agent collaboration and dynamic task routing. By introducing three major modules—"task-aware routing + multi-agent collaborative reasoning + interpretable generator"—it achieves adaptive task scheduling, collaborative reasoning, and interpretable output, thereby improving the model's versatility, efficiency, and reliability.

[0040] The following is combined with Figures 1 to 4 This application describes image reasoning methods, apparatus, devices, and storage media according to embodiments of the present application.

[0041] Figure 1 This is one of the flowcharts illustrating the image reasoning method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes the following: Step 101: Input the static image to be inferred by the user and the natural language prompts into the trained task routing module, identify at least one task category corresponding to the natural language prompts, and activate the trained expert agent corresponding to each task category.

[0042] It should be noted that the executing entity of this application can be an image reasoning system, which includes a trained task routing module, a trained expert agent, and a trained target agent. Optionally, the image reasoning system also includes a trained interpretable generator.

[0043] Specifically, the trained task routing module is used to distinguish user command intentions in a fine-grained manner and identify the categories of each subtask. The task routing module can be a neural network, such as the Transformer model, which performs feature encoding and task classification in sequence.

[0044] The sub-task categories are visual understanding or image reasoning tasks, such as object detection, instance segmentation, object counting, visual question answering, and image-text matching. These task categories cover the complete spectrum of visual tasks, from low-level perception (localization, segmentation) to mid-level analysis (counting) and then to high-level semantic understanding and reasoning (VQA, image-text matching).

[0045] By inputting the static image to be inferred by the user and the natural language prompts into the trained task routing module, at least one task category corresponding to the natural language prompts can be identified. For example, a 5-dimensional vector is output, where the value of each dimension represents whether the sub-task category corresponding to that dimension is activated. Furthermore, based on the output of the trained task routing module, the trained expert agents corresponding to each task category are activated. For example, a 5-dimensional vector is represented as "[1,0,1,0,0]".

[0046] Step 102: Input the static image to be reasoned and the natural language prompts into each of the trained expert agents for the first reasoning, and obtain the shared semantic representation of the static image to be reasoned output by each trained expert agent. Input the shared semantic representations into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned.

[0047] Specifically, the trained expert agent is used for the first inference, reasoning for each subtask separately to obtain the inference results, which include the consensus representation of the static image to be reasoned.

[0048] After activating the pre-trained expert agents corresponding to each sub-task category, the user input (the static image to be reasoned and natural language prompts) is input into each activated pre-trained expert agent, and parallel initial reasoning is performed to obtain the initial reasoning result corresponding to each sub-task output by each pre-trained expert agent. The initial reasoning result includes agent generation. <think> <answer>Structured output in format, where <think>Thinking process for large models <answer>The answer to the subtask.

[0049] In this context, each initial inference result includes a shared semantic representation of the static image to be inferred corresponding to that subtask (i.e., <think>(For the large model thinking process), such as a fixed number of shared tokens of 32, a token is the basic unit into which a language is broken down when an agent processes language. Its content is human-readable, task-related high-level semantics, serving as a "common language" for exchanging information among expert agents within a single graph. It is used by the Collaborative Memory Pool (CMP) to construct the graph, thereby generating a "consensus representation" of the single graph. The "consensus representation" of the single graph is a vector representation with unlimited dimensions.

[0050] The shared semantic representations are input into the Collaborative Memory Pool (CMP) for graph fusion to generate a consensus representation of the static image to be reasoned. The Collaborative Memory Pool is the sole hub for information exchange among multiple expert agents. It uses 32 shared tokens from all activated expert agents within the same image to perform graph fusion and obtain the consensus representation of the static image to be reasoned.

[0051] That is, after task routing, parallel expert agent inference is performed: the input is fed into the activated expert agents, and each expert agent performs regression generation. <think> <answer>Extract each <think>The shared tokens in the system; CMP constructs a graph of all shared tokens and inputs the calculated "consensus" token into the target agent (master agent) for secondary reasoning.

[0052] Step 103: Input the consensus representation and natural language prompts of the static image to be reasoned into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0053] Specifically, the trained target agent (master agent) is used to perform secondary reasoning based on the consensus representation of the static image to be reasoned and natural language prompts to obtain the final answer corresponding to each task category. That is, the master agent is used to correct the answer of the initial reasoning for each task category. The correction is mainly due to the addition of consensus representation, which enables the master agent to learn to analyze the problem and give the answer based on the "consensus representation" summarized by each expert agent.

[0054] The consensus representation of the static image to be reasoned is concatenated with the original natural language prompts input by the user to obtain updated natural language prompts (input for secondary reasoning). These updated natural language prompts (input for secondary reasoning) are then input into the trained target agent (master agent) for secondary reasoning on tasks of various categories. The results of the secondary reasoning also include those generated by the agent. <think> <answer>Structured output in format, where <think>Thinking process for large models <answer>This is the answer to the subtask. The output of the model after the second inference... <answer>As the final predicted answer, the final answer corresponding to each task category of the static image to be inferred is obtained. <answer> .

[0055] For example, the following is a specific example of image reasoning: (1) Input image A 224x224 red, green, and blue (RGB) image containing the text "A black remote control is on a white coffee table." The input is "Please select the remote control and tell me the quantity." After the task routing analysis input is received, the "Detection Expert Agent" and "Counting Expert Agent" are activated.

[0056] (2) , Input the "detection expert agent" and the "counting expert agent" respectively: The detection expert agent generates the following inference results: The counting expert agent generates the inference results as follows: (3) Based on the above output, extract 64 shared tokens as nodes (32 generated by the detection expert agent + 32 generated by the counting expert agent) to construct a graph, and calculate the "consensus representation" of each expert agent. The "consensus representation" returned to the master agent is "Quantity = 1, located in the center of the coffee table" and "Same remote control has been detected". Then, perform the concatenation of the hints and perform the second reasoning to obtain the final answer.

[0057] (4) Perform interpretable generation to obtain a 256×256 heatmap and interpretable triples: Reason: Premise - A black remote control is visible in the image; Operation - Bounding box regression and counting are performed on the remote control; Conclusion - The number is 1, and the Intersection over Union (IoU) of the bounding box and the ground truth is 0.89.

[0058] The method provided in this application first inputs the static image to be reasoned and natural language prompts into a trained task routing module, identifies at least one task category corresponding to the natural language prompts, and activates the trained expert agents corresponding to each task category; then, the static image to be reasoned and the natural language prompts are respectively input into each trained expert agent for initial reasoning, obtaining the shared semantic representation of the static image to be reasoned output by each trained expert agent, and inputting each shared semantic representation into a collaborative memory pool for graph fusion to generate a consensus representation of the static image to be reasoned; furthermore, the consensus representation of the static image to be reasoned and the natural language prompts are input into a trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0059] This application breaks down the "single-model-multi-task-head" approach into a pipeline of "task-aware routing → multi-agent parallel reasoning → intra-graph consensus fusion." Each static image to be reasoned requires only one forward inference to complete sub-tasks such as detection, segmentation, counting, VQA, and image-text matching. Using dynamic task routing, it is possible to finely distinguish user instruction intents, achieving fine-grained task recognition and agent scheduling, thereby enabling multi-agent collaborative reasoning, avoiding inter-task interference, and improving versatility, inference efficiency, reliability, and stability.

[0060] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0061] According to an embodiment of this application, an image reasoning method is provided, wherein the task routing module is a dual-branch network structure including a visual encoder with frozen parameters, a text encoder with frozen parameters, and a trainable classification head. The user-input static image to be inferred is input into a pre-trained task routing module along with natural language prompts. This module identifies at least one task category corresponding to the natural language prompts, including: By using a visual encoder with frozen parameters, the static image to be inferred is visually encoded to obtain a visual encoding vector. By using a text encoder with frozen parameters, the natural language prompts are text-encoded to obtain a text-encoded vector. The visual encoding vector and text encoding vector are input into the trained classification head to identify at least one task category corresponding to the natural language prompt; the task category includes at least one of object detection, instance segmentation, object counting, visual question answering, and image-text matching.

[0062] Specifically, in some embodiments, the task routing module employs a visual-text dual-branch structure followed by a trainable classification head.

[0063] The visual-text dual-branch structure includes a visual encoding branch (a visual encoder with frozen parameters) and a text encoding branch (a text encoder with frozen parameters). For example, the visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, while the text encoder branch uses a pre-trained language model with frozen parameters.

[0064] Classification heads such as Multi-Layer Perceptrons (MLPs) are a type of feedforward neural network consisting of an input layer, hidden layers, and an output layer. Each neuron receives the output of the neuron in the previous layer, and the weighted sum of these outputs is then processed through a non-linear activation function. MLPs can handle non-linear problems by learning appropriate weights and biases to establish a mapping relationship between inputs and outputs, and are widely used in tasks such as classification and regression.

[0065] The process of identifying task categories using a trained task routing module includes the following steps: First, feature extraction and encoding are performed: A visual encoder with frozen parameters, such as the Vision Transformer Base model with a 16×16 patch size (ViT-B / 16), is used to visually encode the static image to be inferred, resulting in a visual encoding vector. Then, a text encoder with frozen parameters, such as the Qwen 2.5 model with 1.5B parameters (Qwen2.5-1.5B), is used to text-encode the natural language prompts, resulting in a text encoding vector.

[0066] Furthermore, the trained classification head MLP is used for task classification. Specifically, the visual encoding vector and text encoding vector are input into the trained classification head to identify at least one task category corresponding to the natural language prompt.

[0067] The task categories include at least one of object detection, instance segmentation, object counting, visual question answering, and image-text matching. Object detection refers to locating one or more objects of interest in an image, marking their positions with rectangular bounding boxes, and identifying the category of each object. Instance segmentation goes a step further than object detection; instead of using bounding boxes, it precisely outlines the pixel contours of each object instance in the image, achieving pixel-level "cutting out." Object counting involves counting the number of objects in an image that belong to a specified category or match a certain description. Visual question answering involves answering an open-ended natural language question about the content of a given image. Image-text matching determines whether a text description matches the content of an image, or retrieves the image that best matches the description from multiple images.

[0068] The method provided in this application combines a user-inputted static image to be inferred with natural language prompts input into a pre-trained task routing module to identify at least one task category corresponding to the natural language prompts. Based on a dual-branch structure of image and text, it achieves fine-grained task recognition and agent scheduling, improving inference efficiency and system stability.

[0069] According to an embodiment of this application, an image reasoning method is provided, in which each trained expert agent includes a low-rank adapter corresponding to the task category and a specific projection layer of a shared base model; the shared base model adopts a parameter-frozen multimodal large model. The static image to be reasoned and natural language prompts are input into each trained expert agent for initial reasoning, resulting in a shared semantic representation of the static image to be reasoned from the outputs of each trained expert agent, including: For any trained expert agent, cross-modal feature encoding is performed on the static image to be reasoned and the natural language prompts through a specific projection layer of the shared base model to obtain a cross-modal feature sequence; By using the parameters of the low-rank adapter corresponding to the task category, task-oriented linear modulation is applied to the cross-modal feature sequence to generate the first inference result, which includes the shared semantic representation of the static image to be inferred.

[0070] Specifically, in some embodiments, each trained expert agent includes a low-rank adaptor LoRA corresponding to the task category and a specific projection layer of a shared base model, which employs a parameter-frozen multimodal large model. For example, the base uses a parameter-frozen Qwen 2.5 Vision-Language large model (7 billion parameters) (Qwen2.5-VL-7B), and attaches 6 LoRAs as expert agents for 5 visual sub-tasks and a master agent. Each LoRA is only inserted into the query (Q) / value (V) projection layer of the base. By fine-tuning these two projection matrices, the LoRA can most effectively change the model's "focus" (controlled by Q) and "interpretation and integration" of the information of interest (controlled by V) during inference, thereby enabling the same base model to quickly adapt and specialize in specific tasks (such as detection or counting). By using different Q / V-LoRAs for different tasks, task-specific modulation at the attention level is achieved, which is a key technical means to build "plug-and-play" expert agents.

[0071] The initial process for each trained expert agent is shown below: For any trained expert agent, cross-modal feature encoding is performed on the static image to be inferred and the natural language prompts through a specific projection layer of the shared base model, resulting in a cross-modal feature sequence. Then, task-oriented linear modulation is applied to the cross-modal feature sequence using the parameters of the low-rank adapter corresponding to the task category to generate the initial inference result. The initial inference result includes the shared semantic representation of the static image to be inferred.

[0072] The method provided in this application generates initial inference results for each subtask through each agent, thereby extracting a shared semantic representation of the static image to be inferred from the initial inference results to facilitate subsequent secondary inference. This application proposes a multi-agent collaborative inference mechanism, constructing task-specific agents through LoRA fine-tuning and introducing cross-agent attention and consensus propagation algorithms to achieve information fusion and collaborative inference between tasks, thereby improving inference efficiency.

[0073] According to an embodiment of this application, an image reasoning method is provided, in which shared semantic representations are input into a collaborative memory pool for graph fusion to generate a consensus representation of the static image to be reasoned, including: By using a collaborative memory pool, an undirected graph is constructed with each semantic unit in each shared semantic representation as a graph node and the semantic similarity between each graph node as an edge. Perform graph convolution on an undirected graph to output a consensus representation of the static image to be inferred.

[0074] Specifically, in some embodiments, the consensus representation of the static image to be inferred is generated through a collaborative memory pool by means of the following steps: Using a collaborative memory pool, each semantic unit in the shared semantic representation is used as a graph node; that is, each token is a node, and the semantic similarity between graph nodes is used as edges to construct an undirected graph. The undirected graph has a total of 32×k nodes, where k is the number of expert agents activated for the image.

[0075] Furthermore, graph convolution is performed on the undirected graph to output the consensus representation Consensus-Tok of the static image to be inferred.

[0076] The method provided in this application generates a consensus representation of the static image to be reasoned through a collaborative memory pool, and then inputs the consensus representation of the static image to be reasoned into the master agent, so that the master agent can learn to analyze the problem and give the answer based on the "consensus representation" summarized by each expert agent. This achieves consensus fusion within a single image, thereby realizing result alignment and denoising for tasks such as "detection-segmentation-counting-VQA-image matching" within a unified image.

[0077] An image reasoning method according to an embodiment of this application further includes: The attention map generated during the secondary reasoning process and the consensus representation of the static image to be reasoned are input into the trained interpretable generator to generate user-oriented structured explanatory text and visual heatmaps. The interpretable generator is a visual-language dual-branch network that includes a shared cross-attention mechanism; the structured interpretable text includes the premises on which the reasoning is based, the operational logic performed, and the conclusions drawn; the visual heatmap is used to highlight the key image regions of interest in the reasoning process.

[0078] Specifically, in some embodiments, the method further includes interpretable generation, a step used to transform "expert answers + model attention" into natural language understanding and heatmaps that can be directly read by humans at the end of the image reasoning chain using an interpretable generator.

[0079] The interpretable generator is a visual-language dual-branch network with a shared cross-attention mechanism. The visual branch is a two-layer lightweight convolutional neural network (CNN), taking as input a 14×14 static attention map output by an expert agent. Through convolutional kernel upsampling, the CNN transforms and refines this coarse attention distribution map into a high-resolution (256×256), smooth, and semantically focused heatmap, visually highlighting the image regions the model focuses on. The language branch uses a two-layer Transformer decoder to autoregressively generate [premise, operation, conclusion]. The language and visual branches share cross-attention to ensure text-pixel alignment. Cross-attention is key to achieving "image-text aligned interpretation." When the language branch (Transformer decoder) generates each interpretive word, the cross-attention mechanism allows it to "gaze" on and fuse visual information from the visual branch (features processed by the CNN). This ensures that the generated textual explanation (such as "Premise: Black remote control is visible in the image") can be dynamically and accurately associated with the highlighted area (remote control location) in the heat map, achieving a reliable explanation that "points to what is described".

[0080] An example of the interpretable generation process is as follows: The attention map generated during the secondary reasoning process, such as the attention map... The consensus representation of the static image to be reasoned is fed into a pre-trained interpretable generator to generate user-oriented structured explanatory text and visual heatmaps as the final interpretable evidence.

[0081] The structured explanatory text includes the premises upon which the reasoning is based, the operational logic performed, and the conclusion reached, such as the [premise, operation, conclusion] triple. Visual heatmaps are used to highlight key areas of the image relevant to the reasoning process. Specific examples of the [premise, operation, conclusion] triple are as follows: 1. Premise: A white Persian cat is visible in the image, located to the left of the sofa; Operation: Perform bounding box regression on the cat category; Conclusion: IoU between the predicted box and the ground truth box is 0.87, with a center deviation of 3 pixels (px).

[0082] 2. Premise: A red car appears in the image, parked in a white parking space; Operation: Perform instance segmentation on the car's pixels; Conclusion: The mean Intersection over Union (mIoU) of the mask is 0.91, and the mean edge error is 1.2 pixels.

[0083] The method provided in this application proposes an interpretable generation mechanism that transforms the black-box reasoning process into structured language and visual heatmaps, rather than a single thought chain text output, thereby improving the credibility of the reasoning.

[0084] According to an embodiment of this application, an image reasoning method is provided, and the training steps of the task routing module include: Obtain a sample dataset containing multiple images; the sample dataset includes multiple triplet training samples, each triplet training sample includes an image, cue text, and task category label; the cue text is a cue text generated for each image corresponding to multiple predefined visual subtask categories; A two-branch network structure is constructed, consisting of a visual encoder branch and a text encoder branch, and a trainable classification head is connected after the fused features of the two-branch network structure; the visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, and the text encoder branch uses a pre-trained language model with frozen parameters. The classification head is trained using the training samples of each triplet to obtain a trained classification head; the classification head is a multilayer perceptron. Based on a pre-trained visual Transformer model with frozen parameters, a pre-trained language model with frozen parameters, and a trained classification head, a trained task routing module is determined.

[0085] In some embodiments, the model training of this application includes three stages: pre-training of the task routing module, multi-agent collaborative training, and interpretable generator training. A three-stage training strategy is adopted to optimize task routing, multi-agent collaboration, and the interpretable generator respectively, supporting modular replacement and task expansion.

[0086] During the pre-training phase, the task routing module is trained to classify user intent (the category of visual subtasks) based on user input prompts and images. The categories of visual subtasks in this application include at least one of the following: detection, segmentation, counting, VQA, and image-text matching.

[0087] The specific training process includes the following steps: Step 1: Training data preparation (i.e., obtaining a sample dataset containing multiple images): The sample dataset includes multiple triplet training samples: such as<I,T,y> Triple data. Each triple training sample includes an image I (RGB image data, centered and cropped to 224×224), a cue text T, and a task category label y (y is an integer from 0 to 4, used to correspond to 5 visual sub-tasks). The cue text is a cue text generated for each image that corresponds to multiple predefined visual sub-task categories, such as: "Please frame the person wearing the red coat in the picture".

[0088] For example, 7,000 images were extracted from publicly available datasets such as Common Objects in Context (COCO), Referring Expressions on COCO (RefCOCO), Large Vocabulary Instance Segmentation (LVIS), Visual Genome, and Visual Question Answering v2 (VQA-v2). Then, a generative pre-trained Transformer 4 Omni (GPT-4o) model was used to generate five visual subtask prompts for each image. Further, 20,000 prompts were extracted from these prompts.<I,T,y> Triple data was used as the sample dataset.

[0089] Optionally, the process of constructing the sample dataset can also include data augmentation such as random cropping, random horizontal flipping, and color jittering (randomly adjusting brightness, contrast, saturation, and hue) on the image data.

[0090] Optionally, after obtaining the sample dataset, the data can be divided into 90% training and 10% validation.

[0091] Step 2: Constructing the dual-branch network structure: Specifically, a dual-branch network structure is constructed, including a visual encoder branch and a text encoder branch. That is, a visual-text dual-branch structure is adopted. The visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, such as the visual transformer-base model ViT-B / 16, and the text encoder branch uses a pre-trained language model with frozen parameters, such as Qwen2.5-1.5B.

[0092] Furthermore, a trainable classification head, such as a 4-layer Multi-Layer Perceptron (MLP), is connected after the fused features of the dual-branch network (i.e., after the visual-text dual branch). The trained classification head ultimately outputs a 5-dimensional vector representing the classification prediction for 5 visual subtasks, with each dimension indicating whether each visual subtask is activated.

[0093] Step 3, Training Process: The classification head MLP is trained using the training samples of each triplet to obtain a trained classification head. The training configuration is as follows: the optimizer (AdamWeightDecayOptimizer, AdamW) in the bidirectional encoder representation BERT is used, the cross-entropy loss function is selected, and a load balancing regularization term is added. The purpose of the load balancing regularization term is to prevent task routing from always predicting high-frequency tasks.

[0094] Step 4: Output the trained task routing module Based on a pre-trained visual Transformer model with frozen parameters, a pre-trained language model with frozen parameters, and a trained classification head, a trained task routing module is determined.

[0095] The method provided in this application includes a task routing module comprising a visual-text dual-branch structure and a trainable classification head. Through training, the trained task routing module can classify user intent based on user input prompts and images. Based on the visual-text dual-branch structure and load balancing strategy, fine-grained task recognition and agent scheduling are achieved, improving inference efficiency and system stability.

[0096] According to an embodiment of this application, an image reasoning method is provided, in which multiple agents include expert agents and a target agent. The collaborative training steps of the multiple agents include: For each training image, multiple sub-task prompts corresponding to the training image are randomly selected and the answers to the multiple sub-tasks corresponding to the training image are organized. Independent training samples are constructed for each training image. For each independent training sample of the same training image, the expert agent corresponding to each sub-task is activated to perform the first reasoning, and the output of each expert agent of the training image is obtained; the output of the expert agent includes the thought process of performing the sub-task and the task answer of the sub-task; Extract fixed-length shared semantic representations from the outputs of each expert agent, and input all corresponding shared semantic representations of the same training image into the collaborative memory pool to generate a consensus representation of the training image; The consensus representation of the training image is concatenated with multiple sub-task prompts and then fed into the target agent for secondary reasoning to obtain the corrected answers corresponding to each sub-task of the training image. The reward function is calculated based on the corrected answers for each subtask of each training image, the outputs of each expert agent for each training image, and each independent training sample for each training image. Based on the reward function, the adjusted multi-agent system continues to be trained until the training termination condition is met, resulting in trained expert agents and a trained target agent.

[0097] Specifically, the second stage of model training involves fine-tuning (training) the multi-agent system. The training process includes the following steps: Step 1: Training Sample Construction The system still uses the 7000 images pre-trained from the first-stage task routing module, but randomly selects up to three sub-task labels for each image. Specifically, for each training image, multiple sub-task prompts corresponding to the training image are randomly selected (e.g., selecting up to three sub-task labels), and the answers for the multiple sub-tasks corresponding to the training image are compiled. .

[0098] Construct independent training samples for each training image; for example, for each image, pair it with each subtask cue T. i Construct independent samples Where I represents image data. Subtask hints The corresponding answer, Subtask hints The corresponding task number (i.e., the agent number corresponding to the task).

[0099] Step 2: Training the multi-agent system: The multi-agent system in this embodiment includes expert agents (for task inference of different task categories) and a target agent (master agent). The shared base of the multi-agent system uses a Qwen2.5-VL-7B model with frozen parameters, and is equipped with six low-rank adapters (LoRAs) serving as expert agents for five visual sub-tasks and one master agent. Each LoRA is only inserted into the Q / V projection layer of the base. During training, the shared base is kept completely frozen, and only sub-task number m is activated. i The corresponding intelligent agent is in a "power-off" state, while the other intelligent agents are in a "power-off" state.

[0100] For each independent training sample of the same training image, the expert agent corresponding to each sub-task is activated to perform the first inference, resulting in the output of each expert agent for the training image. The output of each expert agent includes the thought process for executing the sub-task and the task answer for that sub-task. <think> <answer>Structured output in format, where <think>Thinking process for large models <answer>The answer to the subtask.

[0101] In comparing image I with G i After being fed into the expert intelligent agent, the generated <think>A fixed number of 32 shared tokens are used, each containing human-readable, task-related high-order semantics. These tokens serve as a "common language" for information exchange among experts within a single graph, and are used by the Collaborative Memory Pool (CMP) to construct the graph and generate "consensus." For example, fixed-length shared semantic representations are extracted from the outputs of each expert agent, and all corresponding shared semantic representations of the same training image are input into the CMP to generate a consensus representation for that training image.

[0102] Furthermore, the consensus representation of the training images is combined with multiple subtask cues T i After being spliced, the data is fed into the target agent (master agent) for secondary reasoning to obtain the corrected answers for each subtask of the training image.

[0103] Step 3, Rewards and Iteration The reward function is calculated based on the corrected answers for each subtask of each training image, the outputs of each expert agent for each training image, and each independent training sample for each training image.

[0104] The model was trained using reinforcement learning and the Group Relative Policy Optimization (GRPO) algorithm, which generates four responses for each static image-hint pair.

[0105] This application sets the reward function as follows: in, Used for measurement <answer>The output is checked against the required standard format. A value of 1 indicates a valid format, while a value of 0 indicates otherwise. Used to inhibit <think>Repetition makes the explanation more concise. ; Used for measurement <answer>The output indicates whether the result is correct; 1 indicates correct and 0 indicates incorrect. Forced graph convolutional consensus must deliver a measurable improvement, the specific formula of which is as follows: in, For the difference in reasoning results, The result of the second inference after incorporating "consensus" This is the result of the first expert agent's reasoning. As the standard answer, The evaluation index is calculated by comparing the result of the second reasoning with the standard answer. This is an evaluation metric for comparing the results of the first expert agent's reasoning with the calculated standard answer.

[0106] when hour, ;when hour, ;when hour, .

[0107] After obtaining the value of the reward function, the adjusted multi-agent system is trained based on the reward function until the training termination condition is met, resulting in trained expert agents and a trained target agent, thus completing the training of the multi-agent system.

[0108] In order to enable various specialized models such as "detection-segmentation-counting-VQA-image matching" within an image to share local cognition and complement each other for correction, thereby improving the consistency and accuracy of multi-tasks, the method provided in this application designs a multi-agent collaborative structure to achieve modular plug-and-play, cross-task information exchange within the same image, and secondary denoising inference.

[0109] An image reasoning method provided according to an embodiment of this application can explain the training steps of a generator, including: Construct an interpretable generator that includes visual and linguistic branches; Through the visual branch, the attention map generated by the target agent during the reasoning process of each input image is received. After convolution and upsampling, the visual heatmap corresponding to the size of each input image is output. The language branch receives consensus representations and task-related information from the input images and generates structured explanatory texts for each input image via autoregression. The visual branch shares a cross-attention mechanism with the language branch to align the structured explanatory texts with the regions of each visual heatmap. Based on each structured interpretable text and each visual heatmap, the loss function of the interpretable generator is calculated; The interpretable generator is trained using the loss function of the interpretable generator to obtain a trained interpretable generator.

[0110] In some embodiments, the third stage of model training can explain the generator training (fine-tuning) steps, including: Step 1: Obtain training data: Two thousand data points were selected from the second phase of multi-agent collaborative training. Using a combination of manual annotation and a Generative Pre-trained Transformer 4 Omni (GPT-4o) model, [premise, operation, conclusion] triples were labeled. "Key regions" in the graph were then manually outlined as heatmap rectangles to supervise heatmap alignment. Specific examples of [premise, operation, conclusion] triples are shown below: Premise: A white Persian cat is visible in the image, located to the left of the sofa; Operation: Perform bounding box regression on the cat category; Conclusion: IoU between the predicted box and the ground truth box = 0.87, with a center deviation of 3px.

[0111] Premise: A red car appears in the image, parked in a white parking space; Operation: Perform instance segmentation on the car's pixels; Conclusion: Mask mIoU = 0.91, mean edge error 1.2px.

[0112] Step 2: Network Structure Construction and Output An interpretable generator is constructed, comprising a visual branch and a language branch. The visual branch is a two-layer lightweight CNN, taking a 14×14 static attention map output by an expert agent as input, upsampled to 256×256, and outputting a heatmap of the image. The language branch uses a two-layer transformer decoder to autoregressively generate [premise, operation, conclusion]. The language branch shares cross-attention with the visual branch to ensure text-to-pixel alignment.

[0113] Step 3, Training Configuration: Based on the output of each branch, the model parameters are adjusted using the AdamW optimizer and the loss function of the interpretable generator until the iteration termination condition is met, resulting in a trained interpretable generator. The loss function of the interpretable generator includes language modeling loss and heatmap alignment loss.

[0114] According to an embodiment of this application, an image reasoning method can be interpreted as a weighted sum of language modeling loss and heatmap alignment loss.

[0115] Specifically, in some embodiments, the generator's loss function can be interpreted as follows: in, L_lm is the loss function for the interpretable generator, L_lm is the language modeling cross-entropy (language modeling loss), and L_heat is the pixel-wise L2 distance between the predicted heatmap and the artificial bounding box mask (heatmap alignment loss).

[0116] The method provided in this application provides an interpretable generator that is designed and trained to transform "expert answers + model attention" into natural language understanding and heatmaps that humans can directly read at the end of the image reasoning chain, rather than a single thought chain text output, which greatly expands interpretability.

[0117] Figure 2 This is a second schematic flowchart of the image reasoning method provided in the embodiments of this application, such as... Figure 2 As shown, the method includes the following steps: Receive user input, including prompts (T) and images (I); Input the prompt T and image I into the task route, obtain the task classification result, and activate the corresponding expert agent; Parallel expert agent reasoning yields the initial reasoning result, which includes... <answer>(Initial reasoning) and <think>(Initial reasoning); Based on the initial inference results <think>A shared token (semantic representation) is extracted and input into the CMP (Cooperative Memory Pool) to generate a "consensus" token (consensus representation); alternatively, a "consensus" token (consensus representation) can be generated directly based on the hint T. Combine the prompt T with "consensus (representation)" to generate secondary inference input; Based on the secondary inference input and the activation of the master agent by image I, the final answer is obtained. <answer> 、 <think>(Secondary reasoning) and attention maps; Interpretable generation steps are performed based on attention maps, cues T, and "consensus (representation)". The output of interpretable generation includes heatmaps and [premise, action, conclusion] triples.

[0118] The image reasoning apparatus provided in the embodiments of this application is described below. The image reasoning apparatus described below and the image reasoning method described above can be referred to and correspond to each other.

[0119] Figure 3 This is a schematic diagram of the image reasoning device provided in the embodiments of this application, such as... Figure 3 As shown, the image reasoning device 300 includes the following modules: The task routing unit 310 is used to input the static image to be reasoned by the user and the natural language prompts into the trained task routing module, identify at least one task category corresponding to the natural language prompts, and activate the trained expert agent corresponding to each task category. The first reasoning unit 320 is used to input the static image to be reasoned and the natural language prompt into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and to input the shared semantic representations into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The secondary reasoning unit 330 is used to input the consensus representation of the static image to be reasoned and the natural language prompts into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0120] This application breaks down the "single-model-multi-task-head" approach into a pipeline of "task-aware routing → multi-agent parallel reasoning → intra-graph consensus fusion." Each static image to be reasoned requires only one forward inference to complete sub-tasks such as detection, segmentation, counting, VQA, and image-text matching. Using dynamic task routing, it is possible to finely distinguish user instruction intents, achieving fine-grained task recognition and agent scheduling, thereby enabling multi-agent collaborative reasoning, avoiding inter-task interference, and improving versatility, inference efficiency, reliability, and stability.

[0121] According to an embodiment of this application, an image reasoning device 300 is provided, wherein the task routing module is a dual-branch network structure including a visual encoder with frozen parameters, a text encoder with frozen parameters, and a trainable classification head. The task routing unit 310 is specifically used for: The visual encoder using the frozen parameters performs visual encoding on the static image to be inferred, resulting in a visual encoding vector. The natural language prompt is text-encoded using the text encoder with the frozen parameters to obtain a text-encoded vector; The visual encoding vector and the text encoding vector are input into a trained classification head to identify at least one task category corresponding to the natural language prompt; the task category includes at least one of object detection, instance segmentation, object counting, visual question answering, and image-text matching.

[0122] According to an embodiment of this application, an image reasoning device 300 is provided, wherein each trained expert agent includes a low-rank adapter corresponding to the task category and a specific projection layer of a shared base model; the shared base model adopts a parameter-frozen multimodal large model. The initial inference unit 320 is specifically used for: For any of the trained expert agents, cross-modal feature encoding is performed on the static image to be reasoned and the natural language prompt through a specific projection layer of the shared base model to obtain a cross-modal feature sequence; By using the parameters of the low-rank adapter corresponding to the task category, the cross-modal feature sequence is subjected to task-oriented linear modulation to generate an initial inference result, which includes the shared semantic representation of the static image to be inferred.

[0123] According to an embodiment of this application, an image reasoning device 300 is provided, wherein the initial reasoning unit 320 is further configured to: Using the collaborative memory pool, an undirected graph is constructed with each semantic unit in the shared semantic representation as a graph node and the semantic similarity between the graph nodes as edges. Perform graph convolution operation on the undirected graph to output the consensus representation of the static image to be inferred.

[0124] An image reasoning device 300 according to an embodiment of this application further includes an interpretable generation unit; The interpretable generation unit is used for: The attention map generated during the secondary reasoning process and the consensus representation of the static image to be reasoned are input into the trained interpretable generator to generate user-oriented structured explanatory text and visual heatmaps. The interpretable generator is a visual-language dual-branch network that includes a shared cross-attention mechanism; the structured interpretable text includes the premises on which the reasoning is based, the operational logic performed, and the conclusions drawn; the visual heatmap is used to highlight key image regions of interest during the reasoning process.

[0125] An image reasoning device 300 according to an embodiment of this application further includes a model training unit; The model training unit is used for: Obtain a sample dataset containing multiple images; the sample dataset includes multiple triplet training samples, each triplet training sample including an image, prompt text, and task category label; the prompt text is a prompt text generated for each image corresponding to multiple predefined visual subtask categories; A two-branch network structure is constructed, including a visual encoder branch and a text encoder branch, and a trainable classification head is connected after the fused features of the two-branch network structure; the visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, and the text encoder branch uses a pre-trained language model with frozen parameters. The classification head is trained using the training samples of each triplet to obtain a trained classification head; the classification head is a multilayer perceptron. Based on the pre-trained and frozen parameter visual Transformer model, the pre-trained and frozen parameter language model, and the trained classification head, the trained task routing module is determined.

[0126] An image reasoning device 300 according to an embodiment of this application includes a multi-agent system comprising expert agents and a target agent. The model training unit is further configured to: For each training image, multiple sub-task prompts corresponding to the training image are randomly selected and the answers to the multiple sub-tasks corresponding to the training image are organized, and an independent training sample is constructed for each training image. For each independent training sample of the same training image, the expert agent corresponding to each sub-task is activated to perform the first inference, and the output of each expert agent of the training image is obtained; the output of the expert agent includes the thought process of performing the sub-task and the task answer of the sub-task; Extract fixed-length shared semantic representations from the outputs of each expert agent, and input all the corresponding shared semantic representations of the same training image into the collaborative memory pool to generate a consensus representation of the training image; The consensus representation of the training image is concatenated with the multiple sub-task prompts and then fed into the target agent for secondary reasoning to obtain the corrected answers corresponding to each of the sub-tasks of the training image. The reward function is calculated based on the corrected answers corresponding to the sub-tasks of each training image, the outputs of the expert agents of each training image, and the independent training samples of each training image. Based on the reward function, the adjusted multi-agent system is trained until the training termination condition is met, resulting in the trained expert agents and the trained target agent.

[0127] According to an embodiment of this application, an image reasoning device 300 is provided, wherein the model training unit is further configured to: Construct an interpretable generator that includes visual and linguistic branches; Through the visual branch, the attention map generated by the target agent during the reasoning process of each input image is received, and after convolution and upsampling processing, a visual heatmap corresponding to the size of each input image is output. The language branch receives consensus representations and task-related information of the input images and generates structured explanatory texts corresponding to each input image via autoregression. The visual branch shares a cross-attention mechanism with the language branch to align the structured explanatory texts with the regions of each visual heatmap. Based on the structured interpretable texts and the visual heatmaps, the loss function of the interpretable generator is calculated; The interpretable generator is trained based on the loss function of the interpretable generator to obtain a trained interpretable generator.

[0128] According to an embodiment of this application, an image reasoning device 300 is provided, wherein the loss function of the interpretable generator is a weighted sum of language modeling loss and heatmap alignment loss.

[0129] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute an image reasoning method, which includes: The user-input static image to be reasoned and natural language prompts are input into the trained task routing module, which identifies at least one task category corresponding to the natural language prompts and activates the trained expert agent corresponding to each task category. The static image to be reasoned and the natural language prompts are respectively input into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and the shared semantic representations are input into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The consensus representation of the static image to be reasoned and the natural language prompts are input into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0130] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image reasoning method provided by the above methods, the method including: The user-input static image to be reasoned and natural language prompts are input into the trained task routing module, which identifies at least one task category corresponding to the natural language prompts and activates the trained expert agent corresponding to each task category. The static image to be reasoned and the natural language prompts are respectively input into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and the shared semantic representations are input into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The consensus representation of the static image to be reasoned and the natural language prompts are input into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0132] In another aspect, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image reasoning methods provided by the methods described above, the method comprising: The user-input static image to be reasoned and natural language prompts are input into the trained task routing module, which identifies at least one task category corresponding to the natural language prompts and activates the trained expert agent corresponding to each task category. The static image to be reasoned and the natural language prompts are respectively input into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and the shared semantic representations are input into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The consensus representation of the static image to be reasoned and the natural language prompts are input into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / think> < / answer> < / think> < / think> < / answer> < / answer> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / answer> < / answer> < / think> < / answer> < / think> < / think> < / answer> < / think> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / think>

Claims

1. An image reasoning method, characterized in that, include: The user-input static image to be reasoned and natural language prompts are input into the trained task routing module, which identifies at least one task category corresponding to the natural language prompts and activates the trained expert agent corresponding to each task category. The static image to be reasoned and the natural language prompts are respectively input into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and the shared semantic representations are input into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The consensus representation of the static image to be reasoned and the natural language prompts are input into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

2. The image reasoning method according to claim 1, characterized in that, The task routing module is a dual-branch network structure that includes a visual encoder with frozen parameters, a text encoder with frozen parameters, and a trainable classification head. The step of training a task routing module that combines the user-input static image to be inferred with natural language prompts to identify at least one task category corresponding to the natural language prompts includes: The visual encoder using the frozen parameters performs visual encoding on the static image to be inferred, resulting in a visual encoding vector. The natural language prompt is text-encoded using the text encoder with the frozen parameters to obtain a text-encoded vector; The visual encoding vector and the text encoding vector are input into a trained classification head to identify at least one task category corresponding to the natural language prompt; the task category includes at least one of object detection, instance segmentation, object counting, visual question answering, and image-text matching.

3. The image reasoning method according to claim 1, characterized in that, Each trained expert agent includes a low-rank adapter corresponding to the task category and a specific projection layer of the shared base model; the shared base model adopts a parameter-frozen multimodal large model. The step of inputting the static image to be reasoned and the natural language prompt into each of the trained expert agents for initial reasoning, and obtaining the shared semantic representation of the static image to be reasoned from the output of each of the trained expert agents, includes: For any of the trained expert agents, cross-modal feature encoding is performed on the static image to be reasoned and the natural language prompt through a specific projection layer of the shared base model to obtain a cross-modal feature sequence; By using the parameters of the low-rank adapter corresponding to the task category, the cross-modal feature sequence is subjected to task-oriented linear modulation to generate an initial inference result, which includes the shared semantic representation of the static image to be inferred.

4. The image reasoning method according to claim 1, characterized in that, The step of inputting the shared semantic representations into a collaborative memory pool for graph fusion to generate a consensus representation of the static image to be inferred includes: Using the collaborative memory pool, an undirected graph is constructed with each semantic unit in the shared semantic representation as a graph node and the semantic similarity between the graph nodes as edges. Perform graph convolution operation on the undirected graph to output the consensus representation of the static image to be inferred.

5. The image reasoning method according to any one of claims 1-4, characterized in that, The method further includes: The attention map generated during the secondary reasoning process and the consensus representation of the static image to be reasoned are input into the trained interpretable generator to generate user-oriented structured explanatory text and visual heatmaps. The interpretable generator is a visual-language dual-branch network that includes a shared cross-attention mechanism; the structured interpretable text includes the premises on which the reasoning is based, the operational logic performed, and the conclusions drawn; the visual heatmap is used to highlight key image regions of interest during the reasoning process.

6. The image reasoning method according to any one of claims 1-4, characterized in that, The training steps for the task routing module include: Obtain a sample dataset containing multiple images; the sample dataset includes multiple triplet training samples, each triplet training sample including an image, prompt text, and task category label; the prompt text is a prompt text generated for each image corresponding to multiple predefined visual subtask categories; A two-branch network structure is constructed, including a visual encoder branch and a text encoder branch, and a trainable classification head is connected after the fused features of the two-branch network structure; the visual encoder branch uses a pre-trained visual Transformer model with frozen parameters, and the text encoder branch uses a pre-trained language model with frozen parameters. The classification head is trained using the training samples of each triplet to obtain a trained classification head; the classification head is a multilayer perceptron. Based on the pre-trained and frozen parameter visual Transformer model, the pre-trained and frozen parameter language model, and the trained classification head, the trained task routing module is determined.

7. The image reasoning method according to any one of claims 1-4, characterized in that, The multi-agent system includes expert agents and a target agent, and the collaborative training steps of the multi-agent system include: For each training image, multiple sub-task prompts corresponding to the training image are randomly selected and the answers to the multiple sub-tasks corresponding to the training image are organized, and an independent training sample is constructed for each training image. For each independent training sample of the same training image, the expert agent corresponding to each sub-task is activated to perform the first inference, and the output of each expert agent of the training image is obtained; the output of the expert agent includes the thought process of performing the sub-task and the task answer of the sub-task; Extract fixed-length shared semantic representations from the outputs of each expert agent, and input all the corresponding shared semantic representations of the same training image into the collaborative memory pool to generate a consensus representation of the training image; The consensus representation of the training image is concatenated with the multiple sub-task prompts and then fed into the target agent for secondary reasoning to obtain the corrected answers corresponding to each of the sub-tasks of the training image. The reward function is calculated based on the corrected answers corresponding to the sub-tasks of each training image, the outputs of the expert agents of each training image, and the independent training samples of each training image. Based on the reward function, the adjusted multi-agent system is trained until the training termination condition is met, resulting in the trained expert agents and the trained target agent.

8. The image reasoning method according to any one of claims 1-4, characterized in that, The training steps of an interpretable generator include: Construct an interpretable generator that includes visual and linguistic branches; Through the visual branch, the attention map generated by the target agent during the reasoning process of each input image is received, and after convolution and upsampling processing, a visual heatmap corresponding to the size of each input image is output. The language branch receives consensus representations and task-related information of the input images and generates structured explanatory texts corresponding to each input image via autoregression. The visual branch shares a cross-attention mechanism with the language branch to align the structured explanatory texts with the regions of each visual heatmap. Based on the structured interpretable texts and the visual heatmaps, the loss function of the interpretable generator is calculated; The interpretable generator is trained based on the loss function of the interpretable generator to obtain a trained interpretable generator.

9. The image reasoning method according to claim 8, characterized in that, The loss function of the interpretable generator is a weighted sum of the language modeling loss and the heatmap alignment loss.

10. An image reasoning device, characterized in that, include: The task routing unit is used to input the static image to be reasoned by the user and the natural language prompts into the trained task routing module, identify at least one task category corresponding to the natural language prompts, and activate the trained expert agent corresponding to each task category. The first reasoning unit is used to input the static image to be reasoned and the natural language prompt into each of the trained expert agents for the first reasoning, to obtain the shared semantic representation of the static image to be reasoned output by each of the trained expert agents, and to input the shared semantic representations into the collaborative memory pool for graph fusion to generate the consensus representation of the static image to be reasoned. The secondary reasoning unit is used to input the consensus representation of the static image to be reasoned and the natural language prompts into the trained target agent for secondary reasoning to obtain the final answer corresponding to each task category of the static image to be reasoned.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image reasoning method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image reasoning method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image reasoning method as described in any one of claims 1 to 9.

Images