Multi-modal big language model attribute prediction method based on multi-modal thinking chain

Abstract

The invention discloses a multi-modal thinking chain-based multi-modal large language model attribute prediction method, which comprises the following steps of: obtaining a multi-modal large language model, and forming a multi-modal attribute prediction framework together with a mask generator and a scene graph analyzer; designing an inference method of a hierarchical thinking chain, inputting an attribute prediction task into a multi-modal attribute prediction framework, extracting the hierarchy of an attribute table, and generating a hierarchical and formatted inference prompt; decomposing an attribute prediction task into hierarchical sub-tasks, and selecting a prompt method for the next step of prediction of the model; supervising a model prediction process through critical thinking; identifying object categories in the image by adopting a logic check prompt method, and extracting corresponding attribute values in combination with common knowledge in the model; and combining the generated scene graph with common knowledge in the model to check the predicted attributes. According to the method, the context understanding capability, the logic consistency, the task universality and the model reasoning efficiency of the attribute prediction task can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a multimodal large language model attribute prediction method based on a multimodal thinking chain. Background Art

[0002] In the field of artificial intelligence, the rapid development of multimodal large language models (MLLMs) is redefining the capabilities of human-computer interaction and intelligent systems. These models can simultaneously process and generate information in multiple modalities (such as vision, language, and audio), significantly expanding the application scope of traditional unimodal models. In recent years, with the release of models such as GPT-4V and Gemini, the performance of MLLMs on traditional visual tasks, especially attribute detection tasks (AD), has attracted widespread attention.

[0003] The attribute prediction task is one of the core tasks in the field of computer vision and multimodal reasoning, which involves inferring the visual attributes of objects (such as color, material) and more abstract non-visual attributes (such as function, state) from images. Traditional methods mainly rely on classic feature extraction networks and trained classifiers and specific attribute datasets. However, these methods have obvious deficiencies in generalization ability and flexibility. For example, classifiers with fixed output spaces cannot adapt to changing task requirements, and fine-tuning the model for each new task increases the computational cost; the traditional setting of vertical field datasets-pre-training and fine-tuning methods is very inefficient.

[0004] MLLM provides a new approach to solving these problems by implementing dynamic task definition through natural language prompts (prompt learning). The design of prompts enables the model to adapt to task requirements based on context and significantly improves its inference capabilities in zero-shot and few-shot scenarios. This flexibility not only reduces the cost of retraining, but also makes the model more consistent with human cognitive processes. For example, MLLM can capture the affiliation between object categories and attributes through natural language prompts, thereby achieving more explanatory and reasoning predictions.

[0005] In recent years, researchers have proposed a novel prompting paradigm, Multimodal Chain of Thought (MCoT), which aims to unleash the potential of MLLM. MCoT decomposes complex tasks into a series of explainable subtasks, gradually reasoning and ultimately generating reliable prediction results. For example, in the DetToolChain method, the prompting toolkit guides MLLM to focus on the target area, combining magnification tools, overlay scene graphs, and measurement tools for refined reasoning. This approach not only significantly improves the detection accuracy of the model, but also diagnoses and optimizes the prediction results, showing excellent generalization capabilities.

[0006] Unlike traditional closed vocabulary tasks, open vocabulary tasks require the model to have strong generalization capabilities. The OVAD task further increases the difficulty because it requires the model to not only locate objects in the image, but also infer multiple attributes of the object, including attribute categories that did not appear explicitly in training. This type of task usually involves two major capabilities: Visual basis tasks (grounding): The model needs to accurately locate and identify the target area in the image. Attribute reasoning tasks (reasoning): The model needs to combine visual and language information for complex reasoning, identify visual and non-visual attributes, and explore the semantic relationship between "what" and "how".

[0007] Traditional solutions mostly use similarity comparison-based methods to model the attribute prediction task as a similarity calculation between an attribute and a region of interest (ROI). For example, the OV-DAR framework achieves excellent performance on unseen categories and attributes by combining a category-independent object proposal network and an open vocabulary classification module. However, this approach ignores the subordinate relationship between object categories and attributes, making it difficult for the model to capture complex semantic connections. In addition, existing benchmark tasks and methods focus too much on visual attributes, while the inference ability of non-visual attributes has not been fully explored. Therefore, the development of an MLLM-based open vocabulary attribute detection (OVAD) method has shown significant advantages in flexibility, cost, and human cognitive alignment. Summary of the invention

[0008] The present invention provides a multimodal large language model attribute prediction method based on a multimodal thinking chain, which can effectively improve the context understanding ability, logical consistency, task versatility and model reasoning efficiency of attribute prediction tasks, and is suitable for multi-scenario applications of multimodal large language models.

[0009] A multimodal large language model attribute prediction method based on multimodal thought chain includes the following steps:

[0010] (1) Obtain a large multimodal language model and its pre-trained weights;

[0011] (2) Construct a mask generator and a scene graph parser, which together with the multimodal large language model form a multimodal attribute prediction framework; the mask generator is used to segment objects in the image to generate independent masks, and the scene graph parser is used to extract the spatial and semantic relationships between objects in the image and generate sufficient context information;

[0012] (3) Design a hierarchical reasoning chain method, input the attribute prediction task into the multimodal attribute prediction framework for processing, extract the hierarchy of the attribute table, and generate hierarchical and formatted reasoning prompts; through a dynamic task decomposition strategy, decompose the attribute prediction task into hierarchical subtasks, and select prompt methods for the next step of prediction of the multimodal large language model; supervise the model prediction process through critical thinking;

[0013] (4) Using a logical check prompt method, first identify the object category in the image, and extract the corresponding attribute value based on the common sense knowledge in the model; then check the predicted attributes based on the scene graph generated by the scene graph parser and the common sense knowledge in the model.

[0014] The method of the present invention combines visual cues, scene graph parsing, common sense knowledge database and hierarchical thinking chain. The model can dynamically adjust the reasoning path and generate high-precision responses with detailed attribute descriptions when dealing with complex scenarios.

[0015] The present invention can replace fixed classifiers and hard-coded attribute text with natural language prompts, and the model can quickly adapt to new attribute prediction tasks, reducing the need for retraining and significantly reducing the cost of implementing complex prediction tasks.

[0016] The method of the present invention can be extended to a variety of application scenarios, including but not limited to medical image diagnosis, smart home attribute annotation, social platform graphic analysis, and other complex tasks that require attribute prediction.

[0017] In step (2), the mask generator generates a high-precision mask corresponding to the object area by analyzing the input image, shielding background interference and ensuring that the multimodal large language model focuses on the target area, thereby improving the accuracy of attribute prediction.

[0018] The scene graph parser combines classification capabilities, image description capabilities and knowledge graph technology to parse the scene relationships in the input image, generate a graph structure that describes the semantic relationships between objects, and generate sufficient contextual information to provide to the multimodal large language model for reasoning, thereby enhancing its ability to understand contextual information.

[0019] In step (3), the input attribute prediction task first constructs a hierarchical tree structure of the attribute table and generates exclusive reasoning hints for each level.

[0020] Select a prompt method for the next prediction of the multimodal large language model, specifically:

[0021] The subsequent reasoning path is adjusted according to the output of each stage of the multimodal large language model to improve prediction efficiency and result reliability.

[0022] Monitor the model prediction process through critical thinking, specifically:

[0023] In the reasoning process of the hierarchical thinking chain, the multimodal large language model uses the critical thinking module to verify and feedback the prediction results at each step, forming a closed-loop reasoning to ensure the accuracy and consistency of the final response.

[0024] The critical thinking module uses the logical reasoning ability of the model to verify the output results of each subtask in the task layer by layer. If an anomaly is found, it will automatically backtrack to the previous step and regenerate a more logical result.

[0025] The specific process of step (4) is as follows:

[0026] The attribute prediction mechanism based on object categories can efficiently extract key-value pairs of its core attributes by identifying the object categories in the image and combining them with the common sense knowledge database of the multimodal large language model; intelligently match the common attribute sets of each category to generate a preliminary attribute prediction set;

[0027] Combined with the scene graph information generated by the scene graph parser, the model cross-compares visual content with common sense knowledge to detect inconsistencies in attribute predictions and errors that violate common sense, and automatically generates explanatory feedback.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention designs a reasoning method of hierarchical thinking chain. Hierarchical subtask reasoning replaces the direct visual-text vector similarity calculation, checks the logical relationship between category-attribute key and attribute key-value, avoids the occurrence of wrong combinations, and provides the model with self-correction ability.

[0030] 2. The present invention introduces a visual prompt tool, which shields background interference unrelated to the object through an object mask generator, thereby improving the ability to understand visual attributes; and generates a scene graph through a scene graph parser to model the relationship between objects, thereby improving the ability to understand invisible attributes.

[0031] 3. The present invention supports dynamic definition of tasks and attributes through natural language prompts, so that the model can flexibly adapt to different scenarios and avoid hard coding of attribute text. It eliminates the reliance on knowledge distillation, fine-tuning, building targeted data sets, and training specific classifiers, significantly reducing development costs and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an overall flow chart of a multimodal large language model attribute prediction method based on a multimodal thinking chain according to an embodiment of the present invention;

[0033] Figure 2 This is a diagram showing the effect of the mask generator generating an independent mask in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0035] like Figure 1 As shown, a multimodal large language model attribute prediction method based on a multimodal thinking chain mainly includes the following steps:

[0036] Step 1: Deploy a multimodal large language model and adjust the dataset.

[0037] (1.1) Download the model and related weights

[0038] Downloading models and related weights is one of the basic steps in building the system. In this process, it is necessary to obtain the open source model and its supporting weight files, and ensure that its version matches the requirements of the target task. At the same time, for some closed-source models, it is also necessary to complete the configuration and authorization of the interface according to specific needs. This not only includes accessing and downloading model resources, but also involves initialization and debugging verification of the model environment to ensure that the model can be loaded and run normally in a local or cloud environment. Through the improvement of this link, the system can provide a solid technical foundation for subsequent function realization and optimization. For example: GPT-4V, Gemini Ultra, InternLM-XComposer, LLaVA-OneVision-Chat, Qwen-VL.

[0039] (1.2) Download attribute prediction related datasets and adjust the format

[0040] Downloading attribute prediction related datasets is a basic step in performing attribute prediction tasks. In this step, it is necessary to obtain multiple public or dedicated attribute prediction datasets and ensure that they cover the required attribute categories and task scenarios. After downloading, the dataset needs to be preprocessed, including cleaning, formatting, and standardization to ensure that the structure of the data can adapt to the requirements of batch input. This may involve converting the data from the original format to an input form supported by the model, such as unifying the file structure, adjusting the data label format, normalizing numerical attributes, or generating a specific hierarchical input format. Through this process, not only can the data loading efficiency be improved, but also the performance and stability of the model in the training and inference stages can be ensured, laying a solid data foundation for the attribute prediction task. For example: MSCOCO, VAW, LSA, OVAD.

[0041] Step 2: Design and implement a multimodal thinking chain.

[0042] Designing and implementing a multimodal thinking chain plays a key role in attribute prediction tasks. It can improve the model's inference performance and decision-making effect in complex scenarios. The following are the core components of the design and development of a multimodal thinking chain, including hierarchical prompt generation, task decomposition and prompt method selection, and critical thinking supervision. These modules together build an efficient multimodal reasoning system.

[0043] (2.1) Hierarchical prompt generation

[0044] Hierarchical prompt generation is a process of parsing the attribute information in the data set, extracting and constructing the hierarchical structure of the attribute table, and then processing these attributes in a hierarchical manner. This hierarchical processing method can clearly organize and express complex attribute relationships, providing a solid foundation for multi-level reasoning. On this basis, according to the hierarchical characteristics of the attributes, reasoning prompt templates (Prompt) adapted to each level are designed. These templates not only accurately capture the core information of each level, but also ensure that the generated prompts are logically coherent and easy to read. Finally, the generated formatted reasoning prompts can be efficiently understood by the multimodal large language model (MLLM), helping the model to more accurately process hierarchical information during the reasoning process and improve the overall reasoning ability and accuracy.

[0045] (2.2) Task decomposition and prompt method selection

[0046] When faced with complex attribute prediction tasks, decomposing the overall task into multiple smaller subtasks is an effective strategy. By decomposing the task, the complexity of the problem can be reduced, making each subtask easier to solve. Based on the task decomposition, it is crucial to select the most suitable tools and prompt methods (Prompt) according to the characteristics and requirements of each subtask. By matching the most appropriate tools and prompt methods for each subtask, we can maximize the performance of the model and improve the accuracy and efficiency of attribute prediction.

[0047] (2.3) Critical thinking supervision

[0048] In order to further improve the model's problem-solving effect, the present invention introduces a critical thinking supervision (CTS) mechanism. The core of CTS is to supervise the entire reasoning process of the model, and judge in real time whether the reasoning is reasonable, whether the result is correct, and whether the reasoning process reaches the final conclusion. When deviations or errors are found in the reasoning, CTS can guide the model to trace back to the previous reasoning level, re-analyze and generate a more logical reasoning path.

[0049] Step 3: Implementation of visual prompting method.

[0050] The implementation of visual cueing plays a vital role. It can effectively improve the accuracy and efficiency of processing tasks by generating visual content through intelligent tools. The following are the main modules of the visual cueing implementation, which together constitute a complete visual guidance framework.

[0051] (3.1) Mask generator

[0052] In the process of building the mask generator, one of its core functions is to improve the accuracy of attribute prediction by generating object masks. We use class-independent models (such as Segment Anything Model, SAM) to segment objects in the image and generate independent masks. These masks can effectively eliminate background interference and ensure that attribute prediction focuses on the characteristics of each target object itself, such as Figure 2 shown.

[0053] (3.2) Scene Graph Parser

[0054] Generating a description of the relationship between objects in an image is a key step in improving the model's ability to understand. To this end, the present invention designs and implements a scene graph parser, whose core function is to accurately extract the spatial and semantic relationships between objects in an image and describe them in a structured form. In this way, the scene graph generator not only improves the ability to understand the image content, but also lays a solid foundation for subsequent inspection tasks.

[0055] Step 4: Implement the prompt method for checking the attribute prediction logic.

[0056] This paper proposes an innovative prompt method for attribute prediction logic inspection, the core goal of which is to improve the accuracy of attribute prediction. By introducing multi-level inspection and optimized prompt design, a reliable attribute logic prediction process is achieved. The method consists of the following two core sub-methods.

[0057] (4.1) Attribute key-value extraction based on object category

[0058] This is an object category-based attribute prediction mechanism that aims to efficiently extract key-value pairs of core attributes by identifying the object category in the image and combining the multimodal model's prior knowledge of specific categories. The module generates a preliminary possible attribute prediction set by intelligently matching the common attribute set of each category.

[0059] (4.2) Attribute checking combining visual cues and common sense knowledge

[0060] This is an innovative multimodal checking mechanism that aims to automatically verify and optimize the attribute values ​​predicted by the model by integrating the scene graph generated by visual cues and the common sense knowledge in the multimodal large language model to ensure the accuracy and logical consistency of the prediction results. The module dynamically checks whether the attribute values ​​are consistent with the visual scene information and common sense logic through intelligent comparison and reasoning capabilities, improving the overall performance and reliability of the model.

[0061] Check the predicted values ​​based on the scene graph and common sense knowledge, as shown below:

[0062] Object: bowl; Prediction attributes: ["round", "white"];

[0063] Check result: The attributes round and white are reasonable. According to the picture, the bowl is indeed round and white. No adjustment is needed.

[0064] Object: broccoli; Predicted attributes: ["green","steamed"];

[0065] Check result: The attribute green is correct because the broccoli in the picture is green; the attribute steamed cannot be fully confirmed based on vision, but it is reasonable based on common sense (broccoli is usually a steamed side dish) and the scene graph (broccoli has been put into the bowl). It is recommended to retain the attribute.

[0066] Object: rice; Prediction attributes: ["white", "cooked"];

[0067] Check result: The attribute white is correct because the rice in the picture is white. The attribute cooked makes sense based on common sense because rice is usually cooked and there is no sign of raw rice in the picture. The rice in the scene graph is placed together with the red bean stew and broccoli, indicating that it is part of a cooked dish. No adjustment is required.

[0068] Object: bean stew; Predicted attributes: ["red","saucy","vegetarian"];

[0069] Check results: The attribute red is correct because the bean stew in the picture has a clear red sauce. The attribute saucy is also correct because the stew looks like it has a lot of sauce. From the scene graph, we can see that bean stew and rice are adjacent, and the stew covers part of the rice, which shows that it has the characteristics of a liquid sauce, which meets the definition of saucy. The attribute vegetarian cannot be directly confirmed from the picture, although there seems to be no obvious meat in the stew, but this requires more context or description to verify. Combined with the scene graph, it shows that bean stew is served with broccoli and rice, which is usually consistent with vegetarian options.

[0070] Suggestion: Change vegetarian to ["likely vegetarian"] to indicate that this is a guess based on visuals and common sense rather than a certainty.

[0071] Step 5, deployment.

[0072] After combining the multimodal thought chain prompt method with the multimodal large language model, an attribute prediction system that can be applied to open vocabulary tasks is obtained. The system can dynamically adjust the input according to the needs of the task, avoiding the cumbersome hard coding process of attribute text, and the whole thought chain can give full play to the potential of the multimodal large language model in attribute prediction tasks to meet the needs of different business scenarios.

[0073] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal large language model attribute prediction method based on multimodal thinking chain, characterized in that: The following steps are involved: (1) Obtain a large multimodal language model and its pre-trained weights; (2) Construct a mask generator and a scene graph parser, which together with the multimodal large language model form a multimodal attribute prediction framework; the mask generator is used to segment objects in the image to generate independent masks, and the scene graph parser is used to extract the spatial and semantic relationships between objects in the image and generate sufficient context information; (3) Design a hierarchical reasoning chain method, input the attribute prediction task into the multimodal attribute prediction framework for processing, extract the hierarchy of the attribute table, and generate hierarchical and formatted reasoning prompts; through a dynamic task decomposition strategy, decompose the attribute prediction task into hierarchical subtasks, and select prompt methods for the next step of prediction of the multimodal large language model; supervise the model prediction process through critical thinking; (4) Using a logical check prompt method, first identify the object category in the image, and extract the corresponding attribute value based on the common sense knowledge in the model; then check the predicted attributes based on the scene graph generated by the scene graph parser and the common sense knowledge in the model.

2. The multimodal large language model attribute prediction method based on multimodal thinking chain according to claim 1 is characterized in that: In step (2), the mask generator generates a high-precision mask corresponding to the object area by analyzing the input image, shielding background interference and ensuring that the multimodal large language model focuses on the target area, thereby improving the accuracy of attribute prediction.

3. The multimodal large language model attribute prediction method based on multimodal thinking chain according to claim 1 is characterized in that: In step (2), the scene graph parser combines classification capabilities, image description capabilities and knowledge graph technology to parse the scene relationships in the input image, generate a graph structure that describes the semantic relationships between objects, and generate sufficient contextual information to provide to the multimodal large language model for reasoning, thereby enhancing its ability to understand contextual information.

4. The multimodal large language model attribute prediction method based on multimodal thinking chain according to claim 1 is characterized in that: In step (3), the input attribute prediction task first constructs a hierarchical tree structure of the attribute table and generates exclusive reasoning hints for each level.

5. The multimodal large language model attribute prediction method based on multimodal thinking chain according to claim 1 is characterized in that: In step (3), a prompt method is selected for the next prediction of the multimodal large language model, specifically: The subsequent reasoning path is adjusted according to the output of each stage of the multimodal large language model to improve prediction efficiency and result reliability.

6. The multimodal large language model attribute prediction method based on multimodal thought chain according to claim 1 is characterized in that: In step (3), the model prediction process is supervised by critical thinking, specifically: In the reasoning process of the hierarchical thinking chain, the multimodal large language model uses the critical thinking module to verify and feedback the prediction results at each step, forming a closed-loop reasoning to ensure the accuracy and consistency of the final response.

7. The multimodal large language model attribute prediction method based on multimodal thought chain according to claim 6 is characterized in that: The critical thinking module uses the logical reasoning ability of the model to verify the output results of each subtask in the task layer by layer. If an anomaly is found, it will automatically backtrack to the previous step and regenerate a more logical result.

8. The multimodal large language model attribute prediction method based on multimodal thinking chain according to claim 1 is characterized in that: The specific process of step (4) is as follows: The attribute prediction mechanism based on object categories can efficiently extract key-value pairs of its core attributes by identifying the object categories in the image and combining them with the common sense knowledge database of the multimodal large language model; intelligently match the common attribute sets of each category to generate a preliminary attribute prediction set; Combined with the scene graph information generated by the scene graph parser, the model cross-compares visual content with common sense knowledge to detect inconsistencies in attribute predictions and errors that violate common sense, and automatically generates explanatory feedback.

Images