Multi-modal large model scene understanding method based on scene graph enhancement

Abstract

The invention relates to a multi-modal large model scene understanding method based on scene graph enhancement, and belongs to the field of cross-modal information fusion and generation. The method comprises the following steps: firstly, preprocessing an input image, transmitting the preprocessed image into an open vocabulary target detection model to obtain entity information in the image, and generating an image with an entity bounding box and an entity category; secondly, transmitting the image with the bounding box and the entity category into a scene graph generation model, generating a preliminary entity relationship, and transmitting the preliminary entity relationship into a logic large model for correction to obtain a corrected entity relationship; thirdly, constructing a scene graph according to the entity in the image and the corrected entity relationship, transmitting the scene graph into a prompt large model, and generating a prompt text based on the scene graph; and finally, inputting the prompt text and the preprocessed image into a multi-modal large model, understanding image information by the multi-modal large model according to the prompt text, and generating description of a scene in the image. According to the method, the scene understanding capability of the multi-modal large model image can be effectively enhanced.

Description

Technical Field

[0001] The present invention relates to a multimodal large model scene understanding method based on scene graph enhancement, which belongs to the field of cross-modal information fusion and generation. Background Art

[0002] Scene understanding is a core research direction in the field of computer vision, which involves extracting deep semantic information from images and videos to achieve comprehensive understanding of the scene. This technical background is derived from the progress of deep learning, especially the development of convolutional neural networks (CNN) and generative adversarial networks (GAN), which enables machines to jump from simple image recognition to complex scene analysis, including tasks such as target recognition, relationship understanding, and scene description. With the development of technology, scene understanding has shown broad application potential in many fields such as autonomous driving, augmented reality, and robot navigation, becoming one of the key technologies to promote the transformation of artificial intelligence from perception to cognition.

[0003] Multimodal large models greatly enhance the scene understanding ability of machines by integrating and processing data from different modalities (such as images, text, audio, etc.). These models can not only recognize and understand objects and scenes in images, but also combine text and voice information to provide a deeper level of semantic understanding. This cross-modal interaction makes machines more intelligent and humane, enabling them to understand and interact with the world more comprehensively. The application of multimodal large models in scene understanding includes but is not limited to autonomous driving, medical diagnosis, video understanding and processing, etc. Therefore, it is crucial to strengthen the cross-modal generation and understanding capabilities of multimodal large models. However, the existing multimodal large models are affected by visual uncertainty, and when the model encounters visual uncertainty, the model will over-rely on pre-trained text information, and then make judgments based on pre-trained text rather than actual conditions, resulting in an inability to accurately understand image information. In addition, multimodal large models are also affected by pre-training datasets. Most visual-language pre-training datasets are mainly constructed based on MSCOCO, which itself has problems of unbalanced object distribution and biased object associations. Therefore, when processing image information, existing large multimodal models may misinterpret image scenes and have object hallucination problems. In this context, generating scene graphs can effectively enhance the multimodal model's understanding of scene information in images and improve the multimodal model's cross-modal generation capabilities. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal large model scene understanding method based on scene graph enhancement, which aims to address the problems faced by existing multimodal large models, including the inability to accurately and correctly understand the scene information in the image, object hallucinations and other problems, thereby enhancing the cross-modal generation and understanding capabilities of the multimodal large model. By introducing an open vocabulary target detection model and a large language model, it is possible to effectively solve the problems of missing entity information and inaccurate relationship information when the scene graph generation model constructs the scene graph, and construct a more accurate and reliable scene graph. The prompt text generated by the large language model according to the scene graph can effectively assist the multimodal large model, enhance its ability to understand the image scene, and generate more accurate description text.

[0005] The technical solution adopted by the present invention is: a multimodal large model scene understanding method based on scene graph enhancement, the specific steps are:

[0006] Step 1: Scale the image so that the image size meets the input requirements of the open vocabulary object detection model and the multimodal large model, and obtain a preprocessed image that meets the model input standards;

[0007] Step 2: Pass the preprocessed image into the open vocabulary object detection model to obtain the entity information in the image and generate an image with entity bounding boxes and entity categories;

[0008] Step 3: Pass the image with entity bounding box and entity category into the convolutional neural network to generate preliminary entity relationship;

[0009] Step 4: The preliminary entity relationship is transferred to the logical big model for modification to obtain the modified entity relationship;

[0010] Step 5: Construct a scene graph based on the entity information in the image and the corrected entity relationship;

[0011] Step 6: Pass the constructed scene graph into the prompt model to generate prompt text based on the scene graph;

[0012] Step 7: Input the generated prompt text and the preprocessed image into the multimodal large model. The multimodal large model understands the image information based on the prompt text and generates a description of the scene in the image.

[0013] The specific steps of Step 1 are:

[0014] Step 1: Scale the acquired image data so that its image size meets the input requirements of open vocabulary object detection models such as YOLO-World and multimodal large models to obtain the preprocessed image.

[0015] The specific steps of Step 3 are:

[0016] Step 3.1: Use the image annotated with entity bounding boxes and entity categories as input to the convolutional neural network (CNN);

[0017] Step 3.2: The convolutional neural network reads the image, convolves the image through the convolution kernel, obtains the image entity category code, and extracts visual features;

[0018] Step 3.3: The extracted visual features are passed into the conditional random field (CFR). The conditional random field enhances the entity context information through the spatial relationship and color features between pixels;

[0019] Step 3.4: Use the enhanced entity context information to predict the relationship between entities and obtain the preliminary generated entity relationship.

[0020] The specific steps of Step 4 are:

[0021] Step 4.1: Input the initially generated entity relationship into the logical big model, which is used to determine whether the entity relationship logic is reasonable, such as qwen2.5, GLM-4-long, etc.;

[0022] Step 4.2: The logical large model reads entity relationships, identifies semantic information, finds illogical entity relationships, and modifies them to obtain corrected entity relationships.

[0023] The specific steps of Step 5 are:

[0024] Step 5.1: Merge the entity information with the corrected entity relationship to obtain a complete entity relationship triple;

[0025] Step 5.2: Use the entity relationship triples as input to the graph database Neo4j;

[0026] Step 5.3: Build a scene graph based on the input.

[0027] The specific steps of Step 6 are:

[0028] Step 6.1: The scene graph and the initial prompt text are used as inputs to generate a large prompt model, wherein the large prompt model is used to generate a large language model of prompt information based on the scene graph, such as llama3.2, mixtral, etc.;

[0029] Step 6.2: Generate a prompt large model to read the structured information in the scene graph;

[0030] Step 6.3: Generate a large model to identify field information in the scene graph and extract entity relationships from it;

[0031] Step 6.4: Generate a large prompt model to expand the entity relationship based on the prompt text and convert the structured information into natural language expression;

[0032] Step 6.5: Generate a large model to check the natural language expression. By understanding the semantic information, identify the logically unreasonable or semantically incoherent parts and modify them to obtain a natural language expression that is fluent, logically coherent, and semantically clear.

[0033] Step 6.6: Generate a large model to review the natural language expression, extract the entity relationship in the natural language expression, and then compare it with the scene graph information to check whether the natural language expression accurately reflects the information in the scene graph;

[0034] Step 6.7: Generate prompts. The large model outputs natural language expressions and generates prompt text based on the scene graph.

[0035] The specific steps of Step 7 are:

[0036] Step 7.1: Use the generated prompt text and preprocessed image as input of the multimodal large model;

[0037] Step 7.2: The multimodal large model reads the preprocessed image, understands the image content based on the prompt text, and finally generates an image description and outputs it as a result.

[0038] The beneficial effects of the present invention are: by constructing a scene graph for an input image, the multimodal large model's ability to understand the image scene is enhanced, which can effectively improve the cross-modal retrieval, understanding and generation capabilities of the multimodal large model; by collaboratively constructing a scene graph with an open vocabulary target detection model and a large model, the comprehensiveness and accuracy of entity and relationship information are effectively improved; by using a large model to generate prompt text according to the scene graph, the accuracy and factuality of the prompt text are effectively enhanced, and the multimodal understanding capability of the multimodal large model can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall flow chart of the present invention;

[0040] Figure 2 It is a schematic diagram of generating the relationship between entities and preliminary entities of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0042] Example 1: Figure 1 As shown, the specific steps of a multimodal large model scene understanding method based on scene graph enhancement are as follows:

[0043] Step 1: Get image information. The image information comes from a picture taken in the actual scene or a frame captured from the video stream. After getting the image information, scale the image to ensure that the image size is 640*640 pixels. Thus, the preprocessed image is obtained, such as Figure 2 shown.

[0044] Step 2: The preprocessed image is passed to an open vocabulary object detection model such as YOLO-World. The open vocabulary object detection model obtains the visual features of the image through the Darknet image encoder. After obtaining the visual features, the open vocabulary object detection model performs object detection on the image. Finally, the open vocabulary object detection model outputs an image with annotated bounding boxes and categories and object detection results, and uses the object detection results as entity information.

[0045] Step 3: Pass the image annotated with the bounding box and category into the convolutional neural network. The convolutional neural network reads the image, convolves the image with the convolution kernel, obtains the image entity category encoding, extracts visual features, and passes the extracted visual features into the conditional random field. The conditional random field enhances the entity context information through the spatial relationship and color features between pixels, and uses the enhanced entity context information to predict the relationship between entities to obtain the preliminary generated entity relationship.

[0046] The process of Step 2 and Step 3 is as follows Figure 2 Shown

[0047] Step 4: Input the text: "You are an expert in the field of logic. Please solve the following problem: I will provide some entity relationship information in the form of triples: {entity relationship information}. Please read these triples and determine whether the relationship between entities is reasonable. If the relationship is unreasonable, correct the unreasonable relationship information based on the corpus you were exposed to during pre-training. The following are examples of incorrect entity relationships: {[bench, sit, person], [table, on..., cup], [biscuit, hold, person]}. The correction results of the above incorrect relationships are as follows: {[bench, sat by..., person], [table, supported, cup], [biscuit, held by..., person]}" as prompt text and preliminary entity relationship information into the logic model. The logic model corrects the unreasonable relationship information according to the prompt text and outputs the corrected entity relationship information.

[0048] Step 5: After merging the entity and the corrected entity relationship, they are transferred to the graph database Neo4j, where the entity relationship is stored in a graph structure to construct a scene graph.

[0049] Step 6: Use the text "You are an expert in natural language processing. I have input a scene graph to you. You need to read the entity relationship information in the scene graph and expand the entity relationship information to form a complete text information. The relevant examples are as follows: {[person, sitting, bench], [bench, close to, table], [cup, on, table], [person, holding, biscuit]}. The above is the entity relationship information. The text obtained after expansion is as follows: A person is sitting on a bench close to the table, holding a biscuit in his hand, and there is a cup on the table." as a prompt information and the scene constructed in the previous step Figure 1 And pass it to the generated prompt model. After reading the scene graph, the generated prompt model extracts the entity relationship information in the scene graph, expands the entity relationship according to the prompt information, and generates a complete text. Then the generated prompt model preliminarily checks the text information to ensure that the text information is logically reasonable, the language is fluent, and the semantics are clear. After the preliminary check is completed, the generated prompt model reviews the text information, extracts the entity relationship in the text information, and then compares it with the information in the scene graph to ensure that the text information accurately reflects the content contained in the scene graph.

[0050] Step 7: Input the preprocessed image and the prompt text into the multimodal model. The multimodal model aligns the image content with the prompt text to achieve cross-modal understanding, and finally generates a description text about the image and outputs the description text as a result.

Claims

1. A multimodal large model scene understanding method based on scene graph enhancement, characterized by: Step 1: Scale the image so that the image size meets the input requirements of the open vocabulary object detection model and the multimodal large model, and obtain a preprocessed image that meets the model input standards; Step 2: Pass the preprocessed image into the open vocabulary object detection model to obtain the entity information in the image and generate an image with entity bounding boxes and entity categories; Step 3: Pass the image with entity bounding box and entity category into the convolutional neural network to generate preliminary entity relationship; Step 4: The preliminary entity relationship is transferred to the logical big model for modification to obtain the modified entity relationship; Step 5: Construct a scene graph based on the entity information in the image and the corrected entity relationship; Step 6: Pass the constructed scene graph into the prompt model to generate prompt text based on the scene graph; Step 7: Input the generated prompt text and the preprocessed image into the multimodal large model. The multimodal large model understands the image information based on the prompt text and generates a description of the scene in the image.

2. According to claim 1, a multimodal large model scene understanding method based on scene graph enhancement is characterized in that: The Step 3 is specifically as follows: Step 3.1: Use the image annotated with the entity bounding box and entity category as the input of the convolutional neural network; Step 3.2: The convolutional neural network reads the image, convolves the image through the convolution kernel, obtains the image entity category code, and extracts visual features; Step 3.3: The extracted visual features are passed into the conditional random field. The conditional random field enhances the entity context information through the spatial relationship and color features between pixels; Step 3.4: Use the enhanced entity context information to predict the relationship between entities and obtain the preliminary generated entity relationship.

3. The multimodal large model scene understanding method based on scene graph enhancement according to claim 1, characterized in that: The Step 4 is specifically as follows: Step 4.1: Input the initially generated entity relationships into the logical model; Step 4.2: The logical large model reads entity relationships, identifies semantic information, finds illogical entity relationships, and modifies them to obtain corrected entity relationships.

4. The multimodal large model scene understanding method based on scene graph enhancement according to claim 1, characterized in that: The Step 5 is specifically as follows: Step 5.1: Merge the entity information with the corrected entity relationship to obtain a complete entity relationship triple; Step 5.2: Use the entity relationship triples as input to the graph database Neo4j; Step 5.3: Build a scene graph based on the input.

5. The multimodal large model scene understanding method based on scene graph enhancement according to claim 1, characterized in that: The Step 6 is specifically as follows: Step 6.1: Use the scene graph and the initial prompt text as input to generate the prompt model; Step 6.2: Generate a prompt large model to read the structured information in the scene graph; Step 6.3: Generate a large model to identify field information in the scene graph and extract entity relationships from it; Step 6.4: Generate a large prompt model to expand the entity relationship based on the prompt text and convert the structured information into natural language expression; Step 6.5: Generate a large model to check the natural language expression. By understanding the semantic information, identify the logically unreasonable or semantically incoherent parts and modify them to obtain a natural language expression that is fluent, logically coherent, and semantically clear. Step 6.6: Generate a large model to review the natural language expression, extract the entity relationship in the natural language expression, and then compare it with the scene graph information to check whether the natural language expression accurately reflects the information in the scene graph; Step 6.7: Generate prompts. The large model outputs natural language expressions and generates prompt text based on the scene graph.

6. The multimodal large model scene understanding method based on scene graph enhancement according to claim 1, characterized in that: The Step 7 is specifically as follows: Step 7.1: Use the generated prompt text and preprocessed image as input of the multimodal large model; Step 7.2: The multimodal large model reads the preprocessed image, understands the image content based on the prompt text, and finally generates an image description and outputs it as a result.

Images