A multi-modal large model scene understanding method based on scene graph enhancement

Abstract

The present application relates to a kind of based on scene graph enhanced multimodal large model scene understanding method, belong to cross-modal information fusion and generation field.First, the input image is preprocessed, and the preprocessed image is transmitted into open vocabulary target detection model, obtains entity information in image, generates image with entity bounding box and entity category;Second, the image with bounding box and entity category is transmitted into scene graph generation model, generates preliminary entity relationship and is transmitted into logic large model to correct, obtains the entity relationship of correction;Then, according to the entity in image and the entity relationship of correction, construct scene graph and are transmitted into prompt large model, generate prompt text based on scene graph;Finally, prompt text and preprocessed image are input into multimodal large model, and multimodal large model understands image information according to prompt text, generates the description of image scene.The present application can effectively enhance the image scene understanding ability of multimodal large model.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-modal large model scene understanding method based on scene graph enhancement, belonging to the field of cross-modal information fusion and generation. BACKGROUND

[0002] Scene understanding is a core research direction in the field of computer vision, which involves extracting deep semantic information from images and videos to achieve comprehensive cognition of the scene. This technology background is born from the progress of deep learning, especially the development of convolutional neural networks (CNN) and generative adversarial networks (GAN), which enables machines to leap from simple image recognition to complex scene analysis, including object recognition, relationship understanding, and scene description tasks. With the development of technology, scene understanding has shown broad application potential in automatic driving, augmented reality, robot navigation, and other fields, becoming one of the key technologies to promote the transition of artificial intelligence from perception to cognition.

[0003] Multi-modal large models greatly enhance the machine's scene understanding ability by integrating and processing data from different modalities such as images, text, audio, etc. These models not only recognize and understand objects and scenes in images, but also combine text and speech information to provide deeper semantic understanding. This cross-modal interaction makes machines more intelligent and humanized, enabling them to more comprehensively understand and interact with the world. Applications of multi-modal large models in scene understanding include but are not limited to automatic driving, medical diagnosis, video understanding and processing, etc. Therefore, it is crucial to strengthen the cross-modal generation and understanding ability of multi-modal large models. However, existing multi-modal large models are affected by visual uncertainty, and when the model encounters visual uncertainty, the model will excessively rely on pre-trained text information, and then make judgments based on pre-trained text rather than actual conditions, resulting in inaccurate understanding of image information. In addition, multi-modal large models are also affected by pre-training data sets. Most visual-linguistic pre-training data sets are constructed based on MSCOCO, which itself has the problem of object distribution imbalance and object association bias. Therefore, existing multi-modal large models may misinterpret image scenes and have object hallucination problems when processing image information. In this context, generating a scene graph can effectively enhance the understanding of image scene information by multi-modal large models, and improve the cross-modal generation ability of multi-modal large models. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-modal large model scene understanding method based on scene graph enhancement, aiming to solve the problems faced by existing multi-modal large models, including the inability to accurately and correctly understand the scene information in the image, object hallucination and other problems, thereby enhancing the cross-modal generation and understanding ability of the multi-modal large model.

[0005] The technical solution adopted by the present application is: a multi-modal large model scene understanding method based on scene graph enhancement, the specific steps of which are:

[0006] Step 1: scale the image to make the image size meet the input requirements of the open vocabulary object detection model and the multi-modal large model, and obtain a preprocessed image that meets the model input standard;

[0007] Step 2: input the preprocessed image into the open vocabulary object detection model to obtain entity information in the image and generate an image with entity bounding boxes and entity categories;

[0008] Step 3: input the image with entity bounding boxes and entity categories into a convolutional neural network to generate preliminary entity relationships;

[0009] Step 4: input the preliminary entity relationships into a logical large model for correction to obtain corrected entity relationships;

[0010] Step 5: construct a scene graph according to the entity information in the image and the corrected entity relationships;

[0011] Step 6: input the constructed scene graph into a prompt large model to generate a prompt text based on the scene graph;

[0012] Step 7: input the generated prompt text and the preprocessed image into a multi-modal large model, and the multi-modal large model understands the image information according to the prompt text to generate a description of the scene in the image.

[0013] The specific steps of Step 1 are:

[0014] Step 1: scale the obtained image data to make the image size meet the input requirements of the open vocabulary object detection model such as YOLO-World and the multi-modal large model, and obtain a preprocessed image.

[0015] The specific steps of Step 3 are:

[0016] Step3.1: input the image labeled with entity bounding box and entity category into a convolutional neural network (CNN);

[0017] Step3.2: the convolutional neural network reads the image, convolves the image through a convolution kernel, obtains an image entity category code, and extracts visual features;

[0018] Step3.3: the extracted visual features are input into a conditional random field (CFR), which enhances entity context information through spatial relationships and color features between pixels;

[0019] Step3.4: use the enhanced entity context information to predict the relationship between entities to obtain a preliminary generated entity relationship.

[0020] The Step4 specific steps are:

[0021] Step4.1: input the preliminary generated entity relationship into a large logic model, which is used to determine whether the entity relationship logic is reasonable, such as qwen2.5, GLM-4-long, etc.

[0022] Step4.2: the large logic model reads the entity relationship, identifies semantic information, finds unreasonable entity relationships, and modifies them to obtain corrected entity relationships

[0023] The Step5 specific steps are:

[0024] Step5.1: fuse the entity information and the corrected entity relationship to obtain a complete entity relationship triple;

[0025] Step5.2: input the entity relationship triple into the graph database Neo4j;

[0026] Step5.3: construct a scene graph according to the input.

[0027] The Step6 specific steps are:

[0028] Step6.1: input the scene graph and initial prompt text into a generation prompt large model, which is used to generate a large language model based on the prompt information of the scene graph, such as llama3.2, mixtral, etc.

[0029] Step6.2: the generation prompt large model reads the structured information in the scene graph;

[0030] Step6.3: the generation prompt large model identifies the field information in the scene graph and extracts the entity relationship therefrom;

[0031] Step 6.4: The prompt large model generates a prompt text based on the prompt text to expand the entity relationship, and converts the structured information into a natural language expression;

[0032] Step 6.5: The prompt large model generates a prompt text to check the natural language expression, identifies logically unreasonable or semantically incoherent parts through semantic information understanding, and modifies to obtain a natural language expression that is fluent, logically coherent and semantically clear;

[0033] Step 6.6: The prompt large model generates a prompt text to review the natural language expression, extracts the entity relationship in the natural language expression, and then compares it with the scene graph information to check whether the natural language expression accurately reflects the information in the scene graph;

[0034] Step 6.7: The prompt large model outputs a natural language expression based on the scene graph.

[0035] The Step 7 specific steps are:

[0036] Step 7.1: The generated prompt text and the preprocessed image are input into the multi-modal large model;

[0037] Step 7.2: The multi-modal large model reads the preprocessed image and understands the image content according to the prompt text, finally generates an image description and outputs it as a result.

[0038] The beneficial effects of the present application are: the multi-modal large model can effectively improve the cross-modal retrieval, understanding and generation capabilities of the multi-modal large model by constructing a scene graph for the input image to enhance the understanding ability of the multi-modal large model for the image scene; the openness of the vocabulary target detection model and the large model is effectively improved by cooperatively constructing the scene graph, which effectively improves the comprehensiveness and accuracy of entity and relationship information; the accuracy and factuality of the prompt text are effectively enhanced by the large model generating the prompt text based on the scene graph, which can effectively improve the multi-modal understanding ability of the multi-modal large model. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is the overall flowchart of the present application;

[0040] Figure 2 is the generation schematic diagram of the entity and the preliminary entity relationship of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0042] Embodiment 1: as shown in the following: Figure 1 The specific steps of a multi-modal large model scene understanding method based on scene graph enhancement are as follows:

[0043] Step1: Obtain image information from a picture or a frame extracted from a video stream of an actual scene. After obtaining the image information, the image is scaled to ensure that the image size is 640*640 pixels. The pre-processed image is obtained as shown in Figure 2 .

[0044] Step2: The pre-processed image is input into an open vocabulary object detection model such as YOLO-World. The open vocabulary object detection model obtains the visual features of the image through the Darknet image encoder. After obtaining the visual features, the open vocabulary object detection model performs object detection on the image. Finally, the open vocabulary object detection model outputs the image labeled with bounding boxes and categories, as well as the object detection results, and the object detection results are used as entity information.

[0045] Step3: The image labeled with bounding boxes and categories is input into a convolutional neural network. The convolutional neural network reads the image, convolves the image through a convolution kernel, obtains image entity category encoding, extracts visual features, and inputs the extracted visual features into a conditional random field. The conditional random field enhances the entity context information through the spatial relationship and color features between pixels, predicts the relationship between entities using the enhanced entity context information, and obtains the preliminary generated entity relationship.

[0046] The processes of Step2 and Step3 are shown in Figure 2 .

[0047] Step4: The text: "You are a logic expert in the field of logic, please solve the following problem: I will provide some entity relationship information in the form of triples: {entity relationship information}, please read these triples and judge whether the relationship between entities is reasonable, if the relationship is not reasonable, then modify the unreasonable relationship information according to the corpus you encountered during pre-training. The following is an example of incorrect entity relationship: {

bench, sit, person

table, on, cup

cookie, hold, person

bench, be sat on, person

table, support, cup

cookie, be held, person

[0048] Step5: The entity and the modified entity relationship are fused and input into the graph database Neo4j to store the entity relationship in the form of a graph structure and construct a scene graph.

[0049] Step 6: Use the text "You are an expert in the field of natural language processing. I have input a scene graph to you. You need to read the entity relationship information in the scene graph and expand the entity relationship information to form a complete text information. The relevant examples are as follows: {[person, sitting, bench], [bench, close to, table], [cup, on..., table], [person, holding, biscuit]}. The above is the entity relationship information. The text obtained after expansion is as follows: A person is sitting on a bench close to the table, holding a biscuit in his hand, and there is a cup on the table." as a prompt information and the scene constructed in the previous step Figure 1 The generated prompt model is then passed to the generated prompt model. After reading the scene graph, the generated prompt model extracts the entity relationship information in the scene graph and expands the entity relationships based on the prompt information to generate a complete text. The generated prompt model then conducts a preliminary check of the text information to ensure that the text is logically sound, the language is fluent, and the semantics are clear. After this preliminary check, the generated prompt model reviews the text information, extracts the entity relationships in the text information, and then compares them with the information in the scene graph to ensure that the text information accurately reflects the content contained in the scene graph.

[0050] Step 7: Input the pre-processed image and the prompt text into the multimodal model. The multimodal model aligns the image content with the prompt text to achieve cross-modal understanding, ultimately generating a description of the image and outputting the description text as the result.

Claims

1. A multi-modal large model scene understanding method based on scene graph enhancement, characterized by: Step1: Scale the image to make the image size meet the input requirements of the open vocabulary object detection model and the multi-modal large model, and obtain a preprocessed image that meets the model input standard; Step2: input the preprocessed image into the open vocabulary object detection model to obtain entity information in the image and generate an image with entity bounding boxes and entity categories; Step3: input the image with entity bounding boxes and entity categories into a convolutional neural network to generate preliminary entity relationships; Step4: input the preliminary entity relationships into a logical large model for correction to obtain corrected entity relationships; Step5: construct a scene graph based on the entity information in the image and the corrected entity relationships; Step6: input the constructed scene graph into a prompt large model to generate a prompt text based on the scene graph; Step7: input the generated prompt text and the preprocessed image into a multi-modal large model, which understands the image information based on the prompt text and generates a description of the scene in the image; Step6 is specifically: Step6.1: input the scene graph and the initial prompt text into the prompt generation model; Step6.2: the prompt generation model reads the structured information in the scene graph; Step6.3: the prompt generation model identifies the field information in the scene graph and extracts the entity relationships therefrom; Step6.4: the prompt generation model expands the entity relationships based on the prompt text and converts the structured information into natural language expressions; Step6.5: the prompt generation model checks the natural language expressions, identifies logically unreasonable or semantically incoherent parts by understanding semantic information, and modifies them to obtain natural language expressions that are fluent, logically coherent, and semantically clear; Step6.6: the prompt generation model reviews the natural language expressions, extracts the entity relationships therefrom, and then compares them with the scene graph information to check whether the natural language expressions accurately reflect the information in the scene graph; Step6.7: the prompt generation model outputs the natural language expressions and generates a prompt text based on the scene graph.

2. The multi-modal large model scene understanding method based on scene graph enhancement according to claim 1, characterized in that, Step3 is specifically: Step3.1: input the image labeled with entity bounding boxes and entity categories into the convolutional neural network; Step3.2: the convolutional neural network reads the image, convolves the image through convolution kernels, obtains image entity category codes, and extracts visual features; Step3.3: input the extracted visual features into a conditional random field, which enhances entity context information through spatial relationships and color features between pixels; Step3.4: use the enhanced entity context information to predict the relationship between entities to obtain the preliminary generated entity relationships.

3. The multi-modal large model scene understanding method based on enhanced scene graph according to claim 1, characterized in that, Step4 is specifically: Step4.1: input the preliminary generated entity relationships into the logical large model; Step4.2: the logical large model reads the entity relationships, identifies semantic information, finds logically inconsistent entity relationships, and modifies them to obtain corrected entity relationships.

4. The multi-modal large model scene understanding method based on enhanced scene graph according to claim 1, characterized in that, Step5 is specifically: Step5.1: Fuse the entity information with the corrected entity relationship to obtain complete entity relationship triples; Step5.2: Take the entity relationship triples as the input of the graph database Neo4j; Step5.3: Construct a scene graph according to the input.

5. The multi-modal large model scene understanding method based on enhanced scene graph according to claim 1, characterized in that, The Step7 is specifically: Step7.1: Take the generated prompt text and the preprocessed image as the input of the multi-modal large model; Step7.2: The multi-modal large model reads the preprocessed image and understands the image content according to the prompt text, finally generates an image description and outputs it as a result.

Images