A mid-high point image scene understanding method and system based on scene interpretation map

Abstract

The application relates to the technical field of graphic data processing, and particularly discloses a medium-high point image scene understanding method and system based on a scene interpretation graph, wherein the method comprises the following steps: inputting a text prompt word and a scene image into an open set target detection model to obtain a target detection frame and a target label required for scene understanding detection; inputting the target detection frame and the scene image into a segmentation model to obtain a mask of the target detection frame; calculating the relative position and size relationship of the target through the label and the mask to obtain a scene interpretation graph; and analyzing the scene interpretation graph through a large language model to output a text result for image scene understanding. The method improves the generalization capability of scene understanding, saves data training cost, improves description accuracy and detail, and improves the interpretability of the understanding result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and particularly relates to a medium-high point image scene understanding method and system based on scene interpretation images. BACKGROUND

[0002] With the development of information technology, especially the progress of computer vision, big data, artificial intelligence and internet technology, the scene monitoring and scene understanding technology of medium-high point have also been significantly developed. The data collected by medium-high point cameras and various sensors grow extremely fast, and a large amount of medium-high point characteristic data with emergency, forest, land, water conservancy, environmental protection, agriculture and other industry attributes are reserved, which provides data support for training scene understanding of medium-high point image acquisition.

[0003] The traditional medium-high point image scene understanding method divides the scene into forest, farmland, factory and other categories by using the labeled scene data, obtains a scene classification model by using the neural network training model method, classifies the scene of the image in the actual application by using the model, and finally obtains the category result of the scene. Then, the scene understanding method based on a large amount of training data target detection model uses a large amount of image annotation data to train a target detection model, and directly uses the detection result as the result of scene image understanding. Then, the scene understanding method based on a large amount of text image training data uses a large amount of image and text annotation information to train an image understanding model, directly inputs the image into the model in the process of scene understanding, and outputs the text explanation of the scene, which is taken as the result of scene understanding. Finally, the scene understanding method based on self-supervised learning uses a large amount of unlabeled data to learn the image features of the image understanding model by using the self-supervised learning technology without manual annotation. The traditional medium-high point image scene understanding method needs to collect a large amount of scene data, annotate the data, and train the model by using the annotated data. In the inference process, the fixed label detection target frame information is obtained by inputting the image. A large amount of data is also needed to train a visual large model, and the classification result of the image and the text understanding content of the image are directly obtained by inputting the image in the inference process.

[0004] Thus, a large amount of training data is required, and when the model training is completed, the application boundary of the model is determined, and when the application scenario changes, the model capability will be weakened, and more training data needs to be introduced for training. In the scene understanding method based on a large model, a large amount of data and huge training resources are required to train a large model, so the research and development cost of using this method is high, and the iteration period is long. When using a visual large model for scene understanding, the model directly gives a text result, and the intermediate reasons and judgment basis cannot be obtained, and when an unreasonable result is generated, the reason cannot be found, so using a visual large model for scene understanding has the disadvantages of weak explainability and untraceability. In traditional scene interpretation diagram analysis, a square frame of a detection result is used, and since the shape of a target is irregular, the details of the target are ignored in the process of using the square frame to represent the target analysis, the relationship between objects is calculated incorrectly, and the final scene understanding result is affected. SUMMARY

[0005] The purpose of the present application is to provide a middle and high point image scene understanding method based on a scene interpretation diagram, improve the generalization ability of scene understanding, save data training cost, improve description accuracy and detail, and improve the explainability of understanding results.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A middle and high point image scene understanding method based on a scene interpretation diagram, comprising the following steps:

[0008] Inputting a text prompt word and a scene image into an open set target detection model to obtain a target detection frame and a target label required for scene understanding detection;

[0009] Inputting the target detection frame and the scene image into a segmentation model to obtain a mask of the target detection frame;

[0010] Calculating the relative position and size relationship of the target through the label and the mask to obtain a scene interpretation diagram;

[0011] Analyzing the scene interpretation diagram through a large language model to output a text result for image scene understanding.

[0012] In a further scheme, the method of inputting a text prompt word and a scene image into an open set target detection model comprises:

[0013] Obtaining a scene image;

[0014] Presetting a text description required for analyzing a scene for the scene image;

[0015] Inputting the text description into a large language model to generate the text prompt word through the large language model.

[0016]

[0016] In a further embodiment, the method for calculating the relative positions and sizes of targets using labels and masks to obtain a scene interpretation map includes:

[0017] Obtain the center position, boundary contour shape, boundary length, and boundary pixel area of ​​the target detection box mask;

[0018] The relative position and size relationship between the target and other objects in the scene image are calculated by using the center position, boundary contour shape, boundary length, and boundary pixel area.

[0019] A scene interpretation map is created by using the relative positions and sizes of the target and other objects in the scene image, along with their labels, to represent the relationships between objects.

[0020] In a further embodiment, the method for analyzing the scene interpretation graph using a large language model and outputting textual results for image scene understanding includes:

[0021] The positional relationships, inclusion relationships, relative area relationships, and target labels between the target and other objects in the scene image are input into a large language model, and the text results of the image scene understanding are output through a large speech model.

[0022] In a further embodiment, the method also includes:

[0023] Input the image scene understanding related questions into a large language model;

[0024] The large language model continues to reason based on the text results of the image scene understanding output, obtains the understanding results of related questions, and outputs feedback to the user.

[0025] In a further embodiment, the relative positional relationship includes the relative distance between the center positions, the relative distance between the boundary contours, and the intersection relationship of the boundary contours; the relative size relationship includes the relative area relationship and the boundary contour length relationship.

[0026] In a further embodiment, the text prompts are different key prompts proposed based on the application scenario image and the problem to be analyzed.

[0027] This invention also proposes a mid-to-high-angle image scene understanding system based on scene interpretation maps, comprising:

[0028] The target detection box and target label acquisition module is used to input text prompts and scene images into the open set target detection model to obtain the target detection boxes and target labels needed for scene understanding.

[0029] The object detection box mask acquisition module is used to input the object detection box and scene image into the segmentation model and obtain the mask of the object detection box;

[0030] The scene interpretation map acquisition module is used to calculate the relative positions and sizes of targets using labels and masks to obtain a scene interpretation map;

[0031] The image scene understanding output module is used to analyze the scene interpretation graph through a large language model and output text results for image scene understanding.

[0032] In a further embodiment, the open set target detection model is either the Grounding DINO model or the YOLO model.

[0033] In a further embodiment, the segmentation model is a Segment Anything Model series model.

[0034] The beneficial effects of this invention are:

[0035] This invention describes the requirements of a scenario and uses a large language model to automatically generate text prompts for detection. It then utilizes a general open-set object detection model to perform object detection. This method can automatically detect key objects based on different scenarios and requirements, exhibiting advantages such as strong adaptability and broad generalization.

[0036] By detecting target labels, a segmentation model is used to segment the targets. Fine-grained masks are then used to calculate the target's orientation, containment, and proximity relationships, ultimately obtaining a scene interpretation map. Compared to the rectangular bounding boxes used in existing methods, this method offers finer granularity, resulting in more accurate calculations and a more precise final scene understanding. This method utilizes target masks to calculate the scene interpretation map, offering advantages such as detailed description and high accuracy.

[0037] This invention transforms scene interpretation diagrams into textual information and uses a large language model to understand and summarize the scene content, ultimately generating answers and corresponding solutions to the requirements. This method utilizes detailed scene interpretation diagrams as input to the large language model, offering advantages such as strong interpretability and generalization. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a method for understanding mid-to-high point image scenes based on scene interpretation maps in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, a method for understanding mid-to-high-angle image scenes based on scene interpretation maps includes the following steps:

[0042] S1: Input text prompts and scene images into the open-set object detection model to obtain the object detection boxes and object labels needed for scene understanding. The open-set object detection model can be an open-set object detection model such as GroundingDINO or YOLO series models. The input text prompts can be the objects to be detected, such as farmland, forest, road and / or pedestrians. Input these text prompts into the open-set object detection model to obtain the object detection boxes, which are the detection boxes corresponding to the input text prompts. These detection boxes are the element objects needed for scene understanding, and the labels of these objects are obtained to distinguish the categories of the objects.

[0043] S2: Input the target detection box and scene image into the segmentation model to obtain the mask of the target detection box; the segmentation model can be a model such as the Segment Anything Model series. Use the segmentation model to segment the target detection boxes detected in step S1 above to obtain the mask corresponding to each detection box and obtain the outer contour of the target at the pixel level.

[0044] S3: Calculate the relative positions and sizes of targets using labels and masks to obtain a scene interpretation map;

[0045] S4: Analyze the scene interpretation graph using a large language model and output the text results for understanding the image scene.

[0046] According to the above embodiments, in some embodiments, step S1 further includes:

[0047] Acquire scene images; the method for acquiring scene images can be from historical data or by taking photos on the spot.

[0048] Preset text descriptions for the scene images to be analyzed; for example: I am now processing a fire scene, the preset text description could be: I now need to detect smoke in the image, which is a city scene, and I need to obtain the smoke targets as well as other related targets in the scene.

[0049] The text description is input into a large language model, which then generates text prompts. The large language model returns text prompts related to words such as fireworks, buildings, pedestrians, roads, chimneys, and factories.

[0050] In some embodiments, step S3 includes:

[0051] Obtain the center position, boundary contour shape, boundary length, and boundary pixel area of ​​the target detection box mask;

[0052] The relative position and size relationship between the target and other objects in the scene image are calculated by using the center position, boundary contour shape, boundary length, and boundary pixel area.

[0053] A scene interpretation map is built by analyzing the relative positions and sizes of targets and other objects in the scene image, along with their labels, to represent the relationships between objects. For example, the acquired scene interpretation map might show two targets, farmland and fireworks, that do not intersect but are far apart in the image, with the fireworks area being smaller and the farmland larger. Similarly, the relationships between each target and other targets can be obtained. The angular and positional relationships (up, down, left, right, and angle relationships) and containment relationships (intersection, containment, and separation relationships) between targets and other scene objects are calculated using the mask boundary contour. The distance relationship between targets and other scene objects is calculated using the mask center position, and the relative area size relationship between targets and other scene objects is calculated using the pixel area occupied by the mask boundary. Using the relationships between targets and other scene objects, along with detection labels, a high-precision scene interpretation map can be built to represent the relationships between objects. A detailed scene interpretation map can help large language models understand scene information more accurately.

[0054] In some embodiments, step S4 includes:

[0055] The positional relationships, containment relationships, relative area relationships, and target labels between the target and other objects in the scene image are input into a large language model, and the large speech model outputs the text results of the image scene understanding. In some embodiments, the method further includes step S5:

[0056] Input the image scene understanding related questions into a large language model;

[0057] The large language model continues to reason based on the text results of the image scene understanding output, obtains the understanding results of related questions, and outputs feedback to the user.

[0058] For example, the previously obtained scene interpretation diagram shows: there are 2 farmlands and 2 fires in the scene. Both fires are located within the first farmland, and both fires are relatively small in size, among other relationships. The user problem that the large-scale speech model needs to solve in this scene is: Is there a fire now, and what should I do? Inputting the scene interpretation diagram (in text form) and the problem to be solved (in text form) into the large-scale language model will provide text feedback: This scene is a farmland scene, and a fire is occurring. There are two fire points, located in the first farmland (with coordinates x and y marked). Please contact the safety officer immediately to extinguish the fire.

[0059] For the scene interpretation graph obtained in step S3, the relationships in the graph are input into the large language model in text form. Utilizing the summarizing capabilities of the large language model, the overall understanding and description of the scene can be output. Simultaneously, if there are other related questions about the scene image, these questions, along with the results of the scene interpretation graph, can be input into the large language model during the reasoning process to obtain the understanding results for the corresponding questions.

[0060] In some embodiments, the relative positional relationship includes the relative distance between the center positions, the relative distance between the boundary contours, and the intersection relationship of the boundary contours; the relative size relationship includes the relative area relationship and the boundary contour length relationship.

[0061] In some embodiments, the text prompts are different key prompts proposed based on the application scenario image and the problem to be analyzed.

[0062] An embodiment can be obtained based on the above method. This embodiment provides a mid-to-high-angle image scene understanding system based on a scene interpretation map, comprising:

[0063] The target detection box and target label acquisition module is used to input text prompts and scene images into the open set target detection model to obtain the target detection boxes and target labels needed for scene understanding.

[0064] The object detection box mask acquisition module is used to input the object detection box and scene image into the segmentation model and obtain the mask of the object detection box;

[0065] The scene interpretation map acquisition module is used to calculate the relative positions and sizes of targets using labels and masks to obtain a scene interpretation map;

[0066] The image scene understanding output module is used to analyze the scene interpretation graph through a large language model and output text results for image scene understanding.

[0067] Compared with traditional scene understanding methods, the method of this invention has the following advantages:

[0068] 1. Using the above technical solution enhances the generalizability of this method. In different application scenarios, different keywords can be extracted based on the characteristics of the application scenario image and the problem to be analyzed. This method generates different scene interpretation maps based on these keywords, thus obtaining different scene understanding results. Compared with existing methods, it achieves good application results without requiring more training data when changing application scenarios.

[0069] 2. Using the above technical solution can make this method more logical. In the process of scene understanding, this method will obtain the scene understanding result based on the calculated scene interpretation graph. The source is clear, and through precise relational numerical calculations, the scene understanding result has stronger logic and traceability.

[0070] 3. Using the above technical solution allows for a finer-grained process and more accurate scene understanding. This method uses a more precise target mask to calculate the scene interpretation map, which has a finer granularity compared to the square bounding boxes used in existing methods. Therefore, the calculation results are more accurate, leading to a more accurate final scene understanding result.

[0071] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein.

[0072] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0073] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for understanding mid-to-high-level image scenes based on scene interpretation maps, characterized in that, Includes the following steps: Input text prompts and scene images into the open set object detection model to obtain the object detection boxes and object labels needed for scene understanding; Input the object detection bounding box and the scene image into the segmentation model to obtain the mask of the object detection bounding box; By using labels and masks, the relative positions and sizes of targets are calculated to obtain a scene interpretation map; By analyzing the scene interpretation graph using a large language model, textual results for understanding the image scene are output. The method for inputting text prompts and scene images into the open-set object detection model includes: Acquire scene images; Pre-defined text descriptions of the scene to be analyzed for the scene images; The text description is input into a large language model, and the large language model generates the text prompt words. The method for obtaining a scene interpretation map by calculating the relative positions and sizes of targets using labels and masks includes: Obtain the center position, boundary contour shape, boundary length, and boundary pixel area of ​​the target detection box mask; The relative position and size relationship between the target and other objects in the scene image are calculated using the center position, boundary contour shape, boundary length, and boundary pixel area; this includes: The angular and positional relationships and containment relationships between the target and other scene objects are calculated by the mask boundary contour; the distance relationship between the target and other scene objects is calculated by the mask center position; and the relative area size relationship between the target and other scene objects is calculated by the pixel area occupied by the mask boundary. By using the relative positions and sizes of the target and other objects in the scene image, along with their labels, a scene interpretation map is created to represent the relationships between objects. The method for analyzing the scene interpretation graph using a large language model and outputting textual results for image scene understanding includes: The positional relationships, inclusion relationships, relative area relationships, and target labels between the target and other objects in the scene image are input into a large language model, which then outputs the text results of the image scene understanding.

2. The method for understanding mid-to-high-level image scenes based on scene interpretation maps according to claim 1, characterized in that, The method further includes: Input the image scene understanding related questions into a large language model; The large language model continues to reason based on the text results of the image scene understanding output, obtains the understanding results of related questions, and outputs feedback to the user.

3. The method for understanding mid-to-high point image scenes based on scene interpretation maps according to claim 1, characterized in that, The relative positional relationships include the relative distance between the center positions, the relative distance between the boundary contours, and the intersection relationship of the boundary contours. The relative size relationships include the relative area relationships and the boundary contour length relationships.

4. The method for understanding mid-to-high-level image scenes based on scene interpretation maps according to claim 1, characterized in that, The text prompts are different key prompts proposed based on the application scenario image and the problem to be analyzed.

5. A mid-to-high-level image scene understanding system based on scene interpretation maps, characterized in that, include: The target detection box and target label acquisition module is used to input text prompts and scene images into the open set target detection model to obtain the target detection boxes and target labels needed for scene understanding. The method for inputting text prompts and scene images into the open-set object detection model includes: Acquire scene images; Pre-defined text descriptions of the scene to be analyzed for the scene images; The text description is input into a large language model, and the large language model generates the text prompt words. The object detection box mask acquisition module is used to input the object detection box and scene image into the segmentation model and obtain the mask of the object detection box; The scene interpretation map acquisition module is used to calculate the relative positions and sizes of targets using labels and masks to obtain a scene interpretation map; The method for obtaining a scene interpretation map by calculating the relative positions and sizes of targets using labels and masks includes: Obtain the center position, boundary contour shape, boundary length, and boundary pixel area of ​​the target detection box mask; The relative position and size relationship between the target and other objects in the scene image are calculated using the center position, boundary contour shape, boundary length, and boundary pixel area; this includes: The angular and positional relationships and containment relationships between the target and other scene objects are calculated by the mask boundary contour; the distance relationship between the target and other scene objects is calculated by the mask center position; and the relative area size relationship between the target and other scene objects is calculated by the pixel area occupied by the mask boundary. By using the relative positions and sizes of the target and other objects in the scene image, along with their labels, a scene interpretation map is created to represent the relationships between objects. The image scene understanding output module is used to analyze the scene interpretation graph through a large language model and output text results for image scene understanding; The method for analyzing the scene interpretation graph using a large language model and outputting textual results for image scene understanding includes: The positional relationships, inclusion relationships, relative area relationships, and target labels between the target and other objects in the scene image are input into a large language model, which then outputs the text results of the image scene understanding.

6. The mid-to-high point image scene understanding system based on scene interpretation map according to claim 5, characterized in that, The open set target detection model is either the Grounding DINO model or the YOLO model.

7. A mid-to-high point image scene understanding system based on scene interpretation map according to claim 5, characterized in that, The segmentation model is a Segment Anything Model series model.

Images