A panoramic visual relationship detection method based on thought chain reasoning

Abstract

This invention provides a panoramic visual relationship detection method based on thought chain reasoning. It introduces image description technology into the thought chain reasoning process, enabling the model to simultaneously perform target detection and mutual verification based on generated description information and image information. Compared to existing technologies, this adds a posterior step, improving the effectiveness and reliability of target detection and making the generated relationships more consistent with actual descriptions. Furthermore, this invention performs relationship detection based on panoramic segmentation, achieving both target recognition and visual relationship detection simultaneously. Compared to existing technologies, it achieves a higher degree of discovery of the relationships between targets within an image. Finally, this invention can achieve visual relationship detection based on panoramic segmentation with zero or few samples, obtaining good relationship detection results without requiring extensive training data.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and in particular relates to a panoramic visual relationship detection method based on thought chain reasoning. Background Technology

[0002] Visual relationships refer to the interactions between objects in an image, while Visual Relationship Detection (VRD) detects visual relationships that exist in an image or video, such as... Figure 1 The image depicts a person riding a horse, where "person" and "horse" are the objects in the image, and "riding" represents the relationship between the two objects, represented by a triple of the form "<person, riding, horse>". In general, visual relationship detection involves two steps: the first step is to identify the objects in the image, typically achieved using object detection or instance segmentation methods; the second step is to determine the pairwise relationships between the existing objects, mainly relying on semantic and visual features. Mainstream VRD datasets include Visual Genome and Visual Relationship Dataset. However, in reality, the combinations of relationships and objects are diverse, and the semantic space is much larger than the objects or relationships themselves. Therefore, small-sample and open-domain visual relationship detection remains a major challenge.

[0003] In recent years, with the emergence of large visual models and multimodal large models, the mainstream approach is to use large models for zero-shot or small-shot VRD. For example, Li et al. designed mind chains to generate weights for different visual cues, thereby enhancing the discriminative power of similarity relationships and improving VRD performance; Huang et al. [4] A multi-stage relation-aware training scheme was designed, and the corresponding dataset was reconstructed. A RelationVLM model with the ability to understand semantic relationships, temporal correlations, and geometric transformations was proposed. Xie et al. segmented visual relationships into subtasks such as human interaction detection, scene graph generation, and object function recognition. They retrained the RelationLMM model for relation recognition using data reconstruction and instance augmentation methods, achieving Recall@50 of 51.0 and 65.9 on the Visual Genome and Open Images datasets, respectively.

[0004] Existing VRD methods mainly suffer from the following three problems: 1. The target and relationship data primarily come from existing datasets such as VG and VRDataset. However, these datasets exhibit a long-tail distribution of relationship data, leading existing methods to predict simpler relationships such as "on," "have," and "wear," which tend to achieve higher scores in these datasets. 2. Current relationship prediction primarily focuses on target detection tasks, i.e., identifying relationships between the main targets in an image. It fails to detect elements outside the target area, thus missing a significant amount of target and relationship information, impacting downstream tasks such as visual question answering and image information retrieval. 3. Current relationship detection relies heavily on semantic priors and visual features, potentially causing the model to depend more on language and prior knowledge from the dataset. For example, in an image of a person standing next to a motorcycle, the relationship between the target "person" and "motorcycle" might be predicted as "riding." Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a panoramic visual relationship detection method based on thought chain reasoning. This method obtains segmentation information of the input image through panoramic segmentation technology, and then uses thought chain reasoning to predict the relationships between all segments in the image based on panoramic segmentation, making the relationship detection results more consistent with the actual description.

[0006] A panoramic visual relationship detection method based on thought chain reasoning includes the following steps:

[0007] S1: Perform corpus segmentation on the image to be tested to obtain a segmented image composed of multiple segmented image corpora with different numbers and colors;

[0008] S2: Input the segmented images and the first prompt word into the multimodal large model for the first reasoning to obtain the category of each segmented image corpus. The first prompt word is: sequentially determine the category of the segmented image corpus corresponding to each number in the image.

[0009] S3: Input the second prompt word into the multimodal large model for the second reasoning to obtain the feature description of each segmented image corpus and the relationship description between each segmented image corpus. The second prompt word includes: a paragraph in JSON format describing the content of the segmented image.

[0010] S4: Clean and filter the results of the second reasoning according to the set rules to obtain visual relationship detection results that meet the requirements.

[0011] Furthermore, the second prompt word also includes: a JSON format with only two fields, segments and relations, where segments represent the feature description of the segmented image corpus, and relations represent the relationship description between the segmented image corpus. Segments only contain id and name, where id represents the sequence number of the segmented image corpus, and name represents the category of the segmented image corpus.

[0012] Furthermore, the second prompt word also includes: an example in JSON format as follows:

[0013] Suppose we have a segmented image whose content is 'a man sitting on a chair, the chair is on the ground', then the output is as follows: {"segments":[{"id":1,"name":"man"},{"id":2,"name":"chair"},{"id":3,"name":"ground"}],{"relations":[["man","chair","sitting"],["man","ground","irrelevant"],["chair","ground","on..."]]}.

[0014] Furthermore, before cleaning and filtering the results of the second reasoning according to the set rules, it is first determined whether there is any abnormality in the results of the second reasoning. If the result is yes, the abnormality is processed before proceeding to step S4. If the result is no, it is directly proceeded to step S4.

[0015] Furthermore, the method for determining whether the result of the second inference is abnormal is as follows:

[0016] Determine whether the result of the second inference exceeds the maximum token length limit and whether it is blocked by the security audit interface;

[0017] The method for handling exceptions is as follows:

[0018] Replace any abnormal results with the string "{"segments":[],"relations":[]}".

[0019] Furthermore, the method for determining whether the result of the second inference is abnormal is as follows:

[0020] Determine whether the result of the second inference contains Chinese characters “】”, “,”, “:”, or whether there are any missing matching punctuation marks;

[0021] The method for handling exceptions is as follows:

[0022] Replace Chinese symbols with English symbols;

[0023] Use a JSON repair tool to repair the result of the second inference. Re-parse the JSON content of the repaired output. If the parsing still fails, return the string "{"segments":[],"relations":[]}".

[0024] Furthermore, the method for cleaning and filtering the results of the second reasoning according to the established rules is as follows:

[0025] The greedy matching algorithm is used to extract the JSON format content from the result of the second inference and filter out redundant information.

[0026] Iterate through the obtained JSON content, reorder the segments by "id", filter out segments with "name" as "", "unknown", or "unrecognized", and delete the relationship corresponding to the segment.

[0027] Furthermore, the redundant information includes inference steps that have not been completely eliminated, annotations, and other fields in the JSON besides the specified fields for segments and relations.

[0028] Beneficial effects:

[0029] This invention provides a panoramic visual relationship detection method based on thought chain reasoning. It introduces image description technology into the thought chain reasoning process, enabling the model to simultaneously perform target detection and mutual verification based on generated description information and image information. Compared to existing technologies, this adds a posterior step, improving the effectiveness and reliability of target detection and making the generated relationships more consistent with actual descriptions. Furthermore, this invention performs relationship detection based on panoramic segmentation, achieving both target recognition and visual relationship detection simultaneously. Compared to existing technologies, it achieves a higher degree of discovery of the relationships between targets within an image. Finally, this invention can achieve visual relationship detection based on panoramic segmentation with zero or few samples, obtaining good relationship detection results without requiring extensive training data. Attached Figure Description

[0030] Figure 1 This invention provides an example of a target detection bounding box;

[0031] Figure 2 The flowchart illustrates a panoramic visual relationship detection method based on thought chain reasoning provided by this invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0033] like Figure 2 As shown, a panoramic visual relationship detection method based on thought chain reasoning includes the following steps:

[0034] S1: Perform corpus segmentation on the image to be tested to obtain a segmented image composed of multiple segmented image corpora with different numbers and colors;

[0035] It should be noted that the following uses Meta's SAM2 system as an example to describe in detail the process by which the present invention quickly obtains image segmentation annotations and overlays segmentation masks using this system;

[0036] S1.1 Calculate the maximum inscribed circle based on the image segmentation labels. Different segments are labeled with numbers using white text with a black border, and the center of the circle represents the position of the number. Regarding the algorithm for generating the maximum inscribed circle, let the input image be a binary mask image I∈{0,1}. H×W Where H and W are the height and width of the image, respectively, and the object region is defined as:

[0037] Ω={P=(x,y)∣I(x,y)=1}

[0038] The boundary region can then be represented as:

[0039]

[0040] Where N(q) represents the neighborhood pixels of q.

[0041] The specific steps of the algorithm are as follows: 1) For each point p∈Ω, calculate its Euclidean distance to the nearest boundary point:

[0042]

[0043] 2) Determine the maximum distance:

[0044]

[0045] And the corresponding center point (i.e., the center of the circle):

[0046]

[0047] 3) The final maximum inscribed circle is defined as:

[0048] C={p∈R 2 ||pc|| 2 ≤r max}

[0049] Where C is the circle formed by the center c and radius r max The defined closed circular region.

[0050] S2: Input the segmented images and the first prompt word into the multimodal large model for the first reasoning to obtain the category of each segmented image corpus. The first prompt word is: sequentially determine the category of the segmented image corpus corresponding to each number in the image.

[0051] S3: Input the second prompt word into the multimodal large model for a second inference to obtain feature descriptions of each segmented image corpus and descriptions of the relationships between each segmented image corpus. The second prompt word includes:

[0052] Generate a paragraph in JSON format describing the content of the segmented image;

[0053] The JSON format contains only two fields: `segments` and `relations`. `segments` describes the features of the segmented image corpus, and `relations` describes the relationships between the segmented image corpora. `segments` contains only `id` and `name`, where `id` represents the sequence number of the segmented image corpus, and `name` represents the category of the segmented image corpus. An example of the JSON format is shown below:

[0054] Suppose we have a segmented image whose content is 'a man sitting on a chair, the chair is on the ground', then the output is as follows: {"segments":[{"id":1,"name":"man"},{"id":2,"name":"chair"},{"id":3,"name":"ground"}],{"relations":[["man","chair","sitting"],["man","ground","irrelevant"],["chair","ground","on..."]]}.

[0055] In other words, this invention strictly controls the output format to be JSON, without any other elements or comments. By inputting prompt words into the large model, it sequentially determines the segmented image corpus corresponding to each number in the segmented image, and then generates a paragraph describing all segments contained in the segmented image and the relationships between each segment. Finally, it extracts the pairwise relationships between segments from the relationship description, and tries to distinguish different segments with different descriptions, such as: {"id":1,"name":"person wearing red clothes"}, {"id":2,"name":"person wearing a hat"}; At the same time, this invention can add examples to the output format of the large model through additional prompt words, such as: "The output is represented by triples, described as a man sitting on a chair, then the relationship is ["man","sitting","chair"].

[0056] S4: Clean and filter the results of the second inference according to the set rules to obtain visual relationship detection results that meet the requirements. The method for cleaning and filtering the results of the second inference according to the set rules is as follows:

[0057] The greedy matching algorithm is used to extract the JSON format content from the result of the second inference and filter out redundant information. The redundant information includes content in the string that does not belong to JSON, such as inference steps that have not been completely eliminated, comments, and other fields in JSON besides the specified fields of segments and relations.

[0058] Iterate through the obtained JSON content, reorder the segments by "id", filter out segments with "name" as "", "unknown", or "unrecognized", and delete the relationship corresponding to the segment.

[0059] It should be noted that the large model of this invention can be GLM-4V. Therefore, when using the GLM-4V API for online inference, exception detection and interrupt handling can be added to the code to prevent the loss of output results due to problems during the call process. Specifically:

[0060] Before cleaning and filtering the results of the second reasoning according to the set rules, it is first determined whether there is any abnormality in the results of the second reasoning. If the result is yes, the abnormality is processed before proceeding to step S4. If the result is no, it is directly proceeded to step S4.

[0061] The method to determine whether the result of the second inference is abnormal is as follows:

[0062] Determine whether the result of the second inference exceeds the maximum token length limit and whether it is blocked by the security audit interface;

[0063] It should be noted that this type of exception usually occurs due to a model call error, resulting in the inability to return a JSON result.

[0064] The method for handling exceptions is as follows:

[0065] In the module that calls the large model to capture the model output, set up exception handling (Python's try-except keyword). When the above exception is triggered, replace the result with the string "{"segments":[],"relations":[]}".

[0066] The method to determine whether the result of the second inference is abnormal is as follows:

[0067] The system checks whether the result of the second inference contains Chinese characters “】”, “,”, “:”, or missing matching punctuation marks, such as only “[”, one “”, or one “{”, which can cause JSON parsing errors. Alternatively, the output may contain incorrectly matched punctuation marks, such as “”’ or “[)”, which can also cause JSON parsing errors.

[0068] It should be noted that this type of exception usually occurs because the model returns a normal result, but the string cannot be correctly parsed into JSON.

[0069] The method for handling exceptions is as follows:

[0070] Replace Chinese symbols with English symbols;

[0071] Use a json repair tool (Python's json_repair module) to repair the result of the second inference. Re-parse the json content of the repaired output. If the parsing still fails, return the string "{"segments":[],"relations":[]}".

[0072] In summary, this invention provides a panoramic visual relationship detection method based on thought chain reasoning. First, existing panoramic segmentation data or data generated by panoramic segmentation models such as SAM are preprocessed to obtain segmented image corpora with serial number annotations in a unified format. Then, a thought chain prompting method is designed, and prompt examples are constructed. Next, the designed thought chain prompts are used in conjunction with image input to perform reasoning in a multimodal large model. Finally, this invention cleans and filters the reasoning results to ensure that the output results meet the requirements.

[0073] Therefore, this invention proposes a new panoramic visual relationship detection task. Compared with existing methods based on object detection or instance segmentation, this invention extends the target to every pixel in the image, thus expanding the amount of information that can be obtained from the image. Secondly, this invention uses a thought chain reasoning method based on image description, which enables the model to simultaneously perform relationship detection from text structure and image, and to verify each other, thereby making the relationship detection results more consistent with the actual description.

[0074] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A panoramic visual relationship detection method based on thought chain reasoning, characterized in that, Includes the following steps: S1: Perform corpus segmentation on the image to be tested to obtain a segmented image composed of multiple segmented image corpora with different numbers and colors; S2: Input the segmented images and the first prompt word into the multimodal large model for the first reasoning to obtain the category of each segmented image corpus. The first prompt word is: sequentially determine the category of the segmented image corpus corresponding to each number in the image. S3: Input the second prompt word into the multimodal large model for the second reasoning to obtain the feature description of each segmented image corpus and the relationship description between each segmented image corpus. The second prompt word includes: a paragraph in JSON format describing the content of the segmented image. S4: Determine if there is an anomaly in the result of the second reasoning. If the result is yes, handle the anomaly and then proceed to step S5. If the result is no, proceed directly to step S5. The method for determining whether the result of the second inference is abnormal is as follows: Determine whether the result of the second inference contains Chinese characters "】", ",", ":", or is missing matching punctuation marks; The method for handling exceptions is as follows: Replace Chinese symbols with English symbols; Use a JSON repair tool to repair the result of the second inference. Re-parse the JSON content of the repaired output. If the parsing still fails, return the string "{"segments": [], "relations": []}". S5: Clean and filter the results of the second reasoning according to the set rules to obtain visual relationship detection results that meet the requirements.

2. The panoramic visual relationship detection method based on thought chain reasoning as described in claim 1, characterized in that, The second prompt also includes: the JSON format has only two fields, segments and relations. Segments represents the feature description of the segmented image corpus, and relations represents the relationship description between the segmented image corpus. Segments only contains id and name, where id represents the sequence number of the segmented image corpus and name represents the category of the segmented image corpus.

3. The panoramic visual relationship detection method based on thought chain reasoning as described in claim 2, characterized in that, The second prompt also includes: Examples in JSON format are as follows: Suppose we have a segmented image containing the content 'a man sitting on a chair, the chair is on the ground', then the output would be as follows: {"segments": [{"id": 1, "name": "man"},{"id": 2, "name": "chair"},{"id": 3, "name": "ground"}],{"relations": [["man", "chair", "sitting"], ["man", "ground", "irrelevant"], ["chair", "ground", "on..."]]}.

4. The panoramic visual relationship detection method based on thought chain reasoning as described in claim 1, characterized in that, Methods for determining whether the result of the second inference is abnormal also include: Determine whether the result of the second inference exceeds the maximum token length limit and whether it is blocked by the security audit interface; The method for handling exceptions is as follows: Replace any results with the string "{"segments": [], "relations": []}".

5. The panoramic visual relationship detection method based on thought chain reasoning as described in claim 1, characterized in that, The method for cleaning and filtering the results of the second inference according to the set rules is as follows: The greedy matching algorithm is used to extract the JSON format content from the result of the second inference and filter out redundant information. Iterate through the obtained JSON content, reorder the segments by "id", filter out segments with "name" as " ", "unknown", or "unrecognized", and delete the relationship corresponding to the segment.

6. The panoramic visual relationship detection method based on thought chain reasoning as described in claim 5, characterized in that, The redundant information includes inference steps that have not been completely eliminated, annotations, and other fields in the JSON besides the specified fields for segments and relations.

Images