A visual positioning method, system and readable storage medium based on thought chain

Abstract

The application relates to the technical field of image processing, in particular to a visual positioning method and system based on a thought chain and a readable storage medium, which comprises the following steps: receiving an input target image and a query text, and preprocessing the target image to obtain a normalized image; inputting the query text into a language large model to obtain a text thought chain and extract a first semantic subject; inputting the first semantic subject and the normalized image into a target detection model and a multi-modal large model to obtain a candidate box list, a global description and a subject area description; constructing an image thought chain based on the candidate box list, the global description and the subject area description; integrating the text thought chain, the image thought chain, the normalized image, the query text and a rule prompt word according to a standard thought chain construction strategy to construct a standard thought chain, and inputting the standard thought chain into the multi-modal large model to obtain a first visual positioning result. The positioning accuracy is enhanced, and the processing capacity of a complex reasoning task is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a visual positioning method, system and readable storage medium based on thought chain. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, multimodal machine learning has made significant progress in fields such as visual understanding, human-computer interaction, and intelligent decision-making. Among these, referring expression comprehension (REC), as a key task bridging natural language and visual space, aims to accurately locate the corresponding object region in an image based on a natural language description, typically outputting it in the form of a bounding box. This task is widely used in scenarios such as image editing, robot navigation, assisted vision, and intelligent monitoring, and is one of the core technologies for realizing "language-guided vision."

[0003] Traditional visual localization methods are mainly based on two-stage detection frameworks, such as Faster R-CNN combined with attention mechanisms, which completes localization through candidate region generation and cross-modal matching. However, these methods rely on large amounts of labeled data and have certain limitations in complex semantic understanding. With the development of deep learning, especially the rise of large-scale pre-trained models, Multimodal Large Models (MLLMs) such as CLIP, BLIP, Qwen-VL, InternVL, and Florence have demonstrated powerful cross-modal alignment and semantic understanding capabilities, and are gradually becoming the mainstream technical approach for visual localization tasks.

[0004] Current mainstream multimodal visual localization methods typically employ an "end-to-end single-stage inference" architecture. This involves simultaneously inputting images and text descriptions into the model and directly outputting target bounding boxes. Specifically, images are encoded as visual tokens and fused with text tokens within the same Transformer architecture. Cross-modal alignment is achieved using a cross-attention mechanism, and the bounding box coordinates are ultimately regressed through a decoder. Furthermore, some models specifically designed for open-vocabulary detection use language embeddings as query vectors to guide the target detection head within the DETR architecture. Through a semantically driven detection mechanism, they can effectively identify entities in text descriptions and generate corresponding bounding boxes.

[0005] While the aforementioned models perform well on standard test sets, their accuracy remains significantly insufficient when handling high-order reasoning tasks such as complex spatial relationships, dynamic behavior descriptions, or multi-objective referencing resolution. According to publicly available benchmarks, current multimodal large models achieve an average ACC@0.5 of only 0.42–0.45 on the MARS2 dataset, far below the accuracy levels of traditional vision tasks such as image classification or object detection. This indicates that existing methods still face serious challenges in complex semantic understanding, contextual modeling, and inference robustness. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides a visual localization method, system and readable storage medium based on thought chain, which solves the technical problem that the accuracy of the current target detection method is still significantly insufficient when dealing with high-order reasoning tasks such as complex spatial relationships, dynamic behavior description or multi-target referencing resolution.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the main technical solutions adopted in this application include:

[0010] In a first aspect, embodiments of this application provide a visual positioning method based on thought chains, comprising:

[0011] S1. Receive the target image and the corresponding query text input by the user, and preprocess the target image to obtain a normalized image;

[0012] S2. Perform structured parsing on the pre-set language model input of the query text to obtain the corresponding text thought chain, and extract the first semantic subject in the text thought chain.

[0013] S3. Input the first semantic subject and the normalized image into a pre-set target detection model to obtain a candidate box list corresponding to the normalized image based on the first semantic subject; at the same time, input the first semantic subject and the normalized image into a pre-set multimodal large model to obtain a global description and subject region description corresponding to the normalized image.

[0014] The candidate box list includes the coordinates, categories, and confidence scores of all candidate bounding boxes in the image; the global description is a normalized natural language description of the overall scene and spatial relationship of the image; and the main region description is a natural language description of the region most relevant to the first semantic subject.

[0015] S4. Construct an image thinking chain based on the candidate box list, global description, and subject region description;

[0016] S5. Based on the pre-set standard thinking chain construction strategy, the text thinking chain, image thinking chain, normalized image, query text and pre-set rule prompt words are structurally integrated to construct the standard thinking chain, and the standard thinking chain is input into the multimodal large model to obtain the first visual positioning result.

[0017] Optionally, in one specific embodiment, the normalized image includes a first resolution normalized image and a second proportionally scaled normalized image;

[0018] Then, the target image is preprocessed, including:

[0019] Based on a pre-set dual-path preprocessing strategy, the target image is scaled at a fixed resolution and then scaled proportionally to obtain a first normalized image and a second normalized image, respectively. The dual-path preprocessing strategy is used to scale the target image at a pre-set fixed resolution to obtain the first normalized image, and to adjust the long side of the target image to a pre-set fixed number of pixels, and after scaling the short side at a fixed ratio, to fill the target image into a square with gray values ​​to obtain the second normalized image.

[0020] Optionally, in a specific embodiment, the first visual localization result includes multiple target detection boxes, and each target detection box has corresponding coordinates, category, confidence level and semantic interpretation;

[0021] After step S5, the method further includes:

[0022] S61. Sort the target detection boxes according to the confidence level of each target detection box, and select a preset number of target detection boxes from high to low confidence level.

[0023] S62. Based on all the selected target detection boxes and the coordinates, category, confidence and semantic interpretation of each target detection box, as well as the pre-set prompt word construction strategy, construct iterative prompt words;

[0024] S63. Input the iterative prompts and target image into the multimodal large model to obtain the first visual localization result after iterative enhancement. Repeat steps S61 to S63 until the IoU of the target detection boxes output in two consecutive rounds is greater than 0.9. Then stop the iteration and obtain the second visual localization result. The second visual localization result is the first visual localization result after the last round of iterative enhancement.

[0025] Optionally, in one specific embodiment, S2 includes:

[0026] Based on the query text and a pre-set structured suggestion strategy, construct structured suggestions corresponding to the query text;

[0027] The query text and structured prompts are input into a pre-set language model to generate a second semantic subject, a global overview, and a bilingual (Chinese and English) expression corresponding to the query text. Based on the second semantic subject, global overview, and bilingual (Chinese and English) expression, a textual thought chain is constructed. The language model is used to extract semantic objects from the query text based on structured prompts to obtain the second semantic subject in the query text, and to perform context-aware rewriting of the query text based on structured prompts and historical records to generate a corresponding global overview. It also generates a bilingual (Chinese and English) expression corresponding to the query text based on structured prompts.

[0028] Obtain the first semantic subject in the text's thought chain, and the first semantic subject is the same as the second semantic subject.

[0029] Optionally, in one specific embodiment, S3 includes:

[0030] The first semantic subject and the normalized image are input into a pre-set target detection model to perform target detection on the normalized image based on the first semantic subject, thereby annotating multiple candidate bounding boxes on the normalized image and obtaining the coordinates, category and confidence of each candidate bounding box.

[0031] Construct a candidate bounding box list based on the coordinates, category, and confidence score of all candidate bounding boxes;

[0032] The first semantic subject and the normalized image are input into a pre-set multimodal large model to perform a global description of the normalized image using natural language, thereby obtaining a global description of the normalized image. Additionally, the region where the first semantic subject is located is described using natural language based on the first semantic subject, thereby obtaining a description of the region corresponding to the normalized image.

[0033] Optionally, in one specific embodiment, after step S63, the method further includes:

[0034] S7. Perform semantic consistency verification on each target detection box in the second visual localization result based on the query text to determine whether each target detection box in the second visual localization result is consistent with the description in the query text; the semantic consistency verification is: based on the query text, the category and semantic interpretation corresponding to each target detection box, calculate the similarity between each target detection box and the query text.

[0035] If any target detection box has a similarity to the query text greater than a preset similarity threshold, then the second visual localization result is output.

[0036] Optionally, in one specific embodiment, S7 further includes:

[0037] If the similarity between all object detection boxes and the query text is less than the preset similarity threshold, then the candidate box list, as well as all object detection boxes in the first and second visual localization results, are input into the preset multimodal large model to obtain the third visual localization result.

[0038] Optionally, in one specific embodiment, the global description includes the image domain, scene type, and image description;

[0039] Then, by using natural language to perform a global description of the normalized image, a global description corresponding to the normalized image is obtained, including:

[0040] The image description corresponding to the normalized image is obtained by performing a global description of the normalized image using natural language;

[0041] Based on semantic subjects and image descriptions, as well as pre-set domain classification strategies, the image domain corresponding to the normalized image is determined.

[0042] Based on semantic subject, image description, and image domain, as well as a pre-set scene classification strategy, the scene type corresponding to the normalized image is determined.

[0043] Secondly, embodiments of this application provide a visual positioning system based on thought chain, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described visual positioning method based on thought chain.

[0044] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned visual positioning method based on thought chain.

[0045] (III) Beneficial Effects

[0046] This application presents a visual localization method based on thought chains. It preprocesses the target image using normalization, generates a text thought chain by structured parsing of the query text using a large language model, then constructs an image thought chain using an object detection model and a multimodal large model. Finally, it integrates the dual thought chains with the normalized image, query text, and rule-based hints, and inputs this into the multimodal large model for inference. This not only alleviates the bounding box offset problem and improves localization accuracy but also enhances the ability to handle complex inference tasks. Furthermore, it reduces the risk of false detections and strengthens inference robustness by utilizing full candidate box coverage and rule-based hints. Attached Figure Description

[0047] Figure 1 A schematic flowchart of a visual positioning method based on thought chain provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram of a method for enhancing knowledge through integrated multimodal reasoning based on thought chain, provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of image multi-scale normalization enhancement provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the text thinking chain generation process provided in the embodiments of this application;

[0051] Figure 5 This is a schematic diagram of the image thought chain generation process provided in the embodiments of this application;

[0052] Figure 6 A schematic diagram of standard thought chain integration provided for embodiments of this application;

[0053] Figure 7 This is a schematic diagram of the integrated reasoning and comparison process provided in the embodiments of this application. Detailed Implementation

[0054] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0055] In recent years, artificial intelligence technology has driven the rapid development of multimodal machine learning. Visual localization, as a core task connecting natural language and visual space, is widely used in scenarios such as image editing and robot navigation. Traditional methods either rely on large amounts of labeled data or adopt an "end-to-end single-stage inference" architecture. Although they achieve cross-modal alignment with large multimodal models, they have significant limitations when handling high-order inference tasks such as complex spatial relationships and dynamic behavior descriptions. Public evaluations show that the average ACC@0.5 of current large multimodal models on the MARS2 dataset is only 0.42~0.45, facing serious challenges in complex semantic understanding and inference robustness.

[0056] This application provides a visual localization method based on thought chains, which effectively overcomes the bottlenecks of existing technologies through standardized image preprocessing, dual thought chain construction, and structured integration reasoning. It first ensures the quality of visual input through preprocessing, then parses semantics through textual thought chains and integrates spatial and scene information through image thought chains, and finally combines rule-based prompts to form a standard thought chain reasoning. This not only alleviates bounding box offset and improves localization accuracy from the source but also enhances the ability to handle complex tasks while reducing the risk of false positives and false negatives. The solution does not require retraining a dedicated model, its modular design adapts to multiple scenarios, and it balances performance improvement with engineering practicality, significantly optimizing the overall performance of visual localization.

[0057] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0058] This application provides a visual positioning method based on thought chain, such as... Figure 1 As shown, it includes:

[0059] S1. Receive the target image and the corresponding query text input by the user, and preprocess the target image to obtain a normalized image;

[0060] S2. Perform structured parsing on the pre-set language model input of the query text to obtain the corresponding text thought chain, and extract the first semantic subject in the text thought chain;

[0061] S3. Input the first semantic subject and the normalized image into a pre-set target detection model to obtain a candidate box list corresponding to the normalized image based on the first semantic subject; at the same time, input the first semantic subject and the normalized image into a pre-set multimodal large model to obtain a global description and subject region description corresponding to the normalized image.

[0062] The candidate box list includes the coordinates, categories, and confidence scores of all candidate bounding boxes in the image; the global description is a normalized natural language description of the overall scene and spatial relationship of the image; and the main region description is a natural language description of the region most relevant to the first semantic subject.

[0063] S4. Construct an image thinking chain based on the candidate box list, global description, and subject region description;

[0064] S5. Based on the pre-set standard thinking chain construction strategy, the text thinking chain, image thinking chain, normalized image, query text and pre-set rule prompt words are structurally integrated to construct the standard thinking chain, and the standard thinking chain is input into the multimodal large model to obtain the first visual positioning result.

[0065] This application provides a visual localization method based on thought chains, which effectively overcomes the bottlenecks of existing technologies through standardized image preprocessing, dual thought chain construction, and structured integration reasoning. It first ensures the quality of visual input through preprocessing, then parses semantics through textual thought chains and integrates spatial and scene information through image thought chains, and finally combines rule-based prompts to form a standard thought chain reasoning. This not only alleviates bounding box offset and improves localization accuracy from the source but also enhances the ability to handle complex tasks while reducing the risk of false positives and false negatives. The solution does not require retraining a dedicated model, its modular design adapts to multiple scenarios, and it balances performance improvement with engineering practicality, significantly optimizing the overall performance of visual localization.

[0066] Optionally, in one specific embodiment, the normalized image includes a first resolution normalized image and a second proportionally scaled normalized image;

[0067] Then, the target image is preprocessed, including:

[0068] Based on a pre-set dual-path preprocessing strategy, the target image is scaled at a fixed resolution and then scaled proportionally to obtain a first normalized image and a second normalized image, respectively. The dual-path preprocessing strategy is used to scale the target image at a pre-set fixed resolution to obtain the first normalized image, and to adjust the long side of the target image to a pre-set fixed number of pixels, and after scaling the short side at a fixed ratio, to fill the target image into a square with gray values ​​to obtain the second normalized image.

[0069] Specifically, this application embodiment introduces image multi-scale normalization enhancement technology at the input end, and its processing flow is as follows: Figure 3 As shown, this method aims to alleviate the problems of target deformation and bounding box offset caused by image scaling. The original image (target image) first undergoes two different preprocessing paths: direct scaling and proportional scaling completion. Direct scaling involves scaling to a fixed resolution to preserve the feature response under a uniform input format. Proportional scaling completion involves scaling the longest side proportionally to a fixed pixel, and then adjusting the short side proportionally and filling it with gray values ​​to form a square, thereby maintaining the original aspect ratio and avoiding geometric distortion.

[0070] This dual-path preprocessing strategy generates normalized images at two scales, which are then input into the subsequent multimodal large model, providing a more accurate visual representation basis for targets at different scales. This preprocessing mechanism, as a pre-enhancement module in the entire inference process, significantly improves the detection stability of small and extreme-scale targets and greatly reduces the bounding box offset problem in the multimodal large model.

[0071] Optionally, in a specific embodiment, the first visual localization result includes multiple target detection boxes, and each target detection box has corresponding coordinates, category, confidence level and semantic interpretation;

[0072] After step S5, the method further includes:

[0073] S61. Sort the target detection boxes according to the confidence level of each target detection box, and select a preset number of target detection boxes from high to low confidence level.

[0074] S62. Based on all the selected target detection boxes and the coordinates, category, confidence and semantic interpretation of each target detection box, as well as the pre-set prompt word construction strategy, construct iterative prompt words;

[0075] S63. Input the iterative prompts and target image into the multimodal large model to obtain the first visual localization result after iterative enhancement. Repeat steps S61 to S63 until the IoU of the target detection boxes output in two consecutive rounds is greater than 0.9. Then stop the iteration and obtain the second visual localization result. The second visual localization result is the first visual localization result after the last round of iterative enhancement.

[0076] Specifically, this application provides a closed-loop feedback iterative inference mechanism to significantly improve positioning accuracy in complex scenarios. The specific implementation process is as follows:

[0077] First, after completing the initial knowledge-enhanced reasoning, i.e. after obtaining the first visual detection, the top-K high-confidence target detection boxes (usually K=3) of the output of this round are extracted, and their coordinates, categories, confidence and corresponding semantic interpretations are recorded (e.g., target detection box 1: wearing red clothes, facing the camera, located on the left).

[0078] Then, construct the iteration prompt words, in the following format:

[0079] The following are some reliable results obtained through model analysis. Please conduct refined verification and correction based on the following candidate results:

[0080] Result 1: [(x1,y1),(x2,y2)], Description: A black cat is sitting on the sofa.

[0081] Result 2: [(x1,y1),(x2,y2)], Description: A spotted cat sits on the sofa.

[0082] Result 3: [(x1,y1),(x2,y2)], Description: Orange cat

[0083] Please re-evaluate: Does any of the above three results contain a correct answer that satisfies the problem? If not, please output the correct bounding box.

[0084] The iterative prompt, along with the target image, is input again into the multimodal large model, guiding it to focus on the candidate region and re-perform spatial ranking and semantic verification. This iterative process can be configured to run 1-3 rounds. The new result output in each round replaces the previous round as the prior for the next round. The system sets a convergence condition: if the IoU of the bounding boxes output in two consecutive rounds is greater than 0.9, the iteration is terminated early.

[0085] This mechanism is particularly suitable for scenarios with dense targets, severe occlusion, or ambiguous descriptions. For example, in the case of "three girls in red standing in a row, with the middle one in profile," the first round might misjudge "middle" as "second," but by introducing candidate position information in the second round, the model can correctly identify that "second from the left" is actually the second person from the left.

[0086] Furthermore, in the embodiments of this application, the coordinates of the candidate detection block / target detection box are all coordinates in the standardized image pixel coordinate system (image coordinate system) in the visual task. This is a common coordinate standard in the field of computer vision, and it is also the core coordinate system for coordinate annotation, IoU calculation and iterative correction in the embodiments of this application.

[0087] The image pixel coordinate system takes the top-left corner of the normalized image as the origin (0,0), with the positive x-axis pointing horizontally to the right and the positive y-axis pointing vertically downwards. The coordinate values ​​are non-negative integers (unit: pixels) and completely correspond to the pixel arrangement positions of the image.

[0088] For example, if the resolution of the normalized image is 1120×1120 (width×height), then the x-coordinate range of any point in the image is 0~1119, and the y-coordinate range is 0~1119; in the coordinate format of the target detection box [(x1,y1),(x2,y2)], (x1,y1) is the pixel coordinate of the upper left corner of the detection box, and (x2,y2) is the pixel coordinate of the lower right corner of the detection box.

[0089] In line with the preprocessing stage, this embodiment of the application performs dual-path preprocessing on the target image to obtain a normalized image. The pixel coordinate system is established based on the normalized image, rather than the original image, to ensure that the coordinate reference is consistent during the iteration process and to avoid errors in IoU calculation and coordinate system correction due to differences in the size of the original image.

[0090] Adaptation to IoU calculation: The core of the iterative bracelet condition (IoU > 0.9 for two consecutive rounds) is to calculate the percentage of overlap between the two detection boxes. The coordinate values ​​in the pixel coordinate system are the basis of IoU calculation (IoU = overlapping pixel area / union pixel area). Only by unifying the pixel coordinate system can we accurately determine whether the detection boxes have converged. As for compatibility with multimodal large models, mainstream multimodal large models understand images based on pixel-level features. The output detection box coordinates are naturally adapted to the pixel coordinate system, without the need for additional coordinate transformation, thus reducing the complexity of iterative inference.

[0091] During the iteration process, the "candidate bounding box coordinates" input to the multimodal large model and the "new detection box coordinates" output in each round are based on the pixel coordinate system of the same normalized image. The combination of "coordinates + description" in the iteration prompt (such as "[(x1,y1),(x2,y2)],Description: A black cat is sitting on the sofa") is essentially a semantic association between the spatial position of the pixel coordinate system.

[0092] This application's embodiments focus on high-potential candidate bounding boxes through confidence screening, and combine structured iterative prompts to guide a multimodal large model to focus on candidate regions for refined semantic verification and spatial correction. Furthermore, an IoU > 0.9 convergence condition ensures stable localization results, significantly improving localization accuracy in complex scenarios such as dense targets, severe occlusion, or ambiguous descriptions. Simultaneously, a standard pixel coordinate system established based on normalized images ensures a unified coordinate reference during iteration, avoiding calculation errors caused by differences in original image sizes. It also naturally adapts to IoU calculation logic and the output characteristics of mainstream multimodal large models, eliminating the need for additional coordinate transformations and reducing inference complexity. Moreover, by binding "coordinates + semantic interpretation," dual convergence of spatial location and semantic description is achieved, further enhancing the accuracy and reliability of the localization results.

[0093] Optionally, in one specific embodiment, S2 includes:

[0094] Based on the query text and a pre-set structured suggestion strategy, construct structured suggestions corresponding to the query text;

[0095] The query text and structured prompts are input into a pre-set language model to generate a second semantic subject, a global overview, and a bilingual (Chinese and English) expression corresponding to the query text. Based on the second semantic subject, global overview, and bilingual (Chinese and English) expression, a textual thought chain is constructed. The language model is used to extract semantic objects from the query text based on structured prompts to obtain the second semantic subject in the query text, and to perform context-aware rewriting of the query text based on structured prompts and historical records to generate a corresponding global overview. It also generates a bilingual (Chinese and English) expression corresponding to the query text based on structured prompts.

[0096] Obtain the first semantic subject in the text's thought chain, and the first semantic subject is the same as the second semantic subject.

[0097] Specifically, this embodiment performs structured parsing on the initial question (query text) to generate a thought chain on the text side, the process of which is as follows: Figure 4 As shown.

[0098] The query text (e.g., "the second person from the left wearing blue clothes and running") is input into the language model, and prompting engineering guides it to output key semantic components: First, the semantic subject, i.e., the core target object (e.g., "blue clothes," "person"); second, a global overview based on multiple questions, generating a global overview by utilizing the spatial relationships between multiple questions; and third, bilingual (Chinese and English) expressions, which generate Chinese and English descriptions of the same semantic meaning in parallel, leveraging the complementarity of cross-linguistic expressions to enhance keyword robustness and reduce the risk of misjudgment due to ambiguity or expression deviation in a single language. These three parts are combined to form a textual thought chain, providing a structured and interpretable linguistic prior for subsequent multimodal alignment.

[0099] This application's embodiments not only transform fuzzy natural language queries into structured, interpretable language priors, providing clear semantic guidance for subsequent multimodal alignment, but also enhance keyword robustness through global overview integration of multi-question spatial associations and bilingual complementary expressions, effectively reducing the risk of misjudgment caused by single-language ambiguity or expression deviations. At the same time, it ensures the consistency and accuracy of semantic subjects, laying a solid language understanding foundation for cross-modal semantic matching and accurate visual positioning in complex scenarios, and improving the reliability and accuracy of overall reasoning.

[0100] The structure of the textual thought chain is as follows:

[0101] The question's English translation is: {Bilingual Expression in Chinese and English};

[0102] The core subject that needs to be detected in the problem is: {the first semantic subject};

[0103] The relationship between other targets in the image and the detected subject is: {Global Overview}.

[0104] Optionally, in one specific embodiment, S3 includes:

[0105] The first semantic subject and the normalized image are input into a pre-set target detection model to perform target detection on the normalized image based on the first semantic subject, thereby annotating multiple candidate bounding boxes on the normalized image and obtaining the coordinates, category and confidence of each candidate bounding box.

[0106] Construct a candidate bounding box list based on the coordinates, category, and confidence score of all candidate bounding boxes;

[0107] The first semantic subject and the normalized image are input into a pre-set multimodal large model to perform a global description of the normalized image using natural language, thereby obtaining a global description of the normalized image. Additionally, the region where the first semantic subject is located is described using natural language based on the first semantic subject, thereby obtaining a description of the region corresponding to the normalized image.

[0108] Furthermore, the global description includes the image domain, scene type, and image description;

[0109] Then, by using natural language to perform a global description of the normalized image, a global description corresponding to the normalized image is obtained, including:

[0110] The image description corresponding to the normalized image is obtained by performing a global description of the normalized image using natural language;

[0111] Based on semantic subjects and image descriptions, as well as pre-set domain classification strategies, the image domain corresponding to the normalized image is determined.

[0112] Based on semantic subject, image description, and image domain, as well as a pre-set scene classification strategy, the scene type corresponding to the normalized image is determined.

[0113] Specifically, the process of constructing an image-based thinking chain is as follows: Figure 5 As shown, on the visual side, an image thought chain is constructed simultaneously to align semantic and spatial information. This process is guided by the semantic subject in the text thought chain, and combined with the image after multi-scale normalization preprocessing, the object detection model and the multimodal large model are called for collaborative analysis respectively.

[0114] The object detection model is responsible for generating candidate bounding boxes for all potential objects in the image. The multimodal large model, based on the semantic subject and the whole image input, outputs a natural language description of the overall scene and spatial relationships of the image (e.g., "The image shows a street scene with pedestrians on the left, vehicles and trees on the right, and many people walking or running"). It further identifies the visual regions most relevant to the semantic subject. Thus, the image thinking chain includes both specific candidate bounding boxes (spatial information) and high-level scene semantics (contextual information), forming the basis of visual reasoning corresponding to the text thinking chain.

[0115] This application's embodiments utilize a collaborative analysis of an object detection model and a multimodal large model. The object detection model generates a full set of candidate bounding boxes containing coordinates, categories, and confidence levels, providing precise spatial information support. Simultaneously, the multimodal large model outputs a global scene and spatial relationship description, as well as a subject-specific description, supplementing the high-level semantic context. This allows the image thinking chain to achieve dual coverage of "spatial coordinates + semantic information." This approach ensures no potential targets are missed, providing a sufficient candidate base for subsequent screening. Furthermore, by establishing a strong semantic and spatial binding through scene and subject association descriptions, it effectively avoids positioning deviations caused by relying solely on coordinates or semantics. This forms a visual reasoning basis that precisely corresponds to the textual thinking chain, significantly improving the accuracy of cross-modal alignment and the reliability of positioning in complex scenes.

[0116] The structure of the image-based thought chain is as follows:

[0117] This image belongs to the {image scene} scene within the {image domain} domain;

[0118] The following is a semantic description of the input image: {image description};

[0119] The target of what might be a "black cat" in the image is: {Main area description}.

[0120] Furthermore, based on a pre-set standard thinking chain construction strategy, text thinking chains, image thinking chains, normalized images, query text, and pre-set rule prompts are structurally integrated to construct a standard thinking chain. This standard thinking chain is then input into the multimodal large model to obtain the first visual localization result (thinking chain knowledge enhancement output). The specific process is as follows: Figure 6 As shown, specifically:

[0121] After obtaining the text and image-based thought chains, these are combined with the original data (i.e., the target image and query text), rule hints (the target object may be small or partially occluded by other objects; consider whether the bounding box accurately covers the target object; consider the spatial description in the question (e.g., on the left, in the middle, etc.); find bounding boxes that can completely cover the target object), and candidate bounding boxes (results directly detected by existing open-source models) to construct an enhanced contextual hint, which is then input into a multimodal large model for final inference. The entire process simulates the cognitive path of human step-by-step reasoning, ultimately outputting a unique bounding box that satisfies all conditions and its confidence score. This integrated mechanism not only improves the localization accuracy under complex queries but also makes the reasoning process traceable and interpretable, significantly outperforming traditional end-to-end single-stage reasoning methods.

[0122] The standard strategy for constructing a mind chain is as follows:

[0123] We now need to perform a visual localization task, which outputs the bounding boxes (in the format [(x1,y1),(x2,y2)]) of the objects described in the question, based on the input question.

[0124] The original question is as follows: {query text};

[0125] The problem can be analyzed from the following perspectives:

[0126] <Textual Thinking Chain>;

[0127] The image is analyzed from the following perspectives;

[0128] <Image Thinking Chain>;

[0129] Please note the following rules during the testing process:

[0130] <Rule prompt>;

[0131] The following are the results obtained using some existing models, for reference:

[0132] <Candidate box list>.

[0133] Optionally, in one specific embodiment, after step S63, the method further includes:

[0134] S7. Perform semantic consistency verification on each target detection box in the second visual localization result based on the query text to determine whether each target detection box in the second visual localization result is consistent with the description in the query text; the semantic consistency verification is: based on the query text, the category and semantic interpretation corresponding to each target detection box, calculate the similarity between each target detection box and the query text.

[0135] If any target detection box has a similarity to the query text greater than a preset similarity threshold, then the second visual localization result is output.

[0136] Furthermore, if the similarity between all target detection boxes and the query text is less than a preset similarity threshold, then the candidate box list, as well as all target detection boxes in the first visual localization result and the second visual localization result, are input into a preset multimodal large model to obtain the third visual localization result.

[0137] Specifically, this embodiment provides a dynamic reasoning triage mechanism based on semantic verification (an integrated reasoning and comparison mechanism), the process of which is as follows: Figure 7 As shown, efficient processing of visual localization tasks of varying difficulty is achieved through the collaborative judgment of a lightweight detection model and a multimodal large model. First, a computationally efficient general-purpose object detection small model (such as Florence or Dino) is invoked to quickly infer the input image, generating preliminary object detection boxes and their category labels. Then, these object detection results are input into the multimodal large model, which performs semantic consistency verification. The multimodal large model is prompted to determine whether the targets detected by the small model are consistent with the user's query description. If the multimodal large model confirms that at least one candidate box highly matches the query in terms of category, attribute, or context, the task is classified as a "simple problem"; conversely, if the small model outputs results but the semantics are inconsistent, or it completely misses key targets, it is judged as a "complex problem."

[0138] For tasks identified as simple problems, the system no longer initiates the multi-model integration process. Instead, it directly uses the corresponding detection results optimized by the Iterative Enhancement (IRR) technique in Phase 2 as the final output. Since the objectives of such problems are clear and the semantics are well-defined, the results after iteration have already converged sufficiently in terms of accuracy. Using a single-path output can significantly save computational resources while maintaining high accuracy.

[0139] For tasks deemed complex, the system activates an ensemble selection mechanism: it aggregates multiple candidate bounding boxes generated from multi-scale images, multi-model architectures, and multiple rounds of iterative optimization to construct a comprehensive candidate set. Subsequently, the multimodal large model is invoked again as a "meta-decision maker" to perform fine-grained semantic scoring on each candidate and the original query in dimensions such as attributes, spatial relationships, behavioral states, and scene rationality. The candidate with the highest comprehensive score is selected as the final answer, i.e., the ensemble result.

[0140] After processing both simple and complex problems, the system aggregates the results of all input samples. For the very few samples that still fail to produce effective output in the above process (e.g., due to severely blurred images, extremely small targets, or highly abstract language descriptions causing all paths to fail), the system activates a fallback completion mechanism: the original image and query text are directly input into the multimodal large model, maximizing its reasoning capabilities (e.g., increasing the number of generation steps, activating the internal vision-language alignment module), performing end-to-end focused reasoning, and forcibly generating a bounding box as the final completion result.

[0141] This application's embodiments directly output iterative optimization results for simple tasks, significantly saving computational resources and improving inference efficiency while ensuring positioning accuracy. For complex tasks, an integrated comparison mechanism is activated, aggregating candidate boxes from multiple scales, models, and iterations, and filtering the optimal solution through fine-grained semantic scoring, effectively solving problems such as semantic mismatch and missed detections. A fallback completion mechanism is added to handle extreme scenarios, ensuring that results are still obtained even when there is no effective output. This balances efficiency and accuracy, while also improving the robustness and full coverage of visual positioning.

[0142] Furthermore, the MARS2 dataset from ICCV 2025 was used for testing. The dataset contains 23,104 image detection samples, covering 1,853 images. Each image corresponds to multiple natural language queries, forming a complex detection task set with multiple labels and perspectives. The dataset covers three core application areas: city, home, and education, and is further subdivided into several typical sub-scenes within each area, specifically including 12 sub-scenes such as roads, stations, airports, restaurants, scenic spots, living rooms, bedrooms, bathrooms, kitchens, playgrounds, classrooms, and libraries, fully reflecting the diversity and complexity of visual localization tasks in real-world environments.

[0143] In terms of problem description, the dataset includes both Chinese and English queries, with English queries predominating, ensuring the system's universality and robustness in cross-language environments. All detection tasks are presented in natural language, such as "A turned-off TV," "A blue chair," and "A man in a yellow shirt," requiring the model to understand the semantics and output the bounding box coordinates of the target object in the standard format [[x1,y1],[x2,y2]]. This data structure simulates the need for users to locate targets through natural language interaction in real-world application scenarios, such as smart security, smart home control, and educational assistance systems.

[0144] Before entering the inference process, all input images undergo preprocessing using the proposed multi-scale normalization enhancement technique (MsIN): each original image is processed in parallel into two scales. The first is a square image directly scaled to 1120×1120 pixels, preserving global feature responses at a uniform input size. The second is an image whose longest side is scaled proportionally to 1400 pixels, while the shorter side is scaled according to the original aspect ratio and filled with grayscale values ​​to a 1400×1400 square, maximizing the preservation of the target's geometric structure and avoiding deformation. These two scale images are used for different subsequent detection paths, providing a foundation for multi-scale result fusion.

[0145] In terms of base model selection, the experiment adopted three mainstream open-vocabulary object detection models as base detectors: Florence-ft (a domain-adjusted Florence-2 model), Grounding DINO, and LLMDet. Meanwhile, Qwen2.5-VL was selected as the core multimodal large model, undertaking key tasks such as thought chain generation, semantic verification, iterative reasoning, and ensemble decision-making. All models ran independently on the aforementioned dual-scale images to generate initial candidate bounding boxes.

[0146] The standard localization accuracy metric ACC@0.5 (where IoU ≥ 0.5 between the predicted bounding box and the ground truth annotation is considered correct) was used as the main evaluation standard. The results show that the performance of each basic detection model on this complex dataset is similar, with ACC@0.5 fluctuating around 0.40 (Florence-ft: 0.40, Grounding DINO: 0.41, LLMDet: 0.40), reflecting that the current method still has significant bottlenecks in handling complex semantics such as spatial relationships, attribute combinations, and referential resolution. When Qwen2.5-VL is used alone for end-to-end inference, its ACC@0.5 is 0.43, which is a slight improvement, but still limited by the semantic parsing capabilities of the single-stage inference mechanism.

[0147] Building upon this foundation, the enhancement module of this invention is introduced. First, a multimodal reasoning enhancement technique based on thought chains (CoT-MR) is applied in conjunction with iterative enhancement technique (IRR). Text and image thought chains are generated using dual-scale images to guide the model in step-by-step reasoning, and the bounding box position is corrected through multiple rounds of feedback. This stage significantly improves the overall ACC@0.5 to 0.59 without modifying any model parameters, and improves ACC@0.5 by 0.07 for complex problems such as occlusion, spatial orientation understanding, and counting / sorting.

[0148] Furthermore, an integrated reasoning and selection mechanism (DESCAR) is introduced: the system first uses a lightweight detection small model for initial screening, and then Qwen2.5-VL verifies the correctness of its output to distinguish between simple and complex problems; for complex problems, candidate results from multiple models and multiple iterations under the 1120×1120 and 1400×1400 dual-scale paths are integrated, and a multimodal large model is used as a meta-decision maker to perform semantic consistency comparison. Finally, fallback completion is initiated for samples without output. Ultimately, the complete solution of this invention achieves ACC@0.5=0.62 on the same test set, which is more than 55% higher than the basic model and about 44% higher than using Qwen2.5-VL alone.

[0149] In addition, this application provides a visual positioning system based on thought chain, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the above-mentioned visual positioning method based on thought chain.

[0150] Above this, such as Figure 2 As shown, this application provides an integrated multimodal reasoning knowledge enhancement method based on thought chain, including: basic detection, thought chain enhancement iteration, and integration selection;

[0151] The basic tests include:

[0152] The process involves acquiring raw data, including the target image and its corresponding query text. First, the target image is preprocessed using a multi-scale normalization enhancement mechanism. This involves generating a first-resolution normalized image and a second proportionally scaled normalized image through a dual-path preprocessing strategy. This preserves global feature responses at a uniform size while avoiding geometric distortion of the target, providing a high-quality visual foundation for subsequent localization. Simultaneously, the query text is processed through a text processing pipeline. A language big data model performs structured parsing to generate a textual thought chain containing a first semantic subject, a global overview, and bilingual (Chinese and English) expressions, clearly defining the core detection target and contextual relationships. The first semantic subject and the normalized image are then simultaneously input into the target detection model and the multimodal big data model. This yields a candidate box list, as well as a global description and subject region description corresponding to the normalized image. Based on these, an image thought chain is constructed. Finally, according to a standard thought chain construction strategy, the text thought chain, image thought chain, normalized image, query text, and pre-set rule prompts are structurally integrated to construct a standard thought chain. This standard thought chain is then input into the multimodal big data model to obtain the first visual localization result.

[0153] The mind chain is enhanced and iterated, including:

[0154] Based on the confidence level of each target detection box in the first visual localization result, sort them and select a preset number of target detection boxes from high to low confidence level. Based on all the selected target detection boxes and the coordinates, category, confidence level and semantic interpretation of each target detection box, as well as the pre-set prompt word construction strategy, construct iterative prompt words.

[0155] Input the iterative prompts and target image into the multimodal large model to obtain the first visual localization result after iterative enhancement. Repeat steps S61 to S63 until the IoU of the target detection boxes output in two consecutive rounds is greater than 0.9. Then stop the iteration and obtain the second visual localization result. The second visual localization result is the first visual localization result after the last round of iterative enhancement, i.e., the iterative enhancement result.

[0156] The integrated selection includes: performing semantic consistency verification on each target detection box in the second visual localization result based on the query text, so as to determine whether each target detection box in the second visual localization result is consistent with the description of the query text; the semantic consistency verification is: calculating the similarity between each target detection box and the query text based on the query text, the category and semantic interpretation corresponding to each target detection box;

[0157] If any target detection box has a similarity to the query text greater than a preset similarity threshold, then the second visual localization result is output.

[0158] This embodiment achieves a progressive process of "preprocessing standardization → thought chain structuring → reasoning enhancement → intelligent integration" through modular design. This not only solves the problem of insufficient understanding of complex semantics in traditional methods, but also balances positioning accuracy and computational efficiency through dynamic diversion.

[0159] Furthermore, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned thought chain-based visual positioning method.

[0160] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0161] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0162] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0163] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0164] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A visual positioning method based on thought chain, characterized in that, include: S1. Receive the target image and the corresponding query text input by the user, and preprocess the target image to obtain a normalized image; S2. Perform structured parsing on the pre-set language model input of the query text to obtain the corresponding text thought chain, and extract the first semantic subject in the text thought chain; S3. Input the first semantic subject and the normalized image into a pre-set target detection model to obtain a candidate box list corresponding to the normalized image based on the first semantic subject; at the same time, input the first semantic subject and the normalized image into a pre-set multimodal large model to obtain a global description and subject region description corresponding to the normalized image. The candidate box list includes the coordinates, categories, and confidence scores of all candidate bounding boxes in the image; the global description is a normalized natural language description of the overall scene and spatial relationship of the image; and the main region description is a natural language description of the region most relevant to the first semantic subject. S4. Construct an image thinking chain based on the candidate box list, global description, and subject region description; S5. Based on the pre-set standard thinking chain construction strategy, the text thinking chain, image thinking chain, normalized image, query text and pre-set rule prompt words are structurally integrated to construct the standard thinking chain, and the standard thinking chain is input into the multimodal large model to obtain the first visual positioning result.

2. The visual positioning method based on thought chain according to claim 1, characterized in that, The normalized image includes a first resolution normalized image and a second proportionally scaled normalized image; Then, the target image is preprocessed, including: Based on a pre-set dual-path preprocessing strategy, the target image is scaled at a fixed resolution and then scaled proportionally to obtain a first normalized image and a second normalized image, respectively. The dual-path preprocessing strategy is used to scale the target image at a pre-set fixed resolution to obtain the first normalized image, and to adjust the long side of the target image to a pre-set fixed number of pixels, and after scaling the short side at a fixed ratio, to fill the target image into a square with gray values ​​to obtain the second normalized image.

3. The visual positioning method based on thought chain according to claim 1, characterized in that, The first visual localization result includes multiple target detection boxes, and each target detection box has corresponding coordinates, category, confidence level and semantic interpretation; After step S5, the method further includes: S61. Sort the target detection boxes according to the confidence level of each target detection box, and select a preset number of target detection boxes from high to low confidence level. S62. Based on all the selected target detection boxes and the coordinates, category, confidence and semantic interpretation of each target detection box, as well as the pre-set prompt word construction strategy, construct iterative prompt words; S63. Input the iterative prompts and target image into the multimodal large model to obtain the first visual localization result after iterative enhancement. Repeat steps S61 to S63 until the IoU of the target detection boxes output in two consecutive rounds is greater than 0.

9. Then stop the iteration and obtain the second visual localization result. The second visual localization result is the first visual localization result after the last round of iterative enhancement.

4. The visual positioning method based on thought chain according to claim 3, characterized in that, S2 includes: Based on the query text and a pre-set structured suggestion strategy, construct structured suggestions corresponding to the query text; The query text and structured prompts are input into a pre-set language model to generate a second semantic subject, a global overview, and a bilingual (Chinese and English) expression corresponding to the query text. Based on the second semantic subject, global overview, and bilingual (Chinese and English) expression, a textual thought chain is constructed. The language model is used to extract semantic objects from the query text based on structured prompts to obtain the second semantic subject in the query text, and to perform context-aware rewriting of the query text based on structured prompts and historical records to generate a corresponding global overview. It also generates a bilingual (Chinese and English) expression corresponding to the query text based on structured prompts. Obtain the first semantic subject in the text's thought chain, and the first semantic subject is the same as the second semantic subject.

5. The visual positioning method based on thought chain according to claim 1, characterized in that, S3 includes: The first semantic subject and the normalized image are input into a pre-set target detection model to perform target detection on the normalized image based on the first semantic subject, thereby annotating multiple candidate bounding boxes on the normalized image and obtaining the coordinates, category and confidence of each candidate bounding box. Construct a candidate bounding box list based on the coordinates, category, and confidence score of all candidate bounding boxes; The first semantic subject and the normalized image are input into a pre-set multimodal large model to perform a global description of the normalized image using natural language, thereby obtaining a global description of the normalized image. Additionally, the region where the first semantic subject is located is described using natural language based on the first semantic subject, thereby obtaining a description of the region corresponding to the normalized image.

6. The visual positioning method based on thought chain according to claim 3, characterized in that, After step S63, the method further includes: S7. Perform semantic consistency verification on each target detection box in the second visual localization result based on the query text to determine whether each target detection box in the second visual localization result is consistent with the description in the query text; the semantic consistency verification is: based on the query text, the category and semantic interpretation corresponding to each target detection box, calculate the similarity between each target detection box and the query text. If any target detection box has a similarity to the query text greater than a preset similarity threshold, then the second visual localization result is output.

7. The visual positioning method based on thought chain according to claim 6, characterized in that, The S7 also includes: If the similarity between all object detection boxes and the query text is less than the preset similarity threshold, then the candidate box list, as well as all object detection boxes in the first and second visual localization results, are input into the preset multimodal large model to obtain the third visual localization result.

8. The visual positioning method based on thought chain according to claim 5, characterized in that, The global description includes the image domain, scene type, and image description; Then, by using natural language to perform a global description of the normalized image, a global description corresponding to the normalized image is obtained, including: The image description corresponding to the normalized image is obtained by performing a global description of the normalized image using natural language; Based on semantic subjects and image descriptions, as well as pre-set domain classification strategies, the image domain corresponding to the normalized image is determined. Based on semantic subject, image description, and image domain, as well as a pre-set scene classification strategy, the scene type corresponding to the normalized image is determined.

9. A visual positioning system based on thought chain, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the visual positioning method based on thought chain as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the visual positioning method based on thought chain as described in any one of claims 1 to 8.

Images