Method and apparatus for automatic construction of target segmentation data guided by instructions

Abstract

This invention provides a method and apparatus for automatically constructing target segmentation data guided by instructions. The method includes: acquiring an input image containing at least one target object; generating spatial cues for guiding segmentation based on the input image, and generating a segmentation mask for the target to be segmented based on the spatial cues; verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained; generating corresponding natural language segmentation instructions based on the segmentation mask that meets the semantic consistency requirements, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instructions. This method effectively ensures accurate alignment between the segmentation mask and the natural language instructions at the semantic level, realizes fully automatic construction of instruction-guided segmentation data without manual intervention, significantly reduces annotation costs, and enables the system to have good adaptability and scalability in open scenarios.

Description

Technical Field

[0001] This invention relates to the field of automatic data annotation technology, and in particular to a method and apparatus for automatically constructing target segmentation data guided by instructions. Background Technology

[0002] Object segmentation, a fundamental task in computer vision, aims to divide specific objects in an image into pixel-level regions. It holds significant value in practical applications such as autonomous driving, intelligent transportation systems, robotic environmental perception, and video surveillance. To achieve high-precision segmentation, mainstream methods typically rely on large amounts of training data with finely labeled pixels. However, this labeling process is highly dependent on human intervention, which is not only time-consuming and labor-intensive, but also susceptible to influences from subjective judgment, occlusion, lighting changes, and target deformation in complex and dynamic real-world scenarios. This makes it difficult to consistently guarantee the accuracy, consistency, and diversity of the labeling quality.

[0003] In recent years, instruction-guided target segmentation methods have gradually attracted attention. These methods attempt to improve the model's adaptability to open vocabularies and diverse semantic descriptions by specifying the target to be segmented through natural language instructions. However, constructing high-quality instruction-mask pairing data still generally relies on manually designing instructions and simultaneously annotating the corresponding segmentation regions, and the overall process remains dependent on intensive manual annotation. Some studies have attempted to introduce weak supervision or automatic annotation strategies to reduce annotation costs, but their evaluation criteria are mostly focused on the degree of matching between the segmented region and the geometric contour of the target, such as boundary overlap and region coverage, lacking an effective mechanism to determine whether the segmentation result truly corresponds to the semantic object referred to by the instruction.

[0004] In complex traffic scenarios, a single image often contains multiple traffic participants that look similar or are spatially adjacent, such as multiple vehicles of the same model or groups of pedestrians. In such cases, even if the segmentation result is close to a target in shape and location, if the actual corresponding object does not match the expected semantics—for example, missegmenting a "black SUV turning left" as a "gray SUV going straight"—it will still lead to serious semantic bias. If such errors are included in the training data, they may be solidified or even amplified during model training, thereby affecting the reliability of decisions in downstream tasks.

[0005] Furthermore, most current automatic segmentation systems are trained and inferred based on predefined closed sets of categories. When faced with new types of targets not encountered during the training phase or open-domain application scenarios, their generalization ability is significantly limited. This makes existing methods highly vulnerable to emerging new targets and interaction patterns in the real world, making it difficult to meet the actual needs of intelligent systems for continuous learning and open perception.

[0006] In summary, how to efficiently construct target segmentation annotation data that possesses both geometric accuracy and semantic correctness without requiring large-scale manual annotation or relying on a fixed category system has become a key bottleneck restricting the large-scale application of instruction-guided segmentation technology in complex real-world scenarios. Summary of the Invention

[0007] This invention provides a method and apparatus for automatically constructing instruction-guided target segmentation data, which solves the problems of high cost, poor scalability, and difficulty in ensuring semantic consistency in the existing technology of instruction-guided target segmentation that relies on manual annotation. It realizes automatic selection of target objects in complex traffic scenarios, zero-sample segmentation, semantic consistency verification, and segmentation instruction generation, thereby constructing high-quality, controllable, and semantically consistent instruction-guided target segmentation annotation data.

[0008] On one hand, the present invention provides an instruction-guided automatic construction method for target segmentation data, comprising: acquiring an input image containing at least one target object; generating a spatial cue for guiding segmentation based on the input image, and generating a segmentation mask for a target to be segmented based on the spatial cue, wherein the target to be segmented is a selected target object; verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirement is obtained; generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction.

[0009] Furthermore, the step of generating spatial cues for guiding segmentation based on the input image includes: performing candidate target detection on the input image to obtain multiple candidate target regions; scoring each candidate target region based on preset rules, and selecting the candidate target region with the highest score as the target to be segmented; and generating corresponding bounding boxes as spatial cues based on the target to be segmented.

[0010] Furthermore, generating a segmentation mask for the target to be segmented based on the spatial cues includes: inputting the input image and the spatial cues into a basic segmentation model, and outputting a segmentation mask for the target to be segmented without needing to train for a specific task or target category.

[0011] Furthermore, verifying the semantic consistency between the segmentation mask and the target to be segmented includes: inputting the input image, the segmentation mask, and the initial semantic description of the target to be segmented into a multimodal semantic understanding model; determining whether the segmentation mask and the target to be segmented are consistent in terms of category, attribute, and contextual relationship through the multimodal semantic understanding model, and outputting the verification result.

[0012] Furthermore, the iterative correction process for segmentation when verification fails includes: adjusting the position or range of the spatial hints and regenerating the segmentation mask based on the adjusted spatial hints; and performing semantic consistency verification on the newly generated segmentation mask again until verification passes or the preset maximum number of iterations is reached.

[0013] Furthermore, the step of generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement includes: extracting visual semantic features of the target to be segmented based on the segmentation mask that meets the semantic consistency requirement; and using a multimodal semantic understanding model to convert the visual semantic features into a natural language description that includes at least one of appearance attributes, spatial location, and relative relationship, as the natural language segmentation instruction.

[0014] Furthermore, the preset rules include at least one of visual saliency rules, spatial location rules, target scale rules, and scene semantic rules; wherein, the visual saliency rules are used to evaluate the saliency of the candidate target region in the input image; the spatial location rules are used to evaluate whether the candidate target region is located in the central region of the input image or a scene interest region; the target scale rules are used to evaluate whether the area of ​​the candidate target region is within a preset reasonable range; the scene semantic rules are used to evaluate whether the category or attribute of the candidate target region meets the annotation priority conditions under the current scene; the score is obtained by weighted fusion of the score results of each rule.

[0015] Furthermore, if the semantic consistency verification is still not passed after the iterative correction segmentation process reaches the preset maximum number of iterations, the corresponding input image and associated segmentation mask are marked as samples to be manually reviewed and are not included in the final output structured annotation results.

[0016] Furthermore, when the method is applied to intelligent transportation scenarios, the target objects include at least one of vehicles, pedestrians, non-motorized vehicles, traffic signs, and road structures; the semantic consistency verification process combines traffic scenario context information to determine whether the segmentation mask conforms to the road topology, driving direction constraints, and the rationality of traffic participant behavior.

[0017] Secondly, the present invention also provides an instruction-guided automatic target segmentation data construction apparatus, comprising: an image acquisition module for acquiring an input image containing at least one target object; a segmentation mask generation module for generating spatial prompts for guiding segmentation based on the input image, and generating a segmentation mask for a target to be segmented based on the spatial prompts, wherein the target to be segmented is a selected target object; a semantic verification and correction module for verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process when verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained; and an annotation generation and output module for generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirements, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction. The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the instruction-guided automatic target segmentation data construction methods described above.

[0018] Thirdly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a target segmentation data automatic construction method guided by any of the above-described instructions.

[0019] Fourthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an automatic target segmentation data construction method guided by any of the above-described instructions.

[0020] Fifthly, the instruction-guided automatic construction method for target segmentation data provided by this invention acquires an input image containing at least one target object, generates spatial cues for guiding segmentation based on the input image, generates a segmentation mask for the target to be segmented based on the spatial cues (the selected target object), verifies the semantic consistency between the segmentation mask and the target to be segmented, and iteratively corrects the segmentation process if the verification fails until a segmentation mask that meets the semantic consistency requirements is obtained. Based on the segmentation mask that meets the semantic consistency requirements, a corresponding natural language segmentation instruction is generated, and a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction is output. This method, by introducing a semantic consistency verification and iterative correction mechanism, effectively ensures accurate alignment between the segmentation mask and the natural language instruction at the semantic level, achieves fully automatic construction of instruction-guided segmentation data without manual intervention, significantly reduces annotation costs, and gives the system good adaptability and scalability in open scenarios. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the instruction-guided automatic construction method for target segmentation data provided in this embodiment of the invention.

[0023] Figure 2 This is a schematic diagram of the overall process of the instruction-guided automatic construction method for target segmentation data provided in the embodiments of the present invention.

[0024] Figure 3 This is a schematic diagram of the selection process for the target to be segmented provided in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the semantic consistency verification process between the segmentation mask and the target to be segmented provided in the embodiments of the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of the instruction-guided automatic construction device for target segmentation data provided in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] With the rapid development of computer vision and artificial intelligence technologies, object segmentation, as a key fundamental task in visual perception, is widely used in scenarios such as autonomous driving, intelligent transportation, robot perception, and intelligent monitoring. In practical applications, to achieve accurate understanding of target objects in complex scenes, a large amount of training data with pixel-level annotations is usually required. However, the pixel-level segmentation annotation process is highly dependent on manual intervention, resulting in high annotation costs, low efficiency, and susceptibility to subjective factors in complex scenarios, making it difficult to meet the practical needs of large-scale data construction and rapid iteration.

[0030] In recent years, instruction-guided target segmentation has gradually become an important research direction in multimodal visual understanding. This type of method introduces natural language instructions, enabling the system to segment specified targets based on text descriptions, thereby improving the model's flexibility and generalization ability. However, most existing instruction-guided segmentation methods rely on manually designed or manually annotated instruction-segmentation mask pairs, still requiring a large amount of manual intervention. This not only results in high annotation costs but also makes it difficult to guarantee the accuracy, consistency, and diversity of instruction descriptions.

[0031] Meanwhile, existing automatic or weakly supervised annotation methods typically focus on geometric correctness, such as whether the segmented region covers the target subject, but lack a systematic verification mechanism for the consistency between the segmentation result and the target semantics. In complex scenarios, even if the segmentation result seems reasonable in spatial location and shape, problems such as target semantic errors, unclear referencing, or mismatched contextual relationships may still occur. Once these errors are introduced into the dataset, they will be continuously amplified during subsequent model training, seriously affecting model performance and reliability.

[0032] Furthermore, existing automated segmentation methods often rely on predefined target category systems or are trained for specific tasks. When the application scenario changes or the target types expand, it is often necessary to re-collect data and retrain the model, making it difficult to adapt to the increasing diversity of targets in open scenarios. This dependence on task-specific training and category constraints limits the application scope and scalability of segmentation technology in real-world complex scenarios.

[0033] With the development of multimodal large language models, they have demonstrated powerful capabilities in cross-modal semantic understanding and reasoning, providing new technological possibilities for visual perception tasks. However, existing technologies lack a complete technical solution that can fully utilize the reasoning capabilities of multimodal large language models to achieve automatic target object selection, semantic verification of segmentation results, and generation of segmentation instructions without task-specific training. In particular, there is a lack of a closed-loop automatic annotation mechanism with semantic consistency verification and error correction capabilities.

[0034] Therefore, how to achieve automatic selection, zero-shot segmentation, semantic consistency verification, and generation of natural language instructions consistent with the segmentation results for target objects in complex scenes without relying on manual frame-by-frame annotation and task-specific training, and how to construct high-quality, controllable, and semantically consistent instruction-guided target segmentation annotation data, remains a technical problem that urgently needs to be solved in this field.

[0035] To address the problems of high cost, poor scalability, and difficulty in ensuring semantic consistency in existing target segmentation and instruction-guided segmentation annotation processes that rely on manual annotation, this invention proposes an automatic construction method for instruction-guided target segmentation data.

[0036] Specifically, Figure 1 The diagram illustrates a flowchart of the instruction-guided automatic construction method for target segmentation data provided in an embodiment of the present invention.

[0037] like Figure 1 As shown, the method includes: S110, acquiring an input image containing at least one target object; S120, generating spatial cues for guiding segmentation based on the input image, and generating a segmentation mask for the target to be segmented based on the spatial cues, wherein the target to be segmented is the selected target object; S130, verifying the semantic consistency between the segmentation mask and the target object, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirement is obtained; S140, generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction.

[0038] The instruction-guided automatic construction method for target segmentation data provided in this invention is used to automatically generate high-quality, semantically consistent segmentation and annotation data without human intervention. It can be applied to scenarios that require large-scale vision-language alignment data, such as intelligent transportation, intelligent monitoring, and robot perception, and is not specifically limited here.

[0039] The following will take the application scenario of intelligent transportation as an example and describe steps S110-S140 and related steps in detail.

[0040] S110, Obtain an input image containing at least one target object.

[0041] The input image is a single frame of traffic scene image captured by an onboard camera or roadside monitoring equipment, with a resolution of 1920×1080 or other standard sizes.

[0042] The target objects are entities with independent semantic identities in the traffic environment, including but not limited to motor vehicles, non-motor vehicles, pedestrians, traffic signs, road markings, or construction cones.

[0043] Input images can be acquired in real time through the image acquisition interface, or retrieved on demand from the historical video database; no specific limitation is made here.

[0044] Based on obtaining the input image of at least one target object in step S110, step S120 is further executed.

[0045] S120, generate a spatial cue for guiding segmentation based on the input image, and generate a segmentation mask for the target to be segmented based on the spatial cue, wherein the target to be segmented is the selected target object.

[0046] Spatial cues are geometric guidance information used to indicate the approximate location of the target to be segmented in the input image. In this embodiment, the spatial cues are represented in the form of bounding boxes, and their generation method is as follows: First, a general object detection model is used to perform preliminary analysis on the input image to obtain multiple candidate target regions; then, a candidate target region is selected based on preset rules (such as regional saliency, center offset, or priority of traffic interest areas), and its bounding rectangle is used as the initial spatial cue.

[0047] A segmentation mask is a binary image with the same spatial dimensions as the input image, where regions with a pixel value of 1 represent the target object and regions with a pixel value of 0 represent the background. The segmentation mask is generated by calling a general-purpose base segmentation model, which can output accurate pixel-level segmentation results based solely on the input image and spatial cues, without relying on specific category training. For example, when a spatial cue defines a car, the general-purpose base segmentation model will output a complete contour mask of the vehicle, even if that car model was not present during training.

[0048] Step S130 is executed after generating spatial cues for guiding segmentation based on the input image in step S120 and generating a segmentation mask for the target to be segmented based on the spatial cues.

[0049] S130, verify the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correct the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained.

[0050] Semantic consistency refers to the fact that the visual content covered by the segmentation mask truly corresponds to the expected target object at the semantic level, rather than being an incorrect segmentation caused by occlusion, similar appearance, or positioning deviation. For example, in an intersection scenario, if the target object is a "white SUV turning left," but the segmentation mask actually covers a gray sedan in the adjacent lane, it is considered semantically inconsistent.

[0051] In this embodiment, semantic consistency verification is performed by a multimodal semantic understanding model. The multimodal semantic understanding model can jointly parse the input image, the segmentation mask, and the initial semantic description of the target to be segmented (such as "a large bus in the center of the image") to determine whether the three are logically consistent in terms of category, color, behavioral intent, and scene context.

[0052] If the verification result is inconsistent, an iterative correction process is initiated: the position or range of the original spatial hint is adjusted (e.g., shifted 50 pixels to the left or increased by 20%), and a new segmentation mask is generated based on the new spatial hint, and verification is performed again. This process is repeated a maximum of a preset number of iterations (e.g., 3 times). If verification still fails, the current processing is terminated and marked as abnormal; otherwise, the segmentation mask that finally passes verification is retained for subsequent steps.

[0053] In step S130, the semantic consistency between the segmentation mask and the target to be segmented is verified, and the segmentation process is iteratively corrected if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained. Then, step S140 is executed.

[0054] S140, Generate a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement, and output a structured annotation result containing the input image, the segmentation mask and the corresponding natural language segmentation instruction.

[0055] Natural language segmentation instructions are statements describing the segmentation target in human-readable text form. Their content includes not only basic categories (such as "bus"), but also multi-dimensional semantic information such as appearance attributes (such as "yellow body"), spatial location (such as "located in the lower right corner of the image"), relative relationships (such as "adjacent to a pedestrian crossing"), and behavioral states (such as "currently stopping at a bus stop"). Natural language segmentation instructions are automatically generated by a multimodal semantic understanding model based on validated segmentation masks, ensuring strict alignment between the language description and the visual region.

[0056] Structured annotation results are data units that organize the input image, segmentation mask, and corresponding natural language segmentation instructions in an associative manner. In this embodiment, the structured annotation results are stored in a unified data format, for example, an image file (.jpg), a mask file (.png), and a text description file (.txt) are grouped together, or encapsulated into a single JSON structure, which includes the image path, mask encoding, and instruction string. The structured annotation results can be directly used to train downstream instruction-guided segmentation models, or as alignment training samples for multimodal large models, significantly reducing the cost of manual annotation and improving the semantic quality of the data.

[0057] In this embodiment, an input image containing at least one target object is acquired, and spatial cues for guiding segmentation are generated based on the input image. A segmentation mask for the target to be segmented is then generated based on the spatial cues, with the target being the selected object. The semantic consistency between the segmentation mask and the target to be segmented is verified, and the segmentation process is iteratively corrected if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained. Based on the segmentation mask that meets the semantic consistency requirements, a corresponding natural language segmentation instruction is generated, and a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction is output. This method, by introducing a semantic consistency verification and iterative correction mechanism, effectively ensures the accurate alignment of the segmentation mask and the natural language instruction at the semantic level, achieving fully automatic construction of instruction-guided segmentation data without manual intervention, significantly reducing annotation costs, and giving the system good adaptability and scalability in open scenarios.

[0058] Based on the above embodiments, the process of generating spatial cues for guiding segmentation based on the input image in step S120 will be described in detail below.

[0059] The process involves generating spatial cues to guide segmentation based on the input image, including: detecting candidate objects in the input image to obtain multiple candidate object regions; scoring each candidate object region based on preset rules and selecting the candidate object region with the highest score as the target to be segmented, which is the selected target object; and generating corresponding bounding boxes as spatial cues based on the target to be segmented.

[0060] The preset rules include at least one of the following: visual saliency rules, spatial location rules, target scale rules, and scene semantic rules. Among them, the visual saliency rules are used to evaluate the saliency of the candidate target region in the input image; the spatial location rules are used to evaluate whether the candidate target region is located in the central region of the input image or the scene of interest; the target scale rules are used to evaluate whether the area of ​​the candidate target region is within a preset reasonable range; and the scene semantic rules are used to evaluate whether the category or attribute of the candidate target region meets the annotation priority conditions under the current scene. The score is obtained by weighted fusion of the score results of each rule.

[0061] It is easy to understand that spatial cues for guiding segmentation are generated based on the input image. This application achieves this by introducing a rule-based target selection mechanism and a bounding box expansion strategy, thereby improving the controllability of target selection and the robustness of subsequent segmentation.

[0062] Specifically, firstly, to avoid indiscriminate processing of all visible objects in the input image, this embodiment performs candidate object detection on the input image to obtain multiple candidate object regions. These candidate object regions are local regions in the input image that may contain independent semantic entities. Their form can be a bounding box output by the object detection model, a mask bounding rectangle generated by the instance segmentation model, or a region of interest generated by the object proposal algorithm; no specific limitations are imposed here. The number and location of candidate object regions are determined by the image content and are not limited in this regard.

[0063] Subsequently, in order to select a target to be segmented from multiple candidate target regions, this embodiment introduces preset rules to uniformly score each candidate target region. The preset rules include at least one of visual saliency rules, spatial location rules, target scale rules, and scene semantic rules.

[0064] Among them, the visual saliency rule is used to evaluate the visual prominence of candidate target regions in the input image, such as calculating their saliency score based on color contrast, texture complexity, or edge density; the spatial location rule is used to evaluate whether the candidate target region is located in the central region of the input image or the region of interest in the current scene, such as near lane lines or pedestrian crossings in a traffic scene; the target scale rule is used to evaluate whether the area of ​​the candidate target region is within a preset reasonable range to exclude abnormal regions that are too small (such as noise) or too large (such as covering the entire image); and the scene semantic rule is used to evaluate whether the category or attribute of the candidate target region meets the labeling priority conditions in the current scene. For example, in an intelligent transportation scene, pedestrians, non-motorized vehicles, or vehicles changing lanes can be set as high-priority labeling objects.

[0065] For each candidate target region, its individual score is calculated according to the aforementioned preset rules, and a comprehensive score is obtained through weighted fusion. The weights of each preset rule can be pre-configured according to application scenario requirements to reflect the relative importance of different rules in the target selection process. Finally, the candidate target region with the highest comprehensive score is selected as the target to be segmented.

[0066] After identifying the target to be segmented, a corresponding bounding box is generated based on the target's spatial location in the input image, serving as a spatial cue to guide segmentation. The bounding box is the smallest rectangular region enclosing the target, and its parameters include the coordinates of its top-left corner, width, and height. The bounding box of the target can be directly derived from the output of the aforementioned candidate target detection stage, or it can be generated by a multimodal semantic understanding model based on the target's semantic description and visual features; no specific limitation is made here.

[0067] To further improve the integrity and robustness of subsequent segmentation, the generated bounding box can be expanded to obtain an expanded bounding box. Expansion refers to extending the original bounding box outwards by a certain number of pixels according to a preset ratio to ensure that the target subject (especially the parts near the boundary) is completely contained within the indicated area. The expansion ratio can be adaptively adjusted according to image resolution, target scale, or specific application requirements. For example, in high-resolution traffic images, a larger expansion ratio is used for small targets (such as cyclists) to compensate for detail loss caused by shooting from a distance.

[0068] In this embodiment, through the target selection and bounding box expansion mechanism constrained by the above rules, it is possible to automatically focus on objects with semantic value and suitable for segmentation without manually specifying the target, and generate spatial hints with fault tolerance for them, thereby providing a high-quality input foundation for subsequent zero-shot segmentation and semantically consistent instruction generation.

[0069] Based on the above embodiments, the process of generating a segmentation mask according to spatial prompts in step S120 will be described in detail below.

[0070] Generating a segmentation mask for the target to be segmented based on spatial cues includes: inputting the input image and spatial cues into a base segmentation model, and outputting a segmentation mask for the target to be segmented without needing to train for a specific task or target category.

[0071] It is easy to understand that, for the step of generating the segmentation mask of the target to be segmented based on spatial cues, this embodiment achieves zero-shot target segmentation by calling the basic segmentation model. This process does not rely on any supervised training for a specific task or target category and is suitable for automatic annotation construction in open vocabulary scenarios.

[0072] After obtaining spatial cues (e.g., expanded target bounding boxes) to guide segmentation, the input image and spatial cues are fed together into the base segmentation model. The base segmentation model is a general-purpose visual model designed to perform pixel-level segmentation of any target using only the input image and spatial cues during the inference phase, without prior supervised training on that target category. The input to the base segmentation model includes at least image data and spatial cue information, and the output is a segmentation mask corresponding to the target to be segmented. The segmentation mask is a binary image with the same spatial dimensions as the input image, where pixels in the target region are labeled as foreground and the remaining regions as background, thus achieving accurate pixel-level region division.

[0073] The meaning of "no need to train for a specific task or target category" in this embodiment is that the basic segmentation model has been pre-trained on large-scale general data during deployment, possessing cross-category and cross-scene generalization capabilities. However, when performing the current segmentation task, it is not fine-tuned, task-specific weights are not loaded, and it does not rely on a category label dictionary. Therefore, even when faced with new types of targets not seen during the training phase, such as new types of special vehicles or rare traffic facilities, the model can still locate and segment their complete outlines based on spatial cues.

[0074] In this embodiment, spatial cues are provided in the form of bounding boxes, serving as spatial prior information to guide the segmentation process to focus on the target region. On the one hand, this significantly suppresses background interference, such as distant buildings, the sky, or unrelated vehicles; on the other hand, it enhances the model's robustness to changes in target scale, partial occlusion (such as pedestrians partially obscured by trees), and irregular shapes (such as deformed construction cones). By introducing spatial priors, the basic segmentation model can stably output geometrically accurate segmentation results in open environments.

[0075] The generated segmentation mask is directly used in the semantic consistency verification and natural language segmentation instruction generation steps, and its quality directly affects the reliability of the final structured annotation results. Therefore, this step, through a zero-shot segmentation mechanism, provides a foundation for constructing semantically consistent and structurally complete instruction-guided segmentation data while ensuring high generalization ability.

[0076] Based on the above embodiments, the following will further describe in detail the process of verifying the semantic consistency between the segmentation mask and the target object in step S130.

[0077] Verifying the semantic consistency between the segmentation mask and the target to be segmented includes: inputting the input image, the segmentation mask, and the initial semantic description of the target to be segmented into a multimodal semantic understanding model; using the multimodal semantic understanding model to determine whether the segmentation mask and the target to be segmented are consistent in terms of category, attribute, and contextual relationship, and outputting the verification result.

[0078] It is easy to understand that this embodiment achieves semantic consistency verification between the segmentation mask and the target to be segmented by constructing a semantically driven closed-loop feedback mechanism, thereby ensuring the reliability of the automatic annotation results at the semantic level.

[0079] Specifically, after obtaining the segmentation mask generated by the base segmentation model, it is determined whether the visual region covered by the segmentation mask semantically corresponds to the target to be segmented, rather than being an incorrect segmentation due to appearance similarity, occlusion, or localization deviation. For example, when the target to be segmented is a "white car turning left," if the segmentation mask actually covers a gray car in the adjacent lane, it is considered semantically inconsistent.

[0080] Specifically, the input image, segmentation mask, and initial semantic description of the target to be segmented are input together into the multimodal semantic understanding model. The multimodal semantic understanding model performs a comprehensive analysis of the three and determines whether the segmentation mask and the target to be segmented are consistent in terms of category (e.g., whether they are all "pedestrians"), attributes (e.g., color, size, orientation, etc.), and contextual relationship (e.g., whether they are located in a reasonable position and whether they conform to the scene's behavioral logic), and outputs the verification results.

[0081] In this process, the initial semantic description of the target to be segmented is textual information generated during the target selection stage to characterize the basic identity of the target (e.g., "a large bus in the center of the image"). Its source can be the result of rule-based reasoning or the intermediate output of a multimodal semantic understanding model. A multimodal semantic understanding model is an intelligent model capable of jointly processing visual and linguistic information, possessing cross-modal reasoning and common-sense understanding capabilities.

[0082] The semantic consistency verification process provided in this embodiment not only focuses on the geometric shape or boundary accuracy of the segmented region, but also emphasizes its semantic correctness, thereby effectively avoiding the misacceptance of segmentation results that are "geometrically correct but semantically incorrect".

[0083] Furthermore, if the verification result fails, an iterative correction segmentation process is immediately triggered. This includes adjusting the position or range of the spatial cue, such as shifting the original bounding box to the left, expanding the coverage area, or recalculating the outward expansion ratio. Based on the adjusted spatial cue, the base segmentation model is then called again to generate a new segmentation mask. Subsequently, the semantic consistency verification described above is performed again on the newly generated segmentation mask.

[0084] The iterative correction segmentation process is executed repeatedly until one of the following conditions is met: (1) the segmentation mask passes the semantic consistency verification; (2) the number of iterations reaches the preset maximum number of iterations, such as 3 or 5, which can be configured according to application requirements.

[0085] If the semantic consistency verification is not passed even after the preset maximum number of iterations in the iterative segmentation process, it is determined that there are semantic ambiguities or segmentation difficulties that are difficult to resolve automatically. The corresponding input image and associated segmentation mask are marked as samples requiring manual review, correction, or confirmation by human annotators. These samples are not included in the final structured annotation results to ensure the overall quality and semantic reliability of the automatically constructed dataset.

[0086] In this embodiment, a closed-loop control structure of segmentation → verification → feedback → re-segmentation is formed in the automatic annotation process through semantic consistency verification and iterative correction mechanism. This process takes semantic correctness as the core criterion, which significantly improves the robustness and practicality of zero-sample automatic annotation in open and complex scenarios (such as intersections, dense traffic flow, and severely occluded areas), and provides a highly reliable data foundation for downstream instruction-guided segmentation model training.

[0087] Based on the above embodiments, the following will further describe in detail the process of generating natural language segmentation instructions in step S140.

[0088] The method generates corresponding natural language segmentation instructions based on segmentation masks that meet semantic consistency requirements, including: extracting visual semantic features of the target to be segmented based on segmentation masks that meet semantic consistency requirements; and using a multimodal semantic understanding model to convert the visual semantic features into natural language descriptions that include at least one of appearance attributes, spatial location, and relative relationships, as natural language segmentation instructions.

[0089] It is easy to understand that, after completing the semantic consistency verification and obtaining the segmentation mask that meets the semantic consistency requirements, this embodiment uses the segmentation mask as a reliable visual basis to convert the pixel-level segmentation mask into human-readable and machine-parseable text instructions, ensuring that the language description is strictly aligned with the visual region.

[0090] Specifically, firstly, based on a segmentation mask that meets the semantic consistency requirement, and combined with the corresponding input image, visual semantic features of the target to be segmented are extracted. Among them, visual semantic features are multi-dimensional information that can characterize the semantic identity of the target to be segmented, including but not limited to its appearance attributes such as color, texture, shape, size, and orientation, as well as its contextual information such as its coordinate position in the input image and its spatial orientation relative to other objects.

[0091] Subsequently, the extracted visual semantic features are input into a multimodal semantic understanding model. This model is an intelligent system capable of cross-modal understanding and generation, transforming visual features into natural language descriptions. By fusing multidimensional semantic information, the natural language segmentation instructions generated by the multimodal semantic understanding model accurately identify the target region corresponding to the segmentation mask, avoiding ambiguity caused by vague descriptions.

[0092] It should be noted that the natural language segmentation instructions generated by the multimodal semantic understanding model are not simple category labels (such as cars), but structured semantic descriptions, which contain at least one or more of the following: (1) appearance attributes, such as red car body, worker wearing a safety helmet; (2) spatial location, such as located in the lower left of the image, close to the right lane line; (3) relative relationship, such as in front of the bus, adjacent to the pedestrian crossing; (4) scene context semantics, such as bus stopping at the station, pedestrian crossing the road.

[0093] Furthermore, to enhance the diversity and robustness of labeled data at the linguistic level, this embodiment supports generating multiple semantically equivalent but differently expressed natural language segmentation instructions for the same segmentation mask that meets semantic consistency requirements, based on different linguistic expression strategies. For example, for the same stopped bus, multiple instructions can be generated, such as "yellow bus on the right side of the image," "large bus stopped next to the platform," or "public transportation vehicle near the sidewalk." This diversity design effectively avoids the problem of a single instruction form and significantly enhances the generalization ability of downstream instruction-guided segmentation models or multimodal perception models during training and inference.

[0094] After generating the natural language segmentation instructions, the final results are uniformly encapsulated and output to form structured annotation results. The structured annotation results include at least the following four parts: (1) Input image: the original image data; (2) Spatial cues, such as target bounding boxes, to guide the initial localization information for segmentation; (3) Segmentation mask that meets the semantic consistency requirements, and the verified pixel-level segmentation results; (4) Corresponding natural language segmentation instructions: text descriptions that are semantically consistent with the segmentation mask.

[0095] The components of the structured annotation results are organized in an associative manner and can be stored or transmitted using a unified data format. For example, images can be saved as standard image files (such as JPEG or PNG format), segmentation masks can be saved as binary mask files, and natural language segmentation instructions can be saved as text files, all linked by unique identifiers; alternatively, all information can be encapsulated into a single structured data object (such as JSON format) for easy programmatic reading and processing.

[0096] The structured annotation results can be directly used to construct large-scale instruction-guided target segmentation datasets, or used as training samples for downstream tasks, including but not limited to referential segmentation, visual question answering, multimodal large model alignment training, and semantic understanding module optimization for autonomous driving perception systems.

[0097] In this embodiment, a high-quality mapping from pixel-level segmentation to natural language instructions is achieved by using a segmentation instruction generation and result output mechanism, while ensuring semantic consistency. This provides a technical solution for automatic annotation in open scenarios that combines accuracy, richness, and practicality.

[0098] In some other embodiments, before outputting a structured annotation result containing the input image, segmentation mask, and corresponding natural language segmentation instructions, the process includes: repeatedly performing the processes of generating spatial hints, generating segmentation masks, verifying semantic consistency, and generating natural language segmentation instructions on the same input image to obtain segmentation masks and natural language segmentation instructions corresponding to multiple different target objects, and constructing a structured annotation result containing a variety of target and instruction combinations.

[0099] In this embodiment, before outputting the structured annotation result containing the input image, segmentation mask, and corresponding natural language segmentation instructions, the processes of generating spatial hints, generating segmentation masks, verifying semantic consistency, and generating natural language segmentation instructions are repeatedly executed on the same input image. Each execution processes different target objects in the image, thereby obtaining segmentation masks and natural language segmentation instructions corresponding to multiple different target objects.

[0100] Here, different target objects refer to multiple entities with independent semantic identities existing in the input image, such as vehicles, pedestrians, and traffic signs in a traffic scene. By calling the target selection mechanism multiple times, each target object is selected as the target to be segmented sequentially or in parallel, and a complete segmentation and instruction generation process is completed independently for each target object.

[0101] Finally, all generated segmentation masks and natural language segmentation instructions are associated and integrated with the original input image to construct a structured annotation result containing diverse combinations of targets and instructions. This structured annotation result supports fine-grained annotation of multiple targets in a single image, which can significantly improve the coverage of the dataset and the multi-target understanding ability of downstream models.

[0102] Furthermore, taking intelligent transportation scenarios as an example, Figure 2 This diagram illustrates the overall flow of the instruction-guided automatic construction method for target segmentation data provided in this embodiment of the invention.

[0103] like Figure 2 As shown, taking a traffic scene image or video frame I as input, the system first detects potential target objects in the image, obtaining multiple candidate target regions. Then, in step 1-1, the candidate target regions are scored according to preset rules, and the best one is selected as the target to be segmented. In step 1-2, a corresponding target bounding box b is generated based on the selected target. Next, in step 1-3, the input image and the target bounding box are input into the basic segmentation model to perform zero-shot target segmentation, generating an initial segmentation mask m.

[0104] The generated segmentation mask m is then used in steps 1-4 for semantic consistency verification: the input image I, the segmentation mask m, and the initial semantic description of the target to be segmented are input into the multimodal semantic understanding model to determine whether they are consistent in terms of category, attribute, and context. If the verification result is inconsistent, the process returns to step 1-2 to adjust spatial hints and re-executes segmentation and verification until it passes or reaches the preset maximum number of iterations.

[0105] After successful verification, proceed to step 2, where visual semantic features are extracted based on the segmentation mask that meets semantic consistency requirements, and a multimodal semantic understanding model is used to generate a natural language segmentation instruction t that includes appearance attributes, spatial location, or relative relationships. Subsequently, in step 3, the input image I, the target bounding box b, the segmentation mask m, and the natural language segmentation instruction t are encapsulated into a structured annotation result A and output.

[0106] Ultimately, the structured annotation result A consists of four parts: input image I, target bounding box b, segmentation mask m, and natural language segmentation instructions t, which can be used to construct instruction-guided segmentation datasets or support downstream multimodal perception tasks.

[0107] Figure 3 A schematic diagram illustrating the selection process for the target to be segmented, provided in an embodiment of the present invention, is shown. Figure 3 As shown, starting with an input image or video frame, multiple candidate target regions in the input image are first obtained to form a candidate target set. Subsequently, the candidate targets are evaluated from two dimensions.

[0108] On the one hand, based on the rule-constrained space, a path is constructed, and then visual saliency rules (such as color contrast and edge intensity), target scale rules (such as motion state and interaction intent), spatial location rules (such as relative position and occlusion relationship), and scene semantic rules (such as prioritizing the labeling of pedestrians or vehicles in traffic scenes) are applied in sequence to score the candidate targets according to the rules.

[0109] On the other hand, based on the candidate target feature extraction path, the appearance and scale features, location information and contextual relationships, as well as motion or dynamic state cues of each candidate target are extracted as the basis for scoring.

[0110] The two sets of information are merged in the rule-based scoring calculation stage. After comprehensive scoring, the candidate target with the highest score is selected as the target to be segmented and output. This process realizes a controllable and scalable target selection mechanism based on multi-dimensional rules and features without relying on a predefined category system.

[0111] It should be noted that the candidate target feature extraction path in this embodiment is an optional path. It can also be as in the embodiment above, where each candidate target region is scored based solely on preset rules. No specific limitation is made here.

[0112] Figure 4 This diagram illustrates the semantic consistency verification process between the segmentation mask and the target to be segmented, as provided in an embodiment of the present invention.

[0113] like Figure 4 As shown, semantic consistency verification is initiated after the segmentation mask is generated. If the verification results are inconsistent, the error correction process is triggered, and the error correction strategy selection stage begins. Based on the current state, one of two error correction methods is selected: first, adjust the spatial hints, return to step 1-2 to regenerate the target bounding box, and perform zero-shot segmentation; second, keep the bounding box unchanged, return to step 1-3 to regenerate the segmentation mask.

[0114] Regardless of the strategy used, the semantic consistency verification is performed again after obtaining the segmentation mask. If the verification passes, the process proceeds to step 2 to generate and refine segmentation instructions; if the consistency is still inconsistent, the iteration count is incremented, and it is determined whether the preset maximum number of iterations has been reached.

[0115] If the maximum number of iterations has not been reached, the error correction and verification process continues in a loop. If the maximum number of iterations has been reached and the verification still fails, the current sample is marked as an anomalous sample or a sample awaiting manual review, and the output is paused to ensure the semantic reliability of the final annotation result. This closed-loop mechanism can effectively improve the robustness and accuracy of automatic annotation in complex scenarios.

[0116] Compared with the prior art, the instruction-guided automatic construction method for target segmentation data provided in this embodiment of the invention has the following beneficial technical effects.

[0117] First, no manual annotation or task-specific training is required: This invention completes target segmentation and instruction generation under zero-sample conditions, avoiding reliance on large-scale manually annotated data and targeted model training, and significantly reducing data construction costs. Secondly, it has the ability to verify and correct semantic consistency: by introducing a multimodal large language model to verify the semantic consistency of the segmentation results, and combined with an iterative error correction mechanism, it effectively avoids the inclusion of geometrically correct but semantically incorrect annotation results into the dataset.

[0118] Thirdly, the target selection process is controllable and scalable: the target selection mechanism with rule constraints ensures that the selection of labeled objects is not affected by model bias, and the rule configuration can be flexibly adjusted according to different application needs, thereby improving data diversity and controllability.

[0119] Fourthly, the joint generation of segmentation masks and instructions: This invention can automatically generate natural language instructions with consistent semantics while generating high-quality segmentation masks, and is suitable for tasks such as instruction-guided segmentation, referential segmentation, and multimodal model training.

[0120] Fifthly, it is applicable to complex traffic scenarios and various application environments: the method of this invention does not limit the target category, can adapt to complex and ever-changing traffic scenarios, and can be extended to other general visual scenarios for automatic data annotation and perception systems.

[0121] Corresponding to the instruction-guided automatic construction method for target segmentation data described in the above embodiments, the present invention also provides an instruction-guided automatic construction device for target segmentation data.

[0122] Specifically, Figure 5 A schematic diagram of the structure of the instruction-guided automatic construction device for target segmentation data provided in an embodiment of the present invention is shown.

[0123] like Figure 5 As shown, the device includes: an image acquisition module 510, used to acquire an input image containing at least one target object; a segmentation mask generation module 520, used to generate spatial cues for guiding segmentation based on the input image, and generate a segmentation mask for the target to be segmented based on the spatial cues, wherein the target to be segmented is the selected target object; a semantic verification and correction module 530, used to verify the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correct the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained; and an annotation generation and output module 540, used to generate corresponding natural language segmentation instructions based on the segmentation mask that meets the semantic consistency requirements, and output a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instructions.

[0124] In this embodiment, the image acquisition module 510 acquires an input image containing at least one target object. The segmentation mask generation module 520 generates spatial cues to guide segmentation based on the input image and generates a segmentation mask for the target to be segmented, which is the selected target object. The semantic verification and correction module 530 then verifies the semantic consistency between the segmentation mask and the target to be segmented. If the verification fails, the segmentation process is iteratively corrected until a segmentation mask that meets the semantic consistency requirements is obtained. The annotation generation and output module 540 then generates corresponding natural language segmentation instructions based on the segmentation mask that meets the semantic consistency requirements and outputs a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instructions. This device, by introducing a semantic consistency verification and iterative correction mechanism, effectively ensures accurate alignment between the segmentation mask and the natural language instructions at the semantic level, achieving fully automatic construction of instruction-guided segmentation data without manual intervention. This significantly reduces annotation costs and gives the system good adaptability and scalability in open scenarios.

[0125] It should be noted that the instruction-guided target segmentation data automatic construction device provided in this embodiment of the invention can be referred to in correspondence with the instruction-guided target segmentation data automatic construction method described in the above embodiments, and will not be repeated here.

[0126] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an instruction-guided automatic target segmentation data construction method. This method includes: acquiring an input image containing at least one target object; generating spatial cues for guiding segmentation based on the input image, and generating a segmentation mask for the target to be segmented based on the spatial cues, wherein the target to be segmented is the selected target object; verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained; generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirements, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction.

[0127] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the instruction-guided automatic target segmentation data construction method provided by the above methods. The method includes: acquiring an input image containing at least one target object; generating a spatial cue for guiding segmentation based on the input image, and generating a segmentation mask of a target to be segmented based on the spatial cue, wherein the target to be segmented is a selected target object; verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirement is obtained; generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction.

[0129] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an automatic target segmentation data construction method guided by instructions provided by the methods described above. The method includes: acquiring an input image containing at least one target object; generating a spatial cue for guiding segmentation based on the input image, and generating a segmentation mask for a target to be segmented based on the spatial cue, wherein the target to be segmented is a selected target object; verifying the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correcting the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirement is obtained; generating a corresponding natural language segmentation instruction based on the segmentation mask that meets the semantic consistency requirement, and outputting a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically constructing target segmentation data guided by instructions, characterized in that, include: Obtain an input image that contains at least one target object; Spatial cues for guiding segmentation are generated based on the input image, and a segmentation mask for the target to be segmented is generated based on the spatial cues, wherein the target to be segmented is the selected target object; Verify the semantic consistency between the segmentation mask and the target to be segmented, and iteratively correct the segmentation process if the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained; Based on the segmentation mask that meets the semantic consistency requirement, a corresponding natural language segmentation instruction is generated, and a structured annotation result containing the input image, the segmentation mask, and the corresponding natural language segmentation instruction is output.

2. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, The step of generating spatial cues for guiding segmentation based on the input image includes: The input image is subjected to candidate target detection to obtain multiple candidate target regions; Each candidate target region is scored based on preset rules, and the candidate target region with the highest score is selected as the target to be segmented. Generate corresponding bounding boxes as spatial cues based on the target to be segmented.

3. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, The step of generating a segmentation mask for the target to be segmented based on the spatial hints includes: The input image and the spatial cue are fed into the basic segmentation model, which outputs the segmentation mask of the target to be segmented without needing to be trained for a specific task or target category.

4. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, The step of verifying the semantic consistency between the segmentation mask and the target to be segmented includes: The input image, the segmentation mask, and the initial semantic description of the target to be segmented are input into the multimodal semantic understanding model; The multimodal semantic understanding model determines whether the segmentation mask and the target to be segmented are consistent in terms of category, attribute, and context, and outputs the verification result.

5. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, The iterative correction process for segmentation when verification fails includes: Adjust the position or range of the spatial hints, and regenerate the segmentation mask based on the adjusted spatial hints; The semantic consistency of the newly generated segmentation mask is verified again until the verification passes or the preset maximum number of iterations is reached.

6. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, The step of generating corresponding natural language segmentation instructions based on the segmentation mask that meets the semantic consistency requirement includes: Based on the segmentation mask that meets the semantic consistency requirement, the visual semantic features of the target to be segmented are extracted; The visual semantic features are transformed into a natural language description that includes at least one of appearance attributes, spatial location, and relative relationship using a multimodal semantic understanding model, and serve as the natural language segmentation instruction.

7. The method for automatically constructing target segmentation data guided by instructions according to claim 2, characterized in that, The preset rules include at least one of visual saliency rules, spatial location rules, target scale rules, and scene semantic rules; wherein... The visual saliency rule is used to evaluate the saliency of candidate target regions in the input image; The spatial location rule is used to evaluate whether the candidate target region is located in the central region of the input image or the scene interest region; The target scale rule is used to evaluate whether the area of ​​the candidate target region is within a preset reasonable range; The scene semantic rules are used to evaluate whether the category or attribute of the candidate target region meets the annotation priority conditions under the current scene; The score is obtained by weighted fusion of the scores from each rule.

8. The method for automatically constructing target segmentation data guided by instructions according to claim 5, characterized in that, If the semantic consistency verification is still not passed after the preset maximum number of iterations in the iterative correction segmentation process, the corresponding input image and associated segmentation mask are marked as samples to be manually reviewed and are not included in the final output structured annotation results.

9. The method for automatically constructing target segmentation data guided by instructions according to claim 1, characterized in that, When the method is applied to intelligent transportation scenarios, the target objects include at least one of vehicles, pedestrians, non-motorized vehicles, traffic signs, and road structures; the semantic consistency verification process combines traffic scenario context information to determine whether the segmentation mask conforms to the road topology, driving direction constraints, and the rationality of traffic participant behavior.

10. An instruction-guided automatic construction device for target segmentation data, characterized in that, include: The image acquisition module is used to acquire an input image that contains at least one target object; The segmentation mask generation module is used to generate spatial cues for guiding segmentation based on the input image, and to generate a segmentation mask for the target to be segmented based on the spatial cues, wherein the target to be segmented is the selected target object; The semantic verification and correction module is used to verify the semantic consistency between the segmentation mask and the target to be segmented, and to iteratively correct the segmentation process when the verification fails, until a segmentation mask that meets the semantic consistency requirements is obtained. The annotation generation and output module is used to generate corresponding natural language segmentation instructions based on the segmentation mask that meets the semantic consistency requirements, and output structured annotation results containing the input image, the segmentation mask and the corresponding natural language segmentation instructions.

Images