Training-free generalized reference image segmentation method based on visual and language decoupling mechanism

Abstract

The present application relates to the technical field of computer vision and image processing, and discloses a training-free generalized reference image segmentation method based on a visual and language decoupling mechanism, which comprises the following steps: obtaining an original image and a reference text, extracting entity words as a prompt condition to call a segmentation network to generate an initial mask, combining area proportion and confidence to perform filtering to output a target candidate mask; extracting geometric parameters of the target candidate mask, assigning a unique identifier and serializing and packaging into geometric structured data; superimposing the identifier on the original image to generate a visual rendering image; comparing the rendering image with the structured data, combining global semantic similarity to output a final target mask; when it is determined that there is a missed detection, starting an abnormal error correction process, generating a spatial boundary prompt to re-decode and extract a rescue mask, and outputting after local feature verification. The present application realizes the decoupling of visual space and pure text numerical modal, eliminates the end-to-end joint training process of the model, and reduces the target false negative rate through a self-adaptive re-inference mechanism.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a training-free generalized reference image segmentation method based on a vision-language decoupling mechanism. Background Technology

[0002] Reference image segmentation aims to locate and segment specific targets in an image based on a given natural language text description at the pixel level. This technology has a wide range of applications in fields such as human-computer interaction, autonomous driving, and visual navigation for intelligent robots.

[0003] Existing reference image segmentation methods construct end-to-end multimodal fusion networks, deeply coupling visual features extracted from images with semantic features extracted from text at the network's bottom layer. Subsequently, the decoder outputs the predicted pixel mask. However, this highly coupled network architecture heavily relies on large-scale pixel-level image-text alignment and annotation data for joint training. This training mechanism not only consumes a large amount of computational resources but also limits the model's feature distribution to the training dataset. When facing cross-domain scenarios or needing to segment unseen target categories, the model's generalization ability is limited, and the segmentation accuracy decreases. At the same time, since visual spatial information and textual semantic information are forcibly bound in the bottom-layer network, existing methods cannot directly call or flexibly replace the rapidly developing independent pre-trained visual pedestal models and large language models, resulting in low system modularity and high upgrade and maintenance costs.

[0004] Furthermore, most existing reference image segmentation models employ a one-time forward inference mechanism. When faced with complex image backgrounds or ambiguous natural language references, the model may miss targets during the global feature matching stage. Since existing systems lack anomaly verification mechanisms for intermediate states during the inference process, when the model fails to accurately extract the referential target, the system cannot adaptively narrow the search range for local re-inference, nor can it intercept the generated error mask. Ultimately, this makes it difficult to control the false positive and false negative rates of image segmentation tasks in complex scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a training-free generalized reference image segmentation method based on a vision-language decoupling mechanism. This method solves the problems of existing reference image segmentation methods, which rely on large-scale labeled data for joint training due to the high coupling of image and text features, resulting in limited cross-domain generalization ability, and the lack of adaptive verification and local re-inference error correction mechanisms when target misses occur.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a training-free generalized reference image segmentation method based on a vision-language decoupling mechanism, the method comprising the following steps: The original image and natural language referential text are obtained. The entity word set of the natural language referential text is extracted as the text prompt condition. The generalized zero-shot image segmentation network is called to process the original image to generate an initial candidate mask and the corresponding confidence score. The relative area ratio of the initial candidate mask and the confidence score are combined to perform adaptive filtering and remove nested local masks, and output the target candidate mask. Extract the relative coordinates, aspect ratio, and relative area percentage of the target candidate mask; assign a unique identifier to each target candidate mask and establish an index mapping relationship; serialize and encapsulate the unique identifier, the relative coordinates, the aspect ratio, and the relative area percentage to generate geometrically structured data. Based on the target candidate mask, a visual rendering image with the unique identifier superimposed on the original image is generated. The visual rendering image is compared with the geometric structured data to output the identifier matching result, and the global semantic similarity is calculated. Based on the identifier matching result and the global semantic similarity, the final target mask and empty set are output in combination with the index mapping relationship, or the anomaly correction process is initiated to generate a rescue mask and perform verification output.

[0007] By employing the above technical solution, this invention renders a target candidate mask with a unique identifier on the original image and simultaneously serializes local geometric features into text data, transforming the image segmentation task into a matching judgment between structured text and visually rendered images. This mechanism decouples low-level pixel features from high-level semantic features, enabling the system to directly connect multimodal models to complete the segmentation task while parameters are frozen. At the same time, by combining global semantic similarity with the joint judgment of identifiers, adaptive missed detection identification and error correction output are achieved.

[0008] Furthermore, the step of extracting the set of entity words from the natural language referential text as text prompt conditions includes: A dependency syntax tree is constructed by a word segmenter and a self-attention encoding layer. After identifying and removing demonstrative pronouns and indefinite pronouns, the core noun nodes are extracted. When a modifying compound node is detected in the core noun node, the corresponding modifier is extracted and concatenated with the text characters of the core noun node to generate entity words, which are then summarized to form an entity word set. When the entity word set is empty, the complete text of the natural language referential text or the preset general target prompt is used as the text prompt condition.

[0009] By adopting the above technical solution, text can be parsed using dependency syntax trees, which can eliminate redundant pronouns and extract modification relationships, reduce semantic ambiguity when inputting long texts, and improve the clarity of text prompts. When the entity word set is empty, a preset prompt mechanism can be used to ensure the system's basic detection capability in the absence of explicit entity word input, thereby improving the robustness of the text parsing stage.

[0010] Furthermore, the step of combining the relative area proportion of the initial candidate mask with the confidence score to perform adaptive filtering and remove nested local masks includes: The relative area ratio of the initial candidate mask is divided into different scale intervals, and a higher dynamic confidence lower limit is configured for the scale interval with a larger relative area ratio. The segmented discrimination function is used for judgment, and the masks that meet the conditions are retained and compiled into an intermediate mask set. Extract the spatially overlapping mask pairs from the intermediate mask set, label the mask pairs as small-area masks and large-area masks, and calculate the coverage degree of the small-area mask with respect to the large-area mask; When the inclusion degree is greater than the set spatial overlap limit value, and the confidence score of the small area mask is not greater than the sum of the confidence score of the large area mask and the preset confidence difference margin, the small area mask is deleted from the intermediate mask set.

[0011] By adopting the above technical solution, by setting a dynamic confidence lower limit based on area ratio and assigning different filtering standards according to the target size, the oversegmentation of large-area targets and the undersegmentation of small-area targets can be reduced; combined with the judgment condition of spatial inclusion and confidence difference, redundant masks nested inside the main body can be identified and eliminated, ensuring the spatial independence of candidate masks.

[0012] Furthermore, the step of extracting the relative coordinates, aspect ratio, and relative area percentage of the target candidate mask includes: Calculate the zero-order moment, horizontal first-order moment, and vertical first-order moment of the target candidate mask, and obtain the absolute centroid coordinates containing the horizontal and vertical axis components by calculating the ratio of the horizontal first-order moment to the zero-order moment and the ratio of the vertical first-order moment to the zero-order moment. Traverse the non-zero pixels of the target candidate mask, record the extreme values ​​of all non-zero pixels on the horizontal and vertical axes, and combine the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate, and maximum vertical coordinate to generate the absolute bounding box coordinates. The horizontal and vertical components of the absolute centroid coordinates, as well as the extreme values ​​of the absolute bounding box coordinates, are divided by the global width and global height of the original image to convert them into relative centroid coordinates and relative bounding box coordinates, which are then used as the relative coordinates. The geometric aspect ratio is calculated by dividing the pixel width of the absolute bounding box coordinates by the pixel height, and the relative area percentage is calculated by dividing the zero-order moment by the total pixel area of ​​the original image.

[0013] Using the above technical solution, spatial boundaries are extracted based on moment operations and the extrema of non-zero pixels, and absolute pixel coordinates are converted into relative geometric parameters to eliminate the scale effect caused by the difference in resolution of different input images. The extracted relative coordinates and scale parameters have scale invariance, providing a unified data format for subsequent structured data encapsulation.

[0014] Furthermore, the step of serializing and encapsulating the unique identifier, the relative coordinates, the geometric aspect ratio, and the relative area percentage to generate geometrically structured data includes: Establish a serialized data dictionary structure that includes an identifier field, a relative center coordinate field, a relative bounding box coordinate field, a width-to-height ratio field, and an area ratio field; Convert the corresponding unique identifier, relative coordinates, geometric aspect ratio, and relative area percentage into key-value pairs; The key-value pairs are stored in the serialized data dictionary structure to generate plain text numerical modal data, completing the feature decoupling transformation from visual spatial modality to plain text numerical modality, so as to generate the geometrically structured data.

[0015] By adopting the above technical solution, the geometric parameters of the visual mask are converted into key-value pairs using the data dictionary structure, realizing the modal conversion from local image features to text numerical data. This conversion enables subsequent inference models that support text input to directly parse the spatial distribution features of the target without performing pixel-level feature alignment.

[0016] Furthermore, the step of generating a visual rendering image with the unique identifier superimposed on the original image based on the target candidate mask includes: All the target candidate masks are sorted in descending order according to the relative area ratio to construct a rendering sequence; The pixel matrix of the original image is cloned as a rendering base map. In accordance with the rendering sequence, the boundary contours of the target candidate mask and the center nodes determined based on the absolute centroid coordinates are superimposed layer by layer on the rendering base map. A visual guide line is generated to connect the center node to an offset position outside the absolute bounding box of the corresponding target candidate mask, and the corresponding unique identifier is added to generate the visual rendering map.

[0017] By adopting the above technical solution, the mask contours are superimposed in descending order of area to prevent large-area mask layers from covering small-area mask layers; by generating visual guide lines, the identifiers are set outside the bounding box to avoid text identifiers obscuring the local texture of the image target and to improve the accuracy of subsequent models in reading the correspondence between images and text.

[0018] Furthermore, the steps of outputting the final target mask or empty set based on the identifier matching result and the global semantic similarity, combined with the index mapping relationship, or initiating an error correction process to generate a rescue mask and verifying the output include: When the identifier matching result is not a null value, the corresponding target candidate mask is called as the final target mask output according to the matched unique identifier; When the identifier matching result is a null identifier and the global semantic similarity is greater than a preset existence threshold, it is determined that a target is missed, the anomaly correction process is initiated to generate the rescue mask and perform verification output; When the identifier matching result is an empty identifier, and the global semantic similarity is less than or equal to the preset existence threshold, it is determined that there is no matching entity in the original image, and the empty set is directly output.

[0019] By adopting the above technical solution and combining the identifier matching results with global semantic similarity for joint judgment, it is possible to distinguish between two situations: the original image does not actually contain any entity and the model misses the detection. When the detection is missed, the anomaly correction process is initiated to provide a re-detection path for potential targets, thereby reducing the false negative rate of image segmentation tasks.

[0020] Furthermore, the step of initiating the anomaly correction process to generate the rescue mask includes: The visual language alignment inference network, guided by text instructions, outputs a relative percentage bounding box containing the minimum and maximum relative scale boundaries of the missing target in the horizontal and vertical directions. The relative percentage coordinate frame is subjected to boundary truncation and validity verification. If the verification passes, the relative percentage boundary values ​​of the relative percentage coordinate frame are multiplied by the global width and global height of the original image to generate absolute physical pixel coordinate frames. The absolute physical pixel coordinate box is used as a spatial boundary cue input to the generalized zero-sample image segmentation network, and pixel-level classification decoding is re-performed within the local image region to generate the rescue mask.

[0021] Using the above technical solution, after confirming missed detections, the target bounding box is obtained and used as a spatial boundary cue input to the segmentation network, guiding the network to re-decode and calculate within a limited local area. This process reduces the global feature search to local cue segmentation, which is beneficial for extracting targets missed during the previous global processing.

[0022] Furthermore, after generating the rescue mask, the step of verifying the output includes: Based on the pixel distribution boundary of the rescue mask, a preset proportion of context pixel margin is extended outward to crop out a sub-region of the image from the original image; After performing symmetrical zero-value pixel filling on the edges of the image sub-region and expanding it to standard input resolution, local visual feature vectors are extracted. Extract the text feature vector of the natural language reference text, calculate the local consistency score between the local visual feature vector and the text feature vector of the natural language reference text, and output the rescue mask as the final target mask when the local consistency score is greater than the set security check threshold; otherwise, output the empty set.

[0023] Using the above technical solution, a local image is cropped around the rescue mask and local visual features are extracted. The consistency score between the local image and the text features is calculated for verification. This local verification eliminates the interference of the global background image and can filter out invalid masks generated by error prompts, ensuring the matching degree between the final output result and the referential text.

[0024] Furthermore, the term "training-free" means that when the system is deployed to perform image segmentation operations, the parameters of each network model called are completely frozen. Only the pre-trained feature weights are loaded for forward inference calculation, and no backpropagation parameter updates are performed for the current image segmentation task.

[0025] By adopting the above technical solution, the system is required to perform forward inference in a parameter-frozen state, which eliminates the process of data labeling and model retraining for different sub-scenarios, reduces computing power overhead and improves the efficiency and applicability of the method during deployment.

[0026] This invention provides a training-free generalized reference image segmentation method based on a vision-language decoupling mechanism. It offers the following advantages: 1. This invention achieves feature decoupling between visual spatial modal and pure text numerical modal by overlaying a visual rendering image with a unique identifier onto the original image and serializing and encapsulating geometric parameters such as the absolute centroid coordinates and relative area ratio of the target candidate mask into geometrically structured data. This technical feature enables the system to directly perform matching and judgment between structured text and visual rendering images when the network model parameters are completely frozen, eliminating the need for end-to-end joint training and backpropagation parameter updates for specific image segmentation tasks, thereby reducing the computational cost of system deployment and the dependence on large-scale pixel-level alignment annotation data.

[0027] 2. This invention divides the relative area proportion of the target candidate mask into scale intervals and configures a dynamic confidence lower limit. It combines the spatial inclusion degree and confidence difference between overlapping masks to perform adaptive filtering. This technical feature can assign differentiated interception criteria according to targets of different sizes, identify and remove local redundant masks nested inside the main target, suppress the common phenomenon of oversegmentation of large-area targets and undersegmentation of small-area targets in reference image segmentation, and improve the spatial independence and morphological integrity of the final output mask.

[0028] 3. This invention constructs a missed detection judgment condition by comparing global semantic similarity with identifier matching results. After a missed detection is determined, a relative percentage coordinate box is generated and converted into an absolute physical pixel coordinate box as a local spatial boundary prompt. The box is then re-input into the segmentation network for decoding, and a local consistency score is calculated by cropping a sub-region of the image for verification. This technical feature adds a self-starting local re-detection path in addition to the conventional global prompt segmentation. It can use explicit spatial prompts to guide the network to extract targets that were missed in the early stage. While reducing the overall false alarm rate of the system, it filters the invalid mask generated by the error prompt through the consistency verification of local images, ensuring the accuracy of the error correction result. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the logic flow of the present invention; Figure 3 This is a schematic diagram of the candidate entity segmentation steps of the present invention; Figure 4 This is a schematic diagram of the geometric decoupling steps of the present invention; Figure 5 This is a schematic diagram of the reasoning and error correction steps of the present invention; Figure 6 This is a schematic diagram comparing the performance of the various image segmentation models of the present invention on a multi-scene dataset. Figure 7 This is a schematic diagram showing the comparison of average processing time for a single image according to the present invention; Figure 8 This is a schematic diagram illustrating the trade-off between segmentation performance and processing time in this invention.

[0030] Among them, 10 is the candidate segmentation module; 20 is the geometric decoupling module; and 30 is the inference error correction module. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 The present invention provides a training-free generalized reference image segmentation system based on a vision and language decoupling mechanism. The system includes a candidate segmentation module 10, a geometric decoupling module 20, and an inference error correction module 30.

[0033] Candidate segmentation module 10 calls the generalized zero-shot image segmentation network. In the specific implementation scenario of the present invention, the generalized zero-shot image segmentation network can adopt the SAM series architecture, such as SAM or SAM derivative architecture or similar basic visual segmentation large model, and use the entity word set as text prompt to perform image segmentation on the original image to obtain the initial candidate mask and the confidence score corresponding to the initial candidate mask.

[0034] The candidate segmentation module 10 performs dynamic threshold filtering on the initial candidate mask based on a preset area ratio condition (the preset area ratio condition has a value range of 0.05 to 0.2) and a confidence lower limit condition (the confidence lower limit condition has a value range of 0.1 to 0.7), to obtain intermediate candidate masks. Subsequently, the candidate segmentation module 10 further evaluates the pixel spatial overlap relationship between the intermediate candidate masks, eliminates nested local masks that meet the asymmetric inclusion condition, and finally generates and outputs the target candidate mask.

[0035] The geometric decoupling module 20 receives the target candidate mask, extracts the centroid coordinates and bounding box coordinates of each target candidate mask based on the image moment features, and converts the centroid coordinates and bounding box coordinates into relative coordinates. Combining the bounding box coordinates with the original image size, the geometric decoupling module 20 calculates the aspect ratio and relative area percentage of each target candidate mask.

[0036] The geometric decoupling module 20 converts the extracted relative centroid coordinates, relative bounding box coordinates, aspect ratio, and relative area ratio into a serialized data structure, generating geometrically structured data corresponding to each target candidate mask. The geometric decoupling module 20 converts the mask tensor at the visual pixel level into geometrically structured data at the numerical level and outputs it.

[0037] The inference error correction module 30 is communicatively connected to the geometric decoupling module 20. The inference error correction module 30 acquires the original image, natural language referential text, and geometric structured data, and obtains the index mapping relationship between the target candidate mask and the identifiers in the geometric structured data. The target candidate mask tensor is cached by the candidate segmentation module 10 or the geometric decoupling module 20. The inference error correction module 30 calls the corresponding target candidate mask according to the index mapping relationship between the target candidate mask and the identifiers in the geometric structured data. The inference error correction module 30 draws the boundary contour and identifiers on the original image in descending order of the pixel area of ​​the target candidate mask, and generates a visual rendering image.

[0038] The inference and error correction module 30 utilizes a visual language alignment inference network to perform attribute matching based on the visual rendering image and geometrically structured data, and outputs the identifier matching result. When the identifier matching result is not empty, the inference and error correction module 30 indexes the corresponding target candidate mask in the target candidate mask set according to the identifier matching result, and outputs the target candidate mask as the final pixel-level target mask. At the same time, the inference and error correction module 30 uses a graph-text contrast ratio network to calculate the global semantic similarity between the natural language referential text and the original image.

[0039] When the identifier matching result is empty and the global semantic similarity is greater than the preset existence threshold (the preset existence threshold ranges from 0.2 to 0.3), the inference error correction module 30 triggers the abnormal error correction process. When the identifier matching result is empty and the global semantic similarity is less than or equal to the preset existence threshold, the inference error correction module 30 determines that there is no target entity in the original image that corresponds to the natural language referential text and outputs an empty set.

[0040] In the error correction process, the inference error correction module 30 calls the visual language alignment inference network to output the relative coordinate prompt box of the target position. The relative coordinate prompt box is represented in the format of a quadruple. The quadruple includes the minimum horizontal relative coordinate, the minimum vertical relative coordinate, the maximum horizontal relative coordinate, and the maximum vertical relative coordinate. The value range of each relative coordinate is from 0 to 1.

[0041] For relative coordinate values ​​that are outside the range, the inference and error correction module 30 performs boundary truncation processing; for relative coordinate prompt boxes that do not meet the condition that the minimum horizontal relative coordinate is less than the maximum horizontal relative coordinate, or do not meet the condition that the minimum vertical relative coordinate is less than the maximum vertical relative coordinate, the inference and error correction module 30 determines that the relative coordinate prompt box is an invalid prompt box and outputs an empty set, and the inference and error correction module 30 converts the relative coordinate prompt box into a physical pixel coordinate box.

[0042] The inference and error correction module 30 uses physical pixel coordinate boxes as location cues to drive the generalized zero-shot image segmentation network to perform local resegmentation on the original image to obtain the rescue mask. The inference and error correction module 30 extracts the local image region covered by the rescue mask and calculates the local semantic similarity between the local image region and the natural language reference text.

[0043] The inference correction module 30 determines the validity of the rescue mask based on local semantic similarity. When the local semantic similarity meets the preset verification conditions, and the value range of the preset verification conditions is 0.25 to 0.35, the inference correction module 30 outputs the rescue mask as the final pixel-level target mask. When the local semantic similarity does not meet the preset verification conditions, the inference correction module 30 outputs an empty set.

[0044] See attached document Figure 2 This invention provides a training-free generalized reference image segmentation method based on a vision-language decoupling mechanism, the method comprising the following steps: S100, the candidate segmentation module 10 acquires natural language reference text and the original image, performs dependency parsing on the natural language reference text to extract core noun phrases to construct an entity word set, uses the entity word set as a text prompt condition to call the generalized zero-shot image segmentation network to process the original image, generates an initial candidate mask set and a corresponding confidence score set, combines the area ratio of the initial candidate mask in the original image with the confidence score, and performs adaptive filtering to obtain an intermediate mask set through the set segmentation threshold conditions. The area ratio threshold value ranges from 0.05 to 0.2, and the lower limit of the confidence score value ranges from 0.1 to 0.7. Then, the inclusion degree of small area masks relative to large area masks in the intermediate mask set is evaluated. When the inclusion degree reaches the nesting condition and the difference of the corresponding confidence scores is less than the preset margin, the nested local masks are removed. The preset margin ranges from 0.05 to 0.15. The target candidate mask is output to the geometric decoupling module 20. S200, the geometric decoupling module 20 reads the target candidate mask, calculates the centroid coordinates and bounding box coordinates of each target candidate mask using two-dimensional image moments, and converts the two into relative coordinates. Based on the bounding box coordinates and the original image size, it calculates the geometric aspect ratio and relative area ratio of each target candidate mask, assigns a unique identifier to each target candidate mask, and establishes an index mapping relationship between the unique identifier and the target candidate mask tensor. The extracted relative centroid coordinates, relative bounding box coordinates, geometric aspect ratio, and relative area percentage are combined into a serialized key-value pair structure, thereby converting the spatial features corresponding to the mask tensor at the visual pixel level into geometrically structured data at the numerical level, and outputting the geometrically structured data and index mapping relationship to the inference and error correction module 30. S300, the inference and error correction module 30 acquires the original image, natural language referential text, and geometric structured data, and obtains the index mapping relationship between the target candidate mask and the unique identifier. Based on the target candidate mask, it generates a visual rendering image on the original image; it calls the visual language alignment inference network to compare the visual rendering image with the geometric structured data and outputs the identifier matching result. At the same time, it calls the image-text contrast measurement network to calculate the global semantic similarity between the natural language referential text and the original image. Based on the identification matching result and the judgment of global semantic similarity, the inference error correction module 30 calls the corresponding target candidate mask as the final pixel-level target mask output, or directly outputs an empty set, or starts the abnormal error correction process when the target is missed. In the anomaly correction process, the inference correction module 30 calls the visual language alignment inference network to output the location coordinates of the missing target, maps the location coordinates to absolute physical coordinates, and guides the generalized zero-sample image segmentation network to perform local resegmentation to generate a rescue mask; performs validity verification on the rescue mask and calculates the local semantic consistency score; outputs the rescue mask when the local consistency verification is passed, otherwise outputs an empty set.

[0045] See attached document Figure 3 In the training-free generalized reference image segmentation method based on vision and language decoupling mechanism provided by this invention, the candidate entity segmentation step may include the following sub-steps: S110, the candidate segmentation module 10 acquires the natural language reference text and the original image. For the part-of-speech analysis and dependency syntax tree construction of the natural language reference text, this embodiment adopts a basic text parsing network that includes a word segmenter and a self-attention encoding layer. Before deployment, the basic text parsing network is pre-trained offline using a large-scale general language corpus and a masked language modeling task to obtain accurate lexical dependency extraction capabilities.

[0046] Natural language text is transformed into a word sequence by a word segmenter and then input into an attention-based encoding layer. The output is a tree-like network data with part-of-speech tags and dependency connections. After obtaining the syntactic tree, the candidate segmentation module 10 traverses the nodes, identifies and removes demonstrative pronouns and indefinite pronouns to reduce the participation of words without physical entity references in subsequent feature matching operations.

[0047] Candidate segmentation module 10 detects whether there are modifying compound nodes in the core noun node. If there are compound nodes, candidate segmentation module 10 extracts the corresponding modifiers and concatenates them with the text characters of the core noun to generate entity words. Candidate segmentation module 10 sums all the parsed entity words and constructs an entity word set.

[0048] When the entity word set is empty, the candidate segmentation module 10 uses the complete text of the natural language referent as the text prompt condition; or it uses a preset general target prompt, such as inputting all objects or everything, etc., to generate a candidate mask set, and the reasoning and error correction module 30 combines the complete natural language referent to perform subsequent matching judgment.

[0049] S120, the candidate segmentation module 10 converts the entity word set into the text prompt format specified by the generalized zero-shot image segmentation network. The candidate segmentation module 10 can concatenate multiple entity words in the entity word set into continuous text for input using a preset connector, or it can traverse the entity word set and input each entity word independently. The generalized zero-shot image segmentation network can specifically adopt the SAM series architecture, such as SAM or a SAM derivative version or a general visual segmentation architecture with the same origin as SAM.

[0050] The training-free mechanism of this invention refers to the fact that when the system is deployed to a specific new scene or downstream image segmentation task, all network models are in a parameter-frozen state. Inference can be directly performed without collecting labeled data for the current task for secondary fine-tuning training. To achieve the training-free mechanism, the parameters of all network layers are frozen when the generalized zero-shot image segmentation network is deployed, and the pre-trained general visual feature weights are directly loaded. In order to enable the generalized zero-shot image segmentation network to have the zero-shot generalization capability without training, the generalized zero-shot image segmentation network is fully pre-trained offline before being connected to this system.

[0051] The method for creating and pre-training a generalized zero-shot image segmentation network includes: First, constructing a general visual segmentation dataset containing hundreds of millions of natural scene images and corresponding high-quality class-independent masks; second, constructing a cue generator model composed of points, bounding boxes, and text; during the training phase, the image encoder of the generalized zero-shot image segmentation network is input into the image encoder to extract features, and the synthesized cue is input into the cue encoder. Finally, the network parameters are iteratively updated using the backpropagation algorithm by calculating the joint loss function composed of the cross-entropy loss and the Dice loss between the predicted mask output by the mask decoder and the ground truth labeled mask, until the model converges. The generalized zero-shot image segmentation network prepared in this way, such as the generalized SAM series architecture pre-trained as described above, can be directly applied without fine-tuning for downstream tasks.

[0052] The network structure of the generalized zero-shot image segmentation network includes an image encoder, a cue encoder, and a mask decoder. The original image is converted into a three-dimensional pixel matrix with height, width, and color channels and input to the image encoder to extract the global image feature tensor. At the same time, the set of entity words in text cue format is input to the cue encoder and converted into a cue feature tensor. The candidate segmentation module 10 sets a fixed small value as the initial confidence threshold, and the value range of the initial confidence threshold is set between 0.01 and 0.05.

[0053] The aforementioned initial confidence threshold setting prompts the visual model to retain pixel features in the edge matching state. The global image feature tensor, the cue feature tensor, and the initial confidence threshold are input together into the mask decoder for feature fusion decoding. The full feature segmentation operation is performed on the original image. The candidate segmentation module 10 receives the initial candidate mask set output by the generalized zero-shot image segmentation network and simultaneously receives the corresponding confidence score set. The confidence score includes at least one of the mask quality prediction score output by the generalized zero-shot image segmentation network, the target existence probability, and the matching score between the text cue and the mask region, or is obtained by normalizing and weighting the above at least two scores.

[0054] S130, the candidate segmentation module 10 extracts the length and width dimensions of the pixel matrix of the original image, calculates the total pixel area of ​​the original image, and calculates the pixel area of ​​each mask in the initial candidate mask set. Regarding the statistical principle of the mask pixel area, the candidate segmentation module 10 obtains the actual physical area by calculating the sum of non-zero pixels in the two-dimensional tensor corresponding to a single mask. By calculating the ratio of the pixel area of ​​a single mask to the total pixel area of ​​the original image, the candidate segmentation module 10 obtains the relative area ratio of each mask.

[0055] Large visual targets possess texture features and are assigned high confidence by the model, while small targets often have lower confidence scores due to their limited pixel information. If a single standard of confidence is used for filtering, small targets may be missed or too much large background noise may be introduced. Therefore, the candidate segmentation module 10 establishes a dynamic threshold determination mechanism, dividing the relative area proportion into different scale intervals and configuring corresponding dynamic confidence lower limits for different scale intervals. The larger the relative area proportion of the interval, the higher the dynamic confidence lower limit value configured for the corresponding scale interval. The candidate segmentation module 10 uses a segmented discriminant function to perform adaptive filtering on the initial candidate mask set.

[0056] The calculation logic of the dynamic threshold discrimination function is as follows: ; In the formula, To adaptively preserve the discriminant function; For the first One initial candidate mask; The sequence index number of the initial candidate mask; For the first The relative area percentage of each initial candidate mask; The first area percentage threshold is set; For the first Confidence scores of the initial candidate masks; This is the first confidence level lower bound set; This is the set second area percentage threshold; This is the set second confidence level lower limit; This is the set third confidence level lower limit; Other determination scenarios that do not meet any of the above preset conditions.

[0057] In practical engineering applications, the first area proportion threshold is set to 0.05, the second area proportion threshold is set to 0.2, the corresponding first confidence lower limit can be set to 0.1, the second confidence lower limit to 0.3, and the third confidence lower limit to 0.7. The above threshold parameters can be statistically adjusted according to the average size distribution of the target object in the specific application scenario.

[0058] Candidate segmentation module 10 intercepts the mask with a judgment result of 1 and assembles the mask with a judgment result of 1 into an intermediate mask set.

[0059] S140, during the feature decoding stage, the visual segmentation model often outputs a multi-layered nested mask structure for the same physical entity. For example, the same physical entity generates both a large-area mask containing the complete outline and a small-area mask containing only local highlighted parts. The topological structure of the intermediate mask set contains local and global overlaps triggered by the same physical entity. The candidate segmentation module 10 determines the pixel spatial overlap relationship between each mask in the intermediate mask set. The candidate segmentation module 10 extracts the mask pairs that have spatial overlap and marks the small-area mask and the large-area mask respectively. The candidate segmentation module 10 calculates the mask coverage of the small-area mask relative to the large-area mask.

[0060] The formula for calculating coverage is as follows: ; In the formula, The coverage of a small-area mask relative to a large-area mask; This is a function to calculate the area of ​​the input pixel set by finding the total number of non-zero pixels. This refers to the set of small-area mask pixels in overlapping mask pairs; It is the set of large-area mask pixels in the overlapping mask pairs.

[0061] To prevent local structures with high confidence from being incorrectly eliminated by large-area background masks, the candidate segmentation module 10 sets asymmetric elimination conditions based on the conventional inclusion degree judgment. The candidate segmentation module 10 obtains the difference in confidence scores between the small-area mask and the large-area mask and constructs a Boolean logic expression for elimination.

[0062] The calculation formula for asymmetric elimination logic is as follows: ; In the formula, The function for eliminating small area masks triggers a discard instruction when the output logic is true. This refers to the set of small-area mask pixels in overlapping mask pairs; The coverage of a small-area mask relative to a large-area mask; This is the set spatial overlap limit value; Confidence score for small area mask; Confidence score for large area mask; The set confidence level difference margin.

[0063] Candidate segmentation module 10 deletes the corresponding small area mask from the intermediate mask set if and only if the inclusion degree is greater than the spatial overlap limit value and the confidence score of the small area mask is not greater than the sum of the confidence score of the large area mask and the confidence difference margin. After traversing the entire intermediate mask set for topological determination and filtering, candidate segmentation module 10 outputs the retained target candidate mask and transmits the retained target candidate mask to geometric decoupling module 20.

[0064] See attached document Figure 4 In the training-free generalized reference image segmentation method based on vision and language decoupling mechanism provided by this invention, the geometric decoupling step may include the following sub-steps: S210, the geometric decoupling module 20 receives the target candidate mask output by the candidate segmentation module 10. The target candidate mask is stored in the computer memory as a two-dimensional pixel tensor data structure. Since the visual language alignment inference network directly processes the original two-dimensional pixel tensor, it will consume a lot of computing resources. The geometric decoupling module 20 reduces the dimension of the high-dimensional two-dimensional pixel tensor to a low-dimensional mathematical coordinate.

[0065] The geometric decoupling module 20 uses a two-dimensional image moment algorithm to process the target candidate mask and extract the spatial distribution features of the target candidate mask. In one embodiment of the present invention, the zeroth moment in the two-dimensional image moment represents the total number of effective pixels of the target candidate mask, and the horizontal first moment and the vertical first moment represent the quality distribution state of the target candidate mask on the horizontal axis and the vertical axis, respectively.

[0066] The absolute centroid coordinates that accurately reflect the center position of the target subject can be obtained by calculating the ratio of the first moment to the zero moment. The geometric decoupling module 20 calculates the zero moment and the first moment of the target candidate mask, and then derives the absolute centroid coordinates of the target candidate mask. The horizontal and vertical components of the absolute centroid coordinates are divided by the global pixel width and global pixel height of the original image, respectively, and converted into relative centroid coordinates.

[0067] The calculation logic for image moments and centroid coordinates is as follows: ; ; ; ; ; In the formula, Let be the zeroth moment of the candidate mask for the target; The horizontal coordinate index in the two-dimensional pixel tensor; The index of the ordinate in the two-dimensional pixel tensor; The global pixel width of the original image; The global pixel height of the original image; The coordinate position in the two-dimensional pixel tensor is Pixel values; Let be the first-order transverse moment of the target candidate mask; The first-order vertical moment of the target candidate mask; The horizontal component of the centroid coordinates of the target candidate mask; The vertical component of the centroid coordinates of the target candidate mask.

[0068] S220, the geometric decoupling module 20 extracts the bounding box of the target candidate mask. The geometric decoupling module 20 traverses the non-zero pixels of the target candidate mask, records the extreme values ​​of all non-zero pixels on the horizontal and vertical axes, and combines the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate, and maximum vertical coordinate to generate the absolute bounding box coordinates. The extreme values ​​of the absolute bounding box coordinates are divided by the global pixel width and global pixel height of the original image to convert them into relative bounding box coordinates. The traversal and extreme value recording of non-zero pixels can be done using existing image processing algorithms.

[0069] The geometric decoupling module 20 calculates the morphological and scale features of the target candidate mask by combining the coordinates of the absolute bounding box. The morphological features are represented by the geometric aspect ratio, and the scale features are represented by the relative area ratio. The geometric decoupling module 20 obtains the global width and global height of the original image and performs algebraic operations to generate morphological and scale features. In order to avoid the division by zero error caused by the horizontally elongated mask with a single pixel height, the geometric decoupling module 20 adds a very small anti-zero constant to the denominator of the geometric aspect ratio calculation formula to ensure the closed loop of the algorithm logic.

[0070] The calculation logic for geometric features is as follows: ; ; ; ; In the formula, The pixel width of the absolute outer bounding box; The maximum x-coordinate of the set of non-zero pixels in the target candidate mask; The minimum x-coordinate of the set of non-zero pixels in the candidate mask for the target; The pixel height of the absolute outer bounding box; The maximum ordinate of the set of non-zero pixels in the target candidate mask; The minimum ordinate of the set of non-zero pixels in the candidate mask for the target; The aspect ratio of the target candidate mask; To avoid the collapse of the denominator when divided by zero; a very small constant; The relative area percentage of the target candidate mask; Let be the zeroth moment of the candidate mask for the target; The global pixel width of the original image; This represents the global pixel height of the original image.

[0071] S230, the geometric decoupling module 20 performs symbolic sequence encapsulation of visual data, establishes a unified serialized data dictionary structure, and assigns a unique digital identifier to each target candidate mask.

[0072] The geometric decoupling module 20 maps the numeric identifier, the horizontal component of the relative centroid coordinate, the vertical component of the relative centroid coordinate, the coordinates of the relative bounding box, the geometric aspect ratio, the relative area ratio, and the target candidate mask reference identifier into key-value pairs in the serialized data dictionary structure. The serialized data dictionary structure contains an identifier field, a relative center coordinate field, a relative bounding box coordinate field, a aspect ratio field, and an area ratio field. The key-value pairs are stored in a general plain text format that the visual language aligned inference network will parse.

[0073] Through serialization encapsulation, the geometric decoupling module 20 does not directly input the underlying visual pixel tensor into the visual language alignment inference network. Instead, it establishes an index mapping relationship between the target candidate mask tensor and the unique numerical identifier, and converts the spatial features of the target candidate mask into plain text numerical modal data. This completes the feature decoupling transformation from visual spatial modality to plain text numerical modality. The plain text numerical modal data generated by serialization encapsulation constitutes geometrically structured data.

[0074] The application of geometrically structured data reduces the computational load of the visual language alignment inference network in the multimodal alignment stage and avoids spatial position evaluation bias when the visual language alignment inference network directly processes complex image background pixels. The geometric decoupling module 20 transmits the generated geometrically structured data to the inference error correction module 30.

[0075] See attached document Figure 5 In the training-free generalized reference image segmentation method based on vision and language decoupling mechanism provided by this invention, the inference error correction step may include the following sub-steps: S310, the inference and error correction module 30 acquires the original image, natural language reference text, and geometric structured data output by the geometric decoupling module 20. The inference and error correction module 30 extracts the relative area ratio of each target candidate mask in the geometric structured data. The inference and error correction module 30 sorts all target candidate masks in descending order according to the relative area ratio and constructs a rendering sequence. Multi-target overlap is a common situation in the field of image segmentation. The descending order of area from large to small can ensure that large-area targets are drawn first and small targets are drawn later, thereby avoiding the small target entities being covered by the rendering layer of large-area background targets.

[0076] The inference and error correction module 30 performs the drawing operation of visual elements layer by layer on the original image according to the rendering sequence. Through the rendering sequence control mechanism of prioritizing the drawing of large-area masks and drawing small-area masks later, the inference and error correction module 30 avoids the visual features of small targets being covered by large overlapping areas.

[0077] For the specific drawing of visual elements, existing image rendering technology can be used. The inference and error correction module 30 first clones the pixel matrix of the original image as the rendering base map, and then superimposes the boundary contour and center node of the corresponding target candidate mask on the rendering base map, and generates visual guide lines to smoothly connect the center node to the offset position outside the absolute outer bounding box. At this point, the numerical identifier of the corresponding target candidate mask is added to generate the visual rendering map.

[0078] S320, the inference error correction module 30 calls the visual language alignment inference network. In order to maintain the training-free features of the system, all network parameters of the visual language alignment inference network are frozen and not updated. In this embodiment, the visual language alignment inference network consists of a visual encoder, a cross-modal alignment projection layer and an autoregressive language decoder.

[0079] To ensure the system's ability to accurately understand the correspondence between spatial visual elements and text semantics, the visual language alignment inference network underwent the following data creation and offline training phases before being integrated into the system: collecting massive image-text correspondence datasets on an internet scale; constructing multimodal instruction fine-tuning tasks using image-text pairs, extracting features from images through a visual encoder and converting them into visual labels through an alignment projection layer, which, along with natural language text labels, are input into an autoregressive language decoder; and using an autoregressive cross-entropy loss function to update the gradients of the decoder and projection layer parameters, enabling it to learn the ability to infer from visual information and generate logical text labels or coordinate bounding boxes.

[0080] The inference and error correction module 30 performs scaling and normalization preprocessing on the visual rendering image, uniformly adjusting the visual rendering image to a fixed-resolution three-dimensional pixel tensor required by the visual encoder. The adjusted visual rendering image is input into the visual encoder and converted into a visual feature sequence. The inference and error correction module 30 concatenates natural language referential text, geometric structured data, and preset task query instructions. The preset task query instructions are, for example: "Based on the provided visual rendering image and geometric structured data, find the target entity that best matches the text description and output only its corresponding numeric identifier." The concatenated complete text is converted into a prompt word sequence after word segmentation.

[0081] The visual feature sequence and the prompt word sequence interact in the autoregressive language decoder. The visual language alignment inference network outputs a numeric identifier that matches the semantics of the natural language referential text. The output format of the numeric identifier includes a single numeric identifier, multiple numeric identifiers, or null value identifiers. When the numeric identifier output by the visual language alignment inference network does not exist in the index mapping relationship of the target candidate mask, the inference error correction module 30 determines that the output is invalid and treats it as a null value identifier.

[0082] When multiple numeric identifiers are output, the inference error correction module 30 determines the final matching identifier based on at least one of the matching confidence, target candidate mask area, and local semantic consistency score output by the visual language alignment inference network. For example, by default, the numeric identifier with the highest matching confidence or the highest local semantic consistency score is selected as the final matching identifier, or the target candidate masks corresponding to multiple numeric identifiers are merged and output. When the visual language alignment inference network determines that there is no matching physical target entity in the visual rendering image, the visual language alignment inference network outputs a null value identifier.

[0083] Meanwhile, the inference and error correction module 30 calls the image-text contrast measurement network, which is also a model component with pre-prepared and frozen weights. Its model creation and offline training process includes: collecting a large-scale sample set containing paired image-text descriptions, constructing a dual-tower network structure composed of independent text encoding branches and independent image encoding branches; using contrastive learning algorithms, such as the InfoNCE loss function, calculating the distance between positive sample image-text pairs and negative sample image-text pairs in the shared high-dimensional mapping space in the training batch, and iteratively optimizing the dual-tower network parameters by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, thereby obtaining the ability to extract consistent features by directly comparing the underlying semantics of images and text without inference.

[0084] The reasoning and error correction module 30 uses the text encoding branch to extract the fixed-dimensional text feature vector of the natural language referent text, and uses the image encoding branch to extract the fixed-dimensional global visual feature vector of the original image. The reasoning and error correction module 30 calculates the cosine similarity between the text feature vector and the global visual feature vector.

[0085] The formula for calculating cosine similarity is as follows: ; In the formula, The cosine similarity between the global visual feature vector and the text feature vector; Text feature vectors that represent text in natural language; This is the global visual feature vector of the original image.

[0086] The existence threshold is used to measure the baseline probability of the objective existence of a target entity in the original image. The existence threshold is set between 0.2 and 0.3. When the visual language alignment inference network outputs a non-empty numeric identifier, the inference error correction module 30 calls the corresponding target candidate mask according to the numeric identifier and outputs the target candidate mask as the final pixel-level target mask.

[0087] When the visual language alignment inference network outputs a null value and the cosine similarity is greater than the set existence threshold, the inference error correction module 30 determines that the target candidate mask set has missed a target detection and triggers the abnormal error correction process. When the visual language alignment inference network outputs a null value and the cosine similarity is less than or equal to the set existence threshold, the inference error correction module 30 determines that there is no target entity in the original image that matches the natural language referential text and outputs an empty set.

[0088] S330, in the anomaly correction process, the inference correction module 30 guides the visual language alignment inference network to output the estimated coordinates of the missing target through text instructions. The visual language alignment inference network outputs a relative percentage coordinate frame. The relative percentage coordinate frame contains the minimum and maximum relative proportion boundaries of the missing target in the horizontal and vertical directions. The relative percentage coordinate frame is represented in a quadruple format as the minimum horizontal relative proportion boundary, the minimum vertical relative proportion boundary, the maximum horizontal relative proportion boundary, and the maximum vertical relative proportion boundary.

[0089] The inference and error correction module 30 performs validity checks on the relative percentage coordinate frame. When any relative proportion boundary value exceeds the range of 0 to 1, boundary truncation is performed. When the minimum horizontal relative proportion boundary is greater than or equal to the maximum horizontal relative proportion boundary, or the minimum vertical relative proportion boundary is greater than or equal to the maximum vertical relative proportion boundary, the inference and error correction module 30 determines that the relative percentage coordinate frame is invalid and outputs an empty set.

[0090] The inference and error correction module 30 multiplies each relative percentage boundary value of the relative percentage coordinate box by the global width and global height of the original image, respectively, to generate an absolute physical pixel coordinate box. The inference and error correction module 30 uses the absolute physical pixel coordinate box as a spatial boundary cue input to the generalized zero-shot image segmentation network. The cue encoder of the generalized zero-shot image segmentation network receives the absolute physical pixel coordinate box and re-executes pixel-level foreground and background binary classification decoding operations within the local image region defined by the absolute physical pixel coordinate box to generate a rescue mask.

[0091] S340, the visual language alignment inference network outputs erroneous predicted boundaries that deviate from actual physical coordinates due to its autoregressive generation mechanism. The inference error correction module 30 performs local semantic consistency verification on the rescue mask to intercept erroneously generated data. When the rescue mask is empty, the pixel area of ​​the rescue mask is less than the preset minimum pixel area threshold, or the width or height of the outer bounding box of the rescue mask is zero, the inference error correction module 30 determines that the rescue mask is invalid and outputs an empty set.

[0092] The inference error correction module 30 expands the context pixel margin by a preset ratio based on the pixel distribution boundary of the rescue mask, and then crops out the image sub-region containing the rescue mask coverage area and the local background from the original image. The preset ratio ranges from 10% to 20%.

[0093] To address the extreme case where the size of the cropped image sub-region is too small, causing matrix dimension collapse in visual feature extraction, the inference correction module 30 performs symmetrical zero-value pixel padding around the edges of the image sub-region to expand it to the standard input resolution specified by the image-text contrast ratio network. The inference correction module 30 then inputs the expanded and padded image sub-region into the image encoding branch of the image-text contrast ratio network to extract local visual feature vectors. Finally, the inference correction module 30 calculates the local consistency score between the local visual feature vectors and the text feature vectors.

[0094] The formula for calculating the local consistency score is as follows: ; In the formula, The local consistency score between local visual feature vectors and text feature vectors; Text feature vectors that represent text in natural language; This is the local visual feature vector of the cropped image sub-region.

[0095] The inference error correction module 30 compares the local consistency score with the set security check threshold. The security check threshold is used to determine whether the locally cropped image and the natural language text meet the minimum requirements for semantic alignment. The value range of the security check threshold is set to 0.25 to 0.35. When the local consistency score is greater than the set security check threshold, the inference error correction module 30 confirms that the rescue mask has semantic correctness and outputs the rescue mask as the final pixel-level target mask of the system. When the local consistency score is less than or equal to the set security check threshold, the inference error correction module 30 determines that the rescue mask is an invalid position offset result, refuses to output the rescue mask data, and directly outputs an empty set.

[0096] To aid in understanding the technical solution of this invention, a specific application example in a warehousing and logistics scenario is provided below.

[0097] Using visual guidance tasks in a warehousing and logistics sorting scenario, explain the data flow and processing of each module in the system.

[0098] The input natural language reference text is: a cardboard box with a damaged label placed on the upper left side of the tray.

[0099] The input original image is a partial top view of the warehouse, including wooden pallets, multiple cardboard boxes of the same size, plastic film, and scattered labels.

[0100] The candidate segmentation module 10 acquires natural language referential text, performs dependency parsing, extracts core noun phrases such as "cardboard box" and "label", constructs an entity word set by combining modifiers, converts the entity word set into text prompts, calls a generalized zero-shot image segmentation network to process the original image, and generates an initial candidate mask set.

[0101] The candidate segmentation module 10 extracts the relative area ratio of each mask, filters out pallet background masks with an area ratio greater than 0.8 and film reflective noise masks with a confidence level lower than the set lower limit. During the inclusion evaluation stage, the system detects that the label mask is contained inside the carton mask. Since the confidence level of the label mask does not exceed the sum of the confidence level of the carton mask and the set preset margin, the candidate segmentation module 10 determines that the label mask meets the nesting condition. Based on this, the local mask where the label mask is located is removed, and the retained carton target candidate mask is output to the geometric decoupling module 20.

[0102] The geometric decoupling module 20 reads the received target candidate mask, calculates the zeroth and first moments, derives the absolute centroid coordinates and bounding box coordinates of each target candidate mask, and, combined with the original image size, converts the absolute centroid coordinates and bounding box coordinates into relative coordinates. It also calculates the geometric aspect ratio and relative area percentage of each target candidate mask. The geometric decoupling module 20 assigns a numeric identifier to each target candidate mask, combines the numeric identifier with spatial geometric features into plain text key-value pairs, generates geometrically structured data, and establishes an index mapping relationship between the numeric identifier and the mask tensor memory address.

[0103] The inference and error correction module 30 receives the original image and geometrically structured data. According to the descending order of relative area proportion, it draws the boundary contours of each target candidate mask and the corresponding numerical identifiers on the pixel matrix copy of the original image to generate a visual rendering image. The inference and error correction module 30 calls the visual language alignment inference network to compare the visual rendering image and the geometrically structured data. When there is a situation where the label damage feature on the cardboard box is blurred, causing the visual language alignment inference network to output an empty set of identifiers, the inference and error correction module 30 calls the image-text contrast ratio network to calculate the global semantic similarity between the natural language referential text and the original image.

[0104] When the global semantic similarity is 0.31, which is greater than the preset threshold of 0.25, the inference error correction module 30 triggers the abnormal error correction process. The inference error correction module 30 guides the visual language to align the relative coordinate prompt box in the upper left area of ​​the output tray of the inference network through text instructions. After converting the relative coordinate prompt box into a physical pixel coordinate box, it is input into the prompt encoder of the generalized zero-shot image segmentation network. The generalized zero-shot image segmentation network performs re-segmentation in the corresponding local area to generate a rescue mask. The inference error correction module 30 extracts the local image of the area corresponding to the rescue mask and calculates the local consistency score. The local consistency score is 0.28, which is greater than the set safety inspection threshold of 0.25. Finally, the inference error correction module 30 outputs the rescue mask as a pixel-level target mask.

[0105] Table 1: Performance Comparison Test Data of Image Segmentation Models on Multi-Scene Association Dataset According to Table 1 and Figures 6 to 8As can be seen, the average intersection-union ratio (IUR) of this scheme is higher than the existing training-free model baseline on three multi-scene referential datasets. On the RefCOCOg dataset containing complex syntactic structures, the average IUR of this scheme reaches 59.53%, which is 10.88% higher than the GroundingDINO+SAM architecture and 5.74% higher than the ReSAM architecture. The performance improvement indicates that the geometric decoupling module 20 reduces the dimensionality of visual pixel modalities and maps them to geometrically structured data of pure text modalities, thereby reducing the computational load of the visual-language alignment inference network in the cross-modal feature fusion stage and avoiding spatial location evaluation bias caused by redundancy of underlying pixel information.

[0106] Compared with supervised learning architectures, this scheme achieves an average intersection-union ratio of 67.21% on the RefCOCO test set, surpassing earlier VLT models that relied on a large amount of labeled data. The performance under zero-shot conditions depends on the anomaly correction process.

[0107] The effectiveness of the error correction mechanism in the supervised learning architecture is marked as - because such end-to-end models directly fit pixel features through network weights and lack explicit multi-stage anomaly detection and post-processing branches, making this metric inapplicable to them. In contrast, the existing training-free baseline models have an effectiveness of 0 because they follow a unidirectional open-loop inference logic. Once the pre-detection module fails to capture the target, the system directly outputs an empty set, lacking the ability to capture and recover from closed-loop anomalies. In comparison, the effectiveness of the error correction mechanism in this solution is 41.17%, indicating that the inference error correction module 30 can recover missed targets by guiding the re-segmentation through local region coordinates when the identifier matching result is an empty set.

[0108] In terms of the average processing time per single image, the measured value of this scheme is 192.8 milliseconds. The increase in time is due to the multi-stage serial computation nodes consisting of candidate segmentation, rendering alignment, and local re-segmentation. However, under the premise of the training-free mechanism, this scheme abandons the time and hardware cost generated by model fine-tuning and can meet the requirements of reference image segmentation in the context of unlabeled data.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A training-free generalized reference image segmentation method based on a vision-language decoupling mechanism, characterized in that, The method includes the following steps: The original image and natural language referential text are obtained. The entity word set of the natural language referential text is extracted as the text prompt condition. The generalized zero-shot image segmentation network is called to process the original image to generate an initial candidate mask and the corresponding confidence score. The relative area ratio of the initial candidate mask and the confidence score are combined to perform adaptive filtering and remove nested local masks, and output the target candidate mask. Extract the relative coordinates, aspect ratio, and relative area percentage of the target candidate mask; assign a unique identifier to each target candidate mask and establish an index mapping relationship; serialize and encapsulate the unique identifier, the relative coordinates, the aspect ratio, and the relative area percentage to generate geometrically structured data. Based on the target candidate mask, a visual rendering image with the unique identifier superimposed on the original image is generated. The visual rendering image is compared with the geometric structured data to output the identifier matching result, and the global semantic similarity is calculated. Based on the identifier matching result and the global semantic similarity, the final target mask and empty set are output in combination with the index mapping relationship, or the anomaly correction process is initiated to generate a rescue mask and perform verification output.

2. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The step of extracting the set of entity words that refer to natural language text as text prompt conditions includes: A dependency syntax tree is constructed by a word segmenter and a self-attention encoding layer. After identifying and removing demonstrative pronouns and indefinite pronouns, the core noun nodes are extracted. When a modifying compound node is detected in the core noun node, the corresponding modifier is extracted and concatenated with the text characters of the core noun node to generate entity words, which are then summarized to form an entity word set. When the entity word set is empty, the complete text of the natural language referential text or the preset general target prompt is used as the text prompt condition.

3. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The step of combining the relative area proportion of the initial candidate mask with the confidence score to perform adaptive filtering and remove nested local masks includes: The relative area ratio of the initial candidate mask is divided into different scale intervals, and a higher dynamic confidence lower limit is configured for the scale interval with a larger relative area ratio. The segmented discrimination function is used for judgment, and the masks that meet the conditions are retained and compiled into an intermediate mask set. Extract the spatially overlapping mask pairs from the intermediate mask set, label the mask pairs as small-area masks and large-area masks, and calculate the coverage degree of the small-area mask with respect to the large-area mask; When the inclusion degree is greater than the set spatial overlap limit value, and the confidence score of the small area mask is not greater than the sum of the confidence score of the large area mask and the preset confidence difference margin, the small area mask is deleted from the intermediate mask set.

4. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The steps of extracting the relative coordinates, aspect ratio, and relative area percentage of the target candidate mask include: Calculate the zero-order moment, horizontal first-order moment, and vertical first-order moment of the target candidate mask, and obtain the absolute centroid coordinates containing the horizontal and vertical axis components by calculating the ratio of the horizontal first-order moment to the zero-order moment and the ratio of the vertical first-order moment to the zero-order moment. Traverse the non-zero pixels of the target candidate mask, record the extreme values ​​of all non-zero pixels on the horizontal and vertical axes, and combine the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate, and maximum vertical coordinate to generate the absolute bounding box coordinates. The horizontal and vertical components of the absolute centroid coordinates, as well as the extreme values ​​of the absolute bounding box coordinates, are divided by the global width and global height of the original image to convert them into relative centroid coordinates and relative bounding box coordinates, which are then used as the relative coordinates. The geometric aspect ratio is calculated by dividing the pixel width of the absolute bounding box coordinates by the pixel height, and the relative area percentage is calculated by dividing the zero-order moment by the total pixel area of ​​the original image.

5. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The step of serializing and encapsulating the unique identifier, the relative coordinates, the geometric aspect ratio, and the relative area percentage to generate geometrically structured data includes: Establish a serialized data dictionary structure that includes an identifier field, a relative center coordinate field, a relative bounding box coordinate field, a width-to-height ratio field, and an area ratio field; Convert the corresponding unique identifier, relative coordinates, geometric aspect ratio, and relative area percentage into key-value pairs; The key-value pairs are stored in the serialized data dictionary structure to generate plain text numerical modal data, completing the feature decoupling transformation from visual spatial modality to plain text numerical modality, so as to generate the geometrically structured data.

6. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 4, characterized in that, The step of generating a visual rendering image with the unique identifier superimposed on the original image based on the target candidate mask includes: All the target candidate masks are sorted in descending order according to the relative area ratio to construct a rendering sequence; The pixel matrix of the original image is cloned as a rendering base map. In accordance with the rendering sequence, the boundary contours of the target candidate mask and the center nodes determined based on the absolute centroid coordinates are superimposed layer by layer on the rendering base map. A visual guide line is generated to connect the center node to an offset position outside the absolute bounding box of the corresponding target candidate mask, and the corresponding unique identifier is added to generate the visual rendering map.

7. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The steps of outputting the final target mask or empty set based on the identifier matching result and the global semantic similarity, combined with the index mapping relationship, or initiating an anomaly correction process to generate a rescue mask and verifying the output include: When the identifier matching result is not a null value, the corresponding target candidate mask is called as the final target mask output according to the matched unique identifier; When the identifier matching result is a null identifier and the global semantic similarity is greater than a preset existence threshold, it is determined that a target is missed, the anomaly correction process is initiated to generate the rescue mask and perform verification output; When the identifier matching result is an empty identifier, and the global semantic similarity is less than or equal to the preset existence threshold, it is determined that there is no matching entity in the original image, and the empty set is directly output.

8. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 7, characterized in that, The step of initiating the error correction process to generate the rescue mask includes: The visual language alignment inference network, guided by text instructions, outputs a relative percentage bounding box containing the minimum and maximum relative scale boundaries of the missing target in the horizontal and vertical directions. The relative percentage coordinate frame is subjected to boundary truncation and validity verification. If the verification passes, the relative percentage boundary values ​​of the relative percentage coordinate frame are multiplied by the global width and global height of the original image to generate absolute physical pixel coordinate frames. The absolute physical pixel coordinate box is used as a spatial boundary cue input to the generalized zero-sample image segmentation network, and pixel-level classification decoding is re-performed within the local image region to generate the rescue mask.

9. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 8, characterized in that, After generating the rescue mask, the step of performing verification output includes: Based on the pixel distribution boundary of the rescue mask, a preset proportion of context pixel margin is extended outward to crop out a sub-region of the image from the original image; After performing symmetrical zero-value pixel filling on the edges of the image sub-region and expanding it to standard input resolution, local visual feature vectors are extracted. Extract the text feature vector of the natural language reference text, calculate the local consistency score between the local visual feature vector and the text feature vector of the natural language reference text, and output the rescue mask as the final target mask when the local consistency score is greater than the set security check threshold; otherwise, output the empty set.

10. The training-free generalized reference image segmentation method based on vision and language decoupling mechanism according to claim 1, characterized in that, The term "no training required" refers to: When the system is deployed to perform image segmentation operations, the parameters of each network model called are completely frozen. Only the pre-trained feature weights are loaded for forward inference calculation, and no backpropagation parameter updates are performed for the current image segmentation task.

Images