An image instance segmentation method, system, device and medium
By fusing features from a semantically activated visual cue encoder with YOLO and DinoV2 models, the problem of high recall and high accuracy in image instance segmentation in complex scenes is solved, achieving efficient object detection and instance segmentation, and adapting to new tasks without retraining.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN INSTITUTE OF ENGINEERING
- Filing Date
- 2026-03-23
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to simultaneously meet the requirements of high recall and high accuracy in complex scenarios such as cluttered backgrounds, diverse object appearances, and occlusion. They suffer from numerous erroneous responses and fragmented candidate regions, resulting in high noise and low accuracy in regional global feature matching.
A semantically activated visual cue encoder is used to generate semantically enhanced cue features, which are fused with multi-scale feature maps of the YOLO backbone network. The intermediate layer details and local block token features of the candidate target segmentation region are extracted by the DinoV2 model. Feature enhancement is performed through deformable convolution and channel-space dual attention mechanism. Similarity matching is performed by multi-scale fused features and reference features. The candidate region generation guided by visual cues and the network fine screening are combined to achieve the final image instance segmentation.
It improves matching accuracy in complex scenarios such as occlusion, changing viewpoints, and cluttered backgrounds, effectively suppresses cluttered backgrounds and fragmented erroneous proposals, and ensures high recall and high accuracy in image instance segmentation.
Smart Images
Figure CN122156638A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and specifically to an image instance segmentation method, system, device, and medium. Background Technology
[0002] In the field of computer vision, object detection and instance segmentation are core foundational tasks for understanding image content, and zero-shot segmentation of unseen objects has good versatility for project deployment and implementation. Early zero-shot segmentation methods for unseen objects mainly fall into two categories: embedding alignment-based methods and generative model-based methods. The former attempts to map visual features and semantic features to the same space for matching, but often suffers from performance degradation due to the semantic gap; the latter synthesizes virtual visual features of unseen categories through generative models to train a classifier, but the quality and diversity of synthesized features are key challenges.
[0003] Existing technologies, which generate initial object detection bounding boxes and segmentation masks using prompts such as points, boxes, and text descriptions, only target the extraction of objects of a specific class. Accurate object IDs require post-processing for determination. Recent zero-shot object detection instance segmentation methods typically obtain an initial candidate mask set through SAM (Simplified Image Acquisition), and then calculate the similarity between the original image texture corresponding to each candidate mask and the DinoV2 features of the reference image of the target set to determine the final instance segmentation result. Current processing still suffers from the following problems: It is difficult to simultaneously achieve both recall and precision. Issues such as cluttered backgrounds, differences in appearance between similar objects, and occlusion in the target image lead to numerous erroneous responses and fragmented incorrect proposals in the matching results, limiting accuracy. Similarly, issues like blurred appearances and occlusion introduce noise and ambiguity into the matching of dense texture features in candidate regions. Directly performing cross-image matching of DinoV2 global features at the region level faces significant noise interference, making it difficult to detect the desired results and resulting in low recall.
[0004] In summary, in complex scenarios such as cluttered backgrounds, significant differences in object appearances, and occlusion, existing methods are prone to generating a large number of erroneous responses and fragmented candidate regions. Furthermore, regional-level global feature matching suffers from high matching noise and low accuracy due to the loss of spatial information, making it difficult to simultaneously meet the application requirements of high recall and high accuracy. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and propose an image instance segmentation method, system, device and medium to solve the problem that the prior art is unable to simultaneously meet the application requirements of high recall and high accuracy.
[0006] A first aspect of the present invention provides an image instance segmentation method, comprising the following steps: Acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested; The visual texture cue image is encoded into semantically enhanced cue features by a semantically activated visual cue encoder; multi-scale feature maps of the original image are extracted based on the YOLO backbone network; the cue features and multi-scale feature maps are fused to generate candidate target segmentation regions and their corresponding foreground texture images; The DinoV2 model is used to extract intermediate layer detail features, local block token features, and global category token features from the foreground texture image corresponding to the candidate target segmentation region. Deformable convolution and channel-space dual attention mechanisms are used to enhance the intermediate layer detail features and local block token features to generate enhanced local features. The enhanced local features are then added element-wise with the upsampled global category token features to obtain multi-scale fusion features. Finally, the multi-scale fusion features are compared with pre-stored reference features of the target object to obtain a joint similarity score. Candidate target segmentation regions are classified according to joint similarity scores to select the final image instance segmentation results.
[0007] Furthermore, the step of encoding the visual texture cue image into semantically enhanced cue features through the semantically activated visual cue encoder specifically includes the following steps: Use a pre-trained backbone network to extract multi-scale feature maps from the input image; Encode the set of visual cue images into a cue mask with the same size as the feature map; By fusing image semantic features with cue masks through an attention mechanism, semantic regions related to the cue are activated, generating semantically enhanced cue features.
[0008] Furthermore, the process of acquiring the visual texture cue image corresponding to the target object to be tested includes the following steps: Collect multiple texture images of each target object from different perspectives; After removing the background from the texture image, the effective texture image is scaled to a fixed size. Multiple scaled effective texture images are stitched together into a complete visual cue image, and the bounding box position of each texture image in the complete image and its corresponding category label are recorded as the visual texture cue image of the target object to be tested.
[0009] Furthermore, the cue features are fused with multi-scale feature maps through channel stitching, and the YOLO segmentation detection head performs target detection and segmentation on the fused features to generate candidate target segmentation regions.
[0010] Furthermore, the step of performing similarity matching between the multi-scale fused features and the pre-stored reference features specifically includes the following steps: The multi-scale fused features and the reference features of each of the target objects to be measured are respectively L2 normalized; The normalized feature pairs are concatenated along the channel dimension to form a joint feature representation; The joint features are input into the perspective proximity discrimination module, which consists of a series of multi-layer convolutional neural networks and a fully connected classifier. The output probability of perspective proximity is used as a similarity score.
[0011] Furthermore, the calculation process of the joint similarity score specifically includes the following steps: For each candidate target segmentation region, calculate the average of the k highest similarity scores as the absolute similarity score; The absolute similarity score is converted into a relative similarity score using the softmax function, and is expressed as follows: ; in, Sᵣ ( c The score represents the relative similarity. c For the target category, S a ( c The score represents the absolute similarity. τ For temperature control coefficient, Nc The number of target objects to be detected; The absolute similarity score and the relative similarity score are weighted and fused to obtain the joint similarity score: ; in, To combine similarity scores, β ∈[0,1] represents the weighting coefficient.
[0012] Furthermore, the step of classifying candidate target segmentation regions based on joint similarity scores to select the final image instance segmentation result specifically includes the following steps: The candidate regions that were not matched after the initial screening are combined to generate the first foreground image; The first foreground image is subjected to image enhancement processing, which includes edge sharpening perturbation and tone space perturbation; The enhanced image is then used to guide the target detection model and multi-scale feature matching network to perform secondary detection and matching through pre-trained visual cues. Non-maximum suppression based on the cross-union ratio of masked regions and similarity scores is applied to the two detection results to output the final instance segmentation result.
[0013] A second aspect of the present invention provides an image instance segmentation system, comprising: The acquisition module is used to acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested; The detection module is used to encode visual texture cue images into semantically enhanced cue features through a semantically activated visual cue encoder; extract multi-scale feature maps of the original image based on the YOLO backbone network; and fuse the cue features with the multi-scale feature maps to generate candidate target segmentation regions and their corresponding foreground texture images. The matching module uses the DinoV2 model to extract intermediate layer detail features, local block token features, and global category token features from the foreground texture image corresponding to the candidate target segmentation region. It then uses deformable convolution and channel-space dual attention mechanisms to enhance the intermediate layer detail features and local block token features, generating enhanced local features. The enhanced local features are then element-wise added to the upsampled global category token features to obtain multi-scale fusion features. Finally, the multi-scale fusion features are matched with pre-stored reference features of the target object to obtain a joint similarity score. The segmentation module is used to classify candidate target segmentation regions based on joint similarity scores in order to select the final image instance segmentation results.
[0014] A third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.
[0015] A fourth aspect of the present invention provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0016] This invention provides an image instance segmentation method, which has the following beneficial effects: This invention uses a complete visual cue image along with its bounding box and category identifier as the visual texture cue image. A semantically activated visual cue encoder generates semantically enhanced cue features, which are then fused with multi-scale feature maps of the original image extracted by the YOLO backbone network. This mechanism allows the detection model to directly utilize visual appearance information to guide target localization, eliminating the need for text descriptions or manually labeled points and boxes, thus avoiding performance loss due to the text-image semantic gap and improving detection efficiency. Simultaneously, it extracts intermediate layer detail features, local block token features, and global category token features from the foreground texture image using DinoV2, and employs deformable convolution and channel-space dual attention mechanisms for feature enhancement. Finally, the enhanced local features are element-wise added to the upsampled global features to obtain multi-scale fused features rich in multi-scale information. This fully utilizes the powerful general representation capabilities of DinoV2 while solving the matching distortion problem caused by spatial information loss in traditional global features, significantly improving matching accuracy in complex scenes such as occlusion, viewpoint changes, and cluttered backgrounds. This method simultaneously achieves target detection and instance segmentation within the same framework, compatible with both tasks without structural adjustments. Meanwhile, the combination of candidate region generation guided by visual cues and fine screening by a multi-scale feature matching network effectively suppresses cluttered backgrounds and fragmented erroneous proposals, achieving high recall while ensuring high accuracy. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the image instance segmentation method in an embodiment of the present invention; Figure 2 This is a structural diagram of the YOLO-based prompting and guidance target detection model in an embodiment of the present invention; Figure 3 This is a flowchart of the DinoV2 local and global feature multi-scale fusion matching network processing in an embodiment of the present invention; Figure 4 This is a schematic diagram of a densely stacked instance segmentation scenario in an embodiment of the present invention; Figure 5 This is a schematic diagram of a daily life target instance segmentation scenario in an embodiment of the present invention. Detailed Implementation
[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0019] This invention proposes an image instance segmentation method. First, a visual cue-guided object detection model based on the YOLO framework and a DinoV2 local and global feature multi-scale fusion texture similarity matching network are trained. Then, the trained models are used to perform the object detection and segmentation process sequentially, including the following steps: S1. Acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested; wherein, collect multiple texture images of each target object to be tested from different perspectives; after removing the background of the texture images, scale the effective texture images to a fixed size; stitch the scaled multiple effective texture images into a complete visual cue image, and record the bounding box position of each texture image in the complete image and the corresponding category label, so as to serve as the visual texture cue image of the target object to be tested.
[0020] S2. Visual cue-guided object detection model based on the YOLO framework.
[0021] The structure of the YOLO-based prompt-guided object detection model is as follows: Figure 2 As shown, this model extracts potential candidate target segmentation regions based on visual cue features, and mainly consists of a pre-trained backbone network, a semantic activation visual cue encoder (SAVPE), a feature fusion module, and a YOLO segmentation detection head. This approach only requires collecting visual texture cues from multiple target objects to guide the target detection model in extracting potential candidate target regions from the inspected image.
[0022] 1. Semantic Activation Visual Cueing Encoder (SAVPE) The SAVPE module is responsible for fusing user-provided visual cues and image semantic information to generate semantically meaningful cue features. This module includes the following steps:
[0023] a. Input: Original image and set of visual cue texture maps of the target object to be inspected; b. Image feature extraction: Use a pre-trained backbone network to extract multi-scale feature maps of the image; c. Cue Encoding: Encode the set of visual cues into a cue mask of the same size as the image feature map; d. Semantic activation: By using an attention mechanism, the semantic features of the image are fused with the cue mask to activate the semantic regions related to the cue and generate semantically enhanced cue features; 2. YOLO-based segmentation detection head-guided target detection and segmentation By combining the semantic cue features generated by SAVPE with the YOLO detection framework, cue-guided object detection is achieved. The YOLO architecture is modified to accept additional cue feature input. The process is as follows:
[0024] a. Feature Fusion: Channel splicing and fusion of the prompt features output by SAVPE and the image features extracted by the YOLO backbone network.
[0025] b. Segmentation Detection Head: The YOLO segmentation detection head is used for target detection based on fused features. Guided by cues, the detector will focus more on regions relevant to the cues, thereby improving the detection and segmentation accuracy of specific targets.
[0026] S3, DinoV2 local and global feature multi-scale fusion texture similarity matching network.
[0027] The texture similarity detection model based on DinoV2 significantly improves the accuracy of texture similarity detection by fusing global generalization features and local patch features from the visual Transformer. It is particularly suitable for handling texture matching problems under conditions of viewpoint changes, noise, and occlusion. Furthermore, the similarity detection and viewpoint proximity discrimination based on local patch tokens leverage the ability of patch tokens to obtain local texture features and preserve spatial information, avoiding the distortion of matching scores between multiple candidate textures and reference images caused by the loss of local spatial information in global pooling clstokens.
[0028] The flowchart of the DinoV2 local and global feature multi-scale fusion matching network processing is as follows: Figure 3 As shown: 1. Input image pair standardization: Image pairs (Image A is the query image to be detected, and Image B is a reference image of the target to be detected prepared in advance) are uniformly scaled to a fixed resolution of 224*224 and channel normalized.
[0029] 2. DinoV2 multi-scale feature extraction: Freeze the weight parameters of the DinoV2 model and extract the three levels of semantic visual features for the standardized input image pairs: The first feature layer is the output feature of the intermediate layer of DinoV2, used to capture high-resolution, detailed local edges, corners, and primary texture primitives; the second feature layer is the local patch feature x_norm_patchtokens output of the last layer of DinoV2, used to capture local structure combinations, which is the core of local texture representation; the third feature layer is the class-level cls feature x_norm_clstokens output of the last layer of DinoV2, used to encode the global semantic content and overall structure of the image, and is robust to spatial transformations.
[0030] 3. Multi-scale feature fusion module: a. Deformable Convolutional Network: Deformable convolution is applied to the first and second feature layers. The sampling points of the convolution kernel can be dynamically shifted according to the image content, thereby adaptively aligning with irregular or distorted texture patterns and overcoming simple geometric deformations.
[0031] b. Channel-spatial dual attention: After applying the channel attention module and the spatial attention module to the deformable volume results respectively, the spatial and channel features are stitched together.
[0032] c. Top-down semantic guidance: After restoring the two-dimensional structure of the global semantic features extracted from the third feature layer, upsampling is performed, and the spliced spatial and channel features are added element-wise to obtain the multi-scale fusion feature F_fused.
[0033] 4. Viewpoint proximity detection module: a. Perform L2 normalization on the multi-scale fusion features F_fused_A and F_fused_B of the image pair respectively, and then concatenate them in the channel dimension to form a joint feature representation.
[0034] b. Input the joint feature into a cascaded multilayer convolutional neural network (CNN) and a fully connected classifier (MLP), and output the probability of "similar viewpoints". This value is the final similarity score of the image pair.
[0035] S4, Two-level visual cue target segmentation process.
[0036] 1. Prepare texture similarity reference information offline: Collect a varying number of texture images for each target object to be detected. After removing the background, scale the effective textures to a fixed size of 224*224 according to the length and width ratios as a reference image for texture similarity detection of a single object, and input them into DinoV2 to extract the multi-scale fusion feature vector Refs_fused.
[0037] 2. Visual cue feature extraction: a) Select several texture maps from the reference image of each object and arrange them neatly to form a complete visual cue image of the target set to be inspected. Record the effective bounding box of each small texture map and assign a category code to all bounding boxes starting from 0. Fill the background of the complete visual cue image with a solid color.
[0038] b. Input the complete image of the visual cue, the bounding box, and the corresponding category code into the visual cue-guided target detection model based on the YOLO framework to obtain the complete features of all visual cues in the target set to be detected and save them in the model for later use.
[0039] 3. After acquiring the image to be inspected, perform preliminary target detection and segmentation using a target detection model with stored visual cue features; 4. Perform cross-category deduplication on test results: All category codes for object detection are set to 0. Maximum suppression (NMS) is performed based on the detection score and the IOU overlap of the detection bounding box. The filtered results are used as potential candidate object segmentation regions. All effective foreground segments are combined to generate the first foreground map.
[0040] 5. For each candidate target segmentation region: after scaling the effective texture to 224*224 size, input it into the DinoV2 local and global feature multi-scale fusion matching network to obtain the fused feature Cur_fused. The similarity score between this feature vector and all feature vectors Refs_fused in the target set is calculated through the perspective proximity discrimination module inside it.
[0041] 6. Target category ID prediction and allocation: For each object in the target set to be inspected, calculate the average of its top k similarity scores and denote it as the absolute similarity score. 'c' represents the object category, and the relative similarity score is calculated using the following softmax formula. : ; in >0 represents the temperature control coefficient controlling the sharpness of the distribution. It is the number of objects in the target set to be inspected, and then the joint similarity score is calculated. : ; in, ∈[0,1] is the absolute score Relative score The weights between them.
[0042] Set a threshold t1, when Targets exceeding this threshold are considered valid targets and are assigned a category code c. Their corresponding mask and valid texture are preserved, and all valid segmented textures are erased from the first foreground image to obtain the second foreground image.
[0043] 7. Enhancement of the second foreground image: a. Edge sharpening perturbation (for image datasets that are originally blurry and have low contrast, moderate sharpening can improve the detection rate of small objects or objects with blurred boundaries).
[0044] b. Hue color space perturbation (changing the hue in the HSV color space to improve the success rate of target detection by changing the color space distribution of the image to make the appearance closer to the visual cues).
[0045] 8. Perform the same visual cue-guided target detection and segmentation as in step 3 on the enhanced second foreground image, and then repeat steps 4 to 6.
[0046] 9. Summarize the two instance segmentation results, perform maximum suppression (NMS) on the masks under the same target ID based on the IOU overlap and similarity score of the mask region, and filter out the final instance segmentation result of the present invention, so as to prevent a part of the object with more complex texture from being incorrectly stripped.
[0047] Experimental verification: like Figure 4 As shown, this is a scenario of densely stacked instance segmentation. Figure 4 (a) Two sets of visual cues for the target to be inspected, generated based on 3D model rendering, with each target ID providing visual cues from 8 different perspectives; Figure 4 (b) The first foreground image extracted after the initial target detection and segmentation of the image to be inspected based on visual cues, with irrelevant areas such as the desktop and walls excluded; Figure 4 (c) Visual cues for multi-level segmentation and DinoV2 local and global feature matching, resulting in the final stacked target instance segmentation.
[0048] like Figure 5 As shown, this is a scenario for segmenting everyday target instances. Figure 5 (a) A set of visual cues for multiple targets to be inspected collected offline, with each target ID providing visual cues from multiple different perspectives; Figure 5 (b) Initial target detection is performed on the image to be inspected based on visual cues. The boxes in the figure represent possible target areas, and the remaining areas that are significantly different from the features of the visual cues are excluded. Figure 5 (c) After performing similarity matching on the proposed target regions extracted in the first step using DinoV2 features, the final target detection and instance segmentation results actually needed are determined.
[0049] The beneficial effects of this invention are as follows: The core of this invention lies in leveraging the powerful generalization ability of a pre-trained large-scale visual basic model to achieve a cross-image feature matching paradigm through direct image feature cues, thereby enabling rapid task adaptation without training.
[0050] Task uniformity: The same framework can simultaneously meet the needs of two tasks, object detection (providing bounding boxes) and instance segmentation (providing pixel-level masks), without structural adjustments.
[0051] High efficiency and low cost: The method of this invention utilizes the potential of the small detector of the YOLO architecture in open recognition capabilities, and the detection algorithm runs with high efficiency; when encountering a new unseen object instance segmentation task, only a few reference images need to be collected, avoiding the expensive process of collecting data and training the model again for each new task, which greatly improves the speed of model deployment and application.
[0052] High recall and high precision: Direct image feature cues for cross-image matching avoid excessive false responses and fragmented false proposals; the DinoV2 local feature similarity discrimination model ensures the accuracy of region-level target feature matching; and the two-level visual cueing target segmentation process effectively guarantees the robustness of object segmentation.
[0053] Based on the above inventive concept, this invention proposes an image instance segmentation system, comprising: The acquisition module is used to acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested.
[0054] The detection module is used to encode visual texture cue images into semantically enhanced cue features through a semantically activated visual cue encoder; extract multi-scale feature maps of the original image based on the YOLO backbone network; and fuse the cue features with the multi-scale feature maps to generate candidate target segmentation regions and their corresponding foreground texture images.
[0055] The matching module is used to extract intermediate layer detail features, local block token features, and global category token features from the foreground texture image corresponding to the candidate target segmentation region using the DinoV2 model. The intermediate layer detail features and local block token features are enhanced by deformable convolution and channel-space dual attention mechanism to generate enhanced local features. The enhanced local features are added element-wise with the upsampled global category token features to obtain multi-scale fusion features. The multi-scale fusion features are then matched with the pre-stored reference features of the target object to obtain a joint similarity score.
[0056] The segmentation module is used to classify candidate target segmentation regions based on joint similarity scores in order to select the final image instance segmentation results.
[0057] The present invention also proposes an image instance segmentation computer device, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image instance segmentation method.
[0058] The present invention also proposes a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, are used to perform the steps of an image instance segmentation method.
[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. An image instance segmentation method, characterized in that, include: Acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested; The visual texture cue image is encoded into semantically enhanced cue features by a semantically activated visual cue encoder; multi-scale feature maps of the original image are extracted based on the YOLO backbone network; the cue features and multi-scale feature maps are fused to generate candidate target segmentation regions and their corresponding foreground texture images; The DinoV2 model is used to extract intermediate layer detail features, local block token features, and global category token features from the foreground texture image corresponding to the candidate target segmentation region; We employ deformable convolution and channel-space dual attention mechanisms to enhance intermediate layer detail features and local block token features, thereby generating enhanced local features. The enhanced local features are added element-wise to the upsampled global category token features to obtain multi-scale fusion features; the multi-scale fusion features are then matched with the pre-stored reference features of the target object to obtain a joint similarity score. Candidate target segmentation regions are classified according to joint similarity scores to select the final image instance segmentation results.
2. The method according to claim 1, characterized in that, The process of encoding visual texture cue images into semantically enhanced cue features using a semantically activated visual cue encoder specifically includes the following steps: Use a pre-trained backbone network to extract multi-scale feature maps from the input image; Encode the set of visual cue images into a cue mask with the same size as the feature map; By fusing image semantic features with cue masks through an attention mechanism, semantic regions related to the cue are activated, generating semantically enhanced cue features.
3. The method according to claim 1, characterized in that, The process of acquiring the visual texture cue image corresponding to the target object to be tested includes the following steps: Collect multiple texture images of each target object from different perspectives; After removing the background from the texture image, the effective texture image is scaled to a fixed size. Multiple scaled effective texture images are stitched together into a complete visual cue image, and the bounding box position of each texture image in the complete image and its corresponding category label are recorded as the visual texture cue image of the target object to be tested.
4. The method according to claim 1, characterized in that, The cue features are fused with multi-scale feature maps by channel stitching, and the YOLO segmentation detection head performs target detection and segmentation on the fused features to generate candidate target segmentation regions.
5. The method according to claim 1, characterized in that, The process of performing similarity matching between the multi-scale fused features and pre-stored reference features specifically includes the following steps: The multi-scale fused features and the reference features of each of the target objects to be measured are respectively L2 normalized; The normalized feature pairs are concatenated along the channel dimension to form a joint feature representation; The joint features are input into the perspective proximity discrimination module, which consists of a series of multi-layer convolutional neural networks and a fully connected classifier. The output probability of perspective proximity is used as a similarity score.
6. The method according to claim 5, characterized in that, The calculation process of the joint similarity score specifically includes the following steps: For each candidate target segmentation region, calculate the average of the k highest similarity scores as the absolute similarity score; The absolute similarity score is converted into a relative similarity score using the softmax function, and is expressed as follows: ； in, Sᵣ ( c The score represents the relative similarity. c For the target category, S a ( c The score represents the absolute similarity. τ For temperature control coefficient, Nc The number of target objects to be detected; The absolute similarity score and the relative similarity score are weighted and fused to obtain the joint similarity score: ； in, To combine similarity scores, β ∈[0,1] represents the weighting coefficient.
7. The method according to claim 1, characterized in that, The step of classifying candidate target segmentation regions based on joint similarity scores to select the final image instance segmentation result specifically includes the following steps: The candidate regions that were not matched after the initial screening are combined to generate the first foreground image; The first foreground image is subjected to image enhancement processing, which includes edge sharpening perturbation and tone space perturbation; The enhanced image is then used to guide the target detection model and multi-scale feature matching network to perform secondary detection and matching through pre-trained visual cues. Non-maximum suppression based on the cross-union ratio of masked regions and similarity scores is applied to the two detection results to output the final instance segmentation result.
8. An image instance segmentation system, characterized in that, include: The acquisition module is used to acquire the original image of the scene to be tested and the visual texture cue image corresponding to the target object to be tested; The detection module is used to encode visual texture cue images into semantically enhanced cue features through a semantically activated visual cue encoder; extract multi-scale feature maps of the original image based on the YOLO backbone network; and fuse the cue features with the multi-scale feature maps to generate candidate target segmentation regions and their corresponding foreground texture images. The matching module is used to extract intermediate layer detail features, local block token features, and global category token features from the foreground texture image corresponding to the candidate target segmentation region using the DinoV2 model; We employ deformable convolution and channel-space dual attention mechanisms to enhance intermediate layer detail features and local block token features, thereby generating enhanced local features. The enhanced local features are added element-wise to the upsampled global category token features to obtain multi-scale fusion features; the multi-scale fusion features are then matched with the pre-stored reference features of the target object to obtain a joint similarity score. The segmentation module is used to classify candidate target segmentation regions based on joint similarity scores in order to select the final image instance segmentation results.
9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.
10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.