Open-vocabulary affordance segmentation method and device based on object shape mask

Abstract

The application discloses an open-vocabulary affordance segmentation method and device based on an object shape mask prompt, and can solve the problems of interference between adjacent component pixels of an object in a complex scene, small-size object segmentation, and incapability of generalization to an open world. The method comprises the following steps: (1) generating an object shape mask prompt; (2) mask prompt feature enhancement; and (3) affordance prediction.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to an open vocabulary availability segmentation method based on object shape mask cues, and an open vocabulary availability segmentation device based on object shape mask cues. Background Technology

[0002] The goal of object affordance segmentation is to segment the functional parts of objects in a given image. However, in real-world scenarios, many vision tasks require segmenting not only known object affordances in the training dataset but also unknown affordances. Currently, there is a lack of research addressing these issues. Therefore, this invention proposes an Open Lexical Affordance Segmentation (OVAS) task. OVAS not only segments object affordances in closed datasets but also extends to the open world by segmenting affordances that did not appear during training. OVAS plays a crucial role in many computer vision fields, including scene understanding, human-computer interaction, human behavior recognition, visual question answering, and robotics. It has received unprecedented attention, particularly in the field of robotics. To enable robots to better work for humans and perform specific interactive tasks, it is necessary to segment object affordances.

[0003] Open-vocabulary availability segmentation (OVAS) is a highly challenging task. On one hand, availability segmentation is an object-part-level image segmentation task, where availability is a fine-grained version of an object. Unlike traditional object-level semantic segmentation tasks, object parts have complex structures, often exhibiting more intricate boundaries and appearance variations than the object as a whole. In common chaotic scenes, interference from background and neighboring object pixels significantly impacts segmentation performance. This is especially true for small objects, where encoding high-quality visual features is difficult, leading to suboptimal segmentation results. On the other hand, OVAS introduces the challenge of open granularity. OVAS requires segmenting availability for objects not present in the training dataset, a challenge for data-driven deep learning segmentation models. Traditional availability segmentation models typically rely on closed datasets where object and availability classes are predefined. Such datasets provide limited but relatively clear information during training, allowing segmentation models to learn discriminative features of different object parts. However, real-world tasks require handling images from open worlds, which may contain diverse scenes and objects, and many object availability may not appear in the training dataset. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide an open vocabulary availability segmentation method based on object shape mask cues, which can solve the problems of interference between adjacent parts pixels of objects in complex scenes, segmentation of small objects, and inability to generalize to open worlds.

[0005] The technical solution of this invention is: an open vocabulary availability segmentation method based on object shape mask cues, comprising the following steps:

[0006] (1) Generating object shape mask cues: Given an input image, a pre-trained object detector is first used to detect the object category; then, the object category and the image are input into an image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output a caption description specific to the mask region, thus obtaining captions specific to the target object region; then, these captions and object categories, as well as open availability words, are used as joint text cues and embedded into the feature space using a text encoder;

[0007] (2) Mask cue feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object, which is used to extract object instance features; extract the features of the object bounding box, which contains object and background information;

[0008] (3) Availability prediction: The encoded instance features are concatenated with the global image features and then fed into a Transformer-based pixel decoder along with the text cue features. The decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with the text embeddings to obtain the availability categories of the mask. By combining the availability categories and the mask, the final availability segmentation map is obtained.

[0009] This invention proposes generating fine-grained object shape masks as cues for generating mask images containing only a single object instance. This eliminates background interference, effectively compensates for object features, and enhances overall visual characteristics. For the open-world affordance segmentation problem, this invention uses masks to generate fine-grained captions as text cues to generate high-quality mask proposals and further accurately classify the masks. It can solve problems such as interference between adjacent pixels of objects in complex scenes, segmentation of small-sized objects, and the inability to generalize to open worlds.

[0010] An open vocabulary availability segmentation device based on object shape mask cues is also provided, the device comprising:

[0011] The object shape mask cue module, given an input image, first uses a pre-trained object detector to detect object categories; then, the object categories and the image are input into an image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output mask-region-specific caption descriptions, obtaining target object region-specific captions; then, these captions, object categories, and open availability words are used as joint text cues, and a text encoder is used to embed them into the feature space;

[0012] The mask-based feature enhancement module uses a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract object instance features; it also extracts the features of the object's bounding box, which contains object and background information.

[0013] The availability prediction module concatenates the encoded instance features with global image features, and then feeds them along with text cue features into a Transformer-based pixel decoder. This decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with the text embeddings to obtain the availability categories of the masks. By combining the availability categories and the masks, the final availability segmentation map is obtained. Attached Figure Description

[0014] Figure 1 An overall framework diagram of the open vocabulary availability segmentation method based on object shape mask cues according to the present invention is shown.

[0015] Figure 2 The pixel decoder is shown.

[0016] Figure 3 An example visualization of the segmentation results on the OVAS-25 dataset is shown. Detailed Implementation

[0017] like Figure 1 As shown, this open vocabulary availability segmentation method based on object shape mask cues includes the following steps:

[0018] (1) Generating object shape mask cues: Given an input image, a pre-trained object detector is first used to detect the object category; then, the object category and the image are input into an image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output a caption description specific to the mask region, thus obtaining captions specific to the target object region; then, these captions and object categories, as well as open availability words, are used as joint text cues and embedded into the feature space using a text encoder;

[0019] (2) Mask cue feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object, which is used to extract object instance features; extract the features of the object bounding box, which contains object and background information;

[0020] (3) Availability prediction: The encoded instance features are concatenated with the global image features and then fed into a Transformer-based pixel decoder along with the text cue features. The decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with the text embeddings to obtain the availability categories of the mask. By combining the availability categories and the mask, the final availability segmentation map is obtained.

[0021] This invention proposes generating fine-grained object shape masks as cues for generating mask images containing only a single object instance. This eliminates background interference, effectively compensates for object features, and enhances overall visual characteristics. For the open-world affordance segmentation problem, this invention uses masks to generate fine-grained captions as text cues to generate high-quality mask proposals and further accurately classify the masks. It can solve problems such as interference between adjacent pixels of objects in complex scenes, segmentation of small-sized objects, and the inability to generalize to open worlds.

[0022] Preferably, the object detection process in step (1) is described as follows:

[0023] S obj =DETR(I;Q) (1)

[0024] Where S obj Let I represent the detected object score, Q be the input image, and Q be the learnable query parameters initialized in DETR. The selection criteria are to keep the predicted score greater than the threshold T = 0.7, and finally remove objects not in the object set O. t From the objects in the set, obtain the final object set O. s The specific process is described as follows:

[0025] O s ={S obj >T|Class(obj)∈O t} (2)

[0026] Where Class(obj) represents the detected object category name, the SAM segmentation process is as follows:

[0027] M os =SAM(I;Class(obj)∈O s (3)

[0028] Where M osThe binary representation of the object shape mask, where I is the input image and obj is the target object to be detected;

[0029] The Alpha-CLIP model uses text prompts to guide availability mask prediction, taking the object mask as input and outputting a text description of the mask region. The generation process of Alpha-CLIP text prompts is as follows:

[0030] w mask =Alpha-CLIP(I;M) os (4)

[0031] Among them, w mask ={sen1,sen2,…,sen n} represents a text hint indicating the area of ​​the object's shape mask.

[0032] Preferably, in step (2), the object shape mask obtained based on MPGM... For the original input image Masking; the occluded image Background information is removed, and only single object instances are included. Pixels in other regions are filled with the average pixel value of the image, where m represents the number of occluded images. These images and the original image are then fed into the backbone network to obtain features. and F o These features are then concatenated and the number of channels is reduced through convolutional layers to obtain enhanced visual features F. v F v The calculation process is as follows:

[0033]

[0034] Where Cat[·] represents the channel concatenation operation, and C(·) represents 1×1 convolution.

[0035] Preferably, in step (3), a Transformer-based visual-text cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of availability mask proposals.

[0036] Preferably, in step (3), the text prompts in the mask prompt generation module are converted into the embedding space, and the word w of the object is introduced. obj ={obj1,obj2,…,obj n}, and all availability w aff ={aff1,aff2,…,aff cThe former provides the features of the instance, while the latter provides all availability classes, as well as categories not found in the training set, thus achieving open-world availability segmentation. Then, the CLIP text encoder is used to convert these joint text titles into an embedded space, as detailed below:

[0037] text join = CLIP.tokenize(w obj ;w aff ;w mask (6)

[0038] F t = CLIP.encode-text(text join (7)

[0039] Among them, text join This represents a joint text embedding embedded via CLIP's tokenize function, F t This indicates a combined text hint embedded via CLIP's encode-text function;

[0040] Subsequently, visual feature F v and text embedding F t The input is fed into the pixel decoder, first processing the visual feature F. v Perform self-attention processing, then combine with text features F t Cross-attention processing is performed, and finally, the fused visual-text attention features are output through a feedforward network (FFN). The FFN consists of two fully connected layers with a non-linear mapping (GELU) inserted in between. After L iterations, the final refined features are output. The specific operation is as follows:

[0041] Q v ,K v V v =Linear(F v (8)

[0042] K t V t =Linear(F t (9)

[0043] Q t =Norm(SA(Q) v ,K v V v (10)

[0044] F vt =FFN(K t +Norm(CA(Q t ,K t Vt (11)

[0045] in,

[0046] These are learnable weights. Norm represents L2 normalization, and SA and CA represent self-attention and cross-attention, respectively. They follow a unified formula.

[0047]

[0048] Finally, F vt The inputs are fed into the MLP layer to generate class-agnostic availability mask proposals. and mask class embedding F cls Then, the dot product between the mask class embedding and the text embedding is used to obtain a set of open classification scores s available for class C. cls The category score is calculated as follows:

[0049]

[0050] Where i∈[1,N] and j∈[1,C] are the indices of object availability and text embedding, respectively, σ(·) is the sigmoid function, and cos(·;·) represents cosine similarity. It is a temperature hyperparameter.

[0051] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an open vocabulary availability segmentation device based on object shape mask cues. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0052] An open vocabulary availability segmentation device based on object shape mask cues is also provided, the device comprising:

[0053] The object shape mask cue module, given an input image, first uses a pre-trained object detector to detect object categories; then, the object categories and the image are input into an image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output mask-region-specific caption descriptions, obtaining target object region-specific captions; then, these captions, object categories, and open availability words are used as joint text cues, and a text encoder is used to embed them into the feature space;

[0054] The mask-based feature enhancement module uses a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract object instance features; it also extracts the features of the object's bounding box, which contains object and background information.

[0055] The availability prediction module concatenates the encoded instance features with global image features, and then feeds them along with text cue features into a Transformer-based pixel decoder. This decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with the text embeddings to obtain the availability categories of the masks. By combining the availability categories and the masks, the final availability segmentation map is obtained.

[0056] Preferably, the object detection process in the object shape mask prompting module is described as follows:

[0057] S obj =DETR(I;Q) (1)

[0058] Where S obj Let I represent the detected object score, Q be the input image, and Q be the learnable query parameters initialized in DETR. The selection criteria are to keep the predicted score greater than the threshold T = 0.7, and finally remove objects not in the object set O. t From the objects in the set, obtain the final object set O. s The specific process is described as follows:

[0059] O s ={S obj >T|Class(obj)∈O t} (2)

[0060] Where Class(obj) represents the detected object category name, the SAM segmentation process is as follows:

[0061] M os =SAM(I;Class(obj)∈O s (3)

[0062] Where M osThe binary representation of the object shape mask, where I is the input image and obj is the target object to be detected;

[0063] The Alpha-CLIP model uses text prompts to guide availability mask prediction, taking the object mask as input and outputting a text description of the mask region. The generation process of Alpha-CLIP text prompts is as follows:

[0064] w mask =Alpha-CLIP(I;M) os (4)

[0065] Among them, w mask ={sen1,sen2,…,sen n} represents a text hint indicating the area of ​​the object's shape mask.

[0066] Preferably, in the mask cue feature enhancement module, the object shape mask is obtained based on MPGM. For the original input image Masking; the occluded image Background information is removed, and only single object instances are included. Pixels in other regions are filled with the average pixel value of the image, where m represents the number of occluded images. These images and the original image are then fed into the backbone network to obtain features. and F o These features are then concatenated and the number of channels is reduced through convolutional layers to obtain enhanced visual features F. v F v The calculation process is as follows:

[0067]

[0068] Where Cat[·] represents the channel concatenation operation, and C(·) represents 1×1 convolution.

[0069] Preferably, in the availability prediction module, a Transformer-based visual-text cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of availability mask proposals.

[0070] Preferably, in the availability prediction module, the text prompts in the mask prompt generation module are converted into the embedding space, and the word w of the object is introduced. obj ={obj1,obj2,…,obj n}, and all availability w aff ={aff1,aff2,…,aff cThe former provides the features of the instance, while the latter provides all availability classes, as well as categories not found in the training set, thus achieving open-world availability segmentation. Then, the CLIP text encoder is used to convert these joint text titles into an embedded space, as detailed below:

[0071] text join = CLIP.tokenize(w obj ;w aff ;w mask (6)

[0072] F t = CLIP.encode-text(text join (7)

[0073] Among them, text join This represents a joint text embedding embedded via CLIP's tokenize function, F t This indicates a combined text hint embedded via CLIP's encode-text function;

[0074] Subsequently, visual feature F v and text embedding F t The input is fed into the pixel decoder, first processing the visual feature F. v Perform self-attention processing, then combine with text features F t Cross-attention processing is performed, and finally, the fused visual-text attention features are output through a feedforward network (FFN). The FFN consists of two fully connected layers with a non-linear mapping (GELU) inserted in between. After L iterations, the final refined features are output. The specific operation is as follows:

[0075] Q v ,K v V v =Linear(F v (8)

[0076] K t V t =Linear(F t (9)

[0077] Q t =Norm(SA(Q) v ,K v V v (10)

[0078] F vt =FFN(K t +Norm(CA(Q t ,K t Vt (11)

[0079] in,

[0080] These are learnable weights. Norm represents L2 normalization, and SA and CA represent self-attention and cross-attention, respectively. They follow a unified formula.

[0081]

[0082] Finally, F vt The inputs are fed into the MLP layer to generate class-agnostic availability mask proposals. and mask class embedding F cls Then, the dot product between the mask class embedding and the text embedding is used to obtain a set of open classification scores s available for class C. cls The category score is calculated as follows:

[0083]

[0084] Where i∈[1,N] and j∈[1,C] are the indices of object availability and text embedding, respectively, σ(·) is the sigmoid function, and cos(·;·) represents cosine similarity. It is a temperature hyperparameter.

[0085] This invention proposes an open-vocabulary availability segmentation model based on object shape masks, also known as MaskPrompt. The model framework is as follows: Figure 1 As shown, MaskPrompt consists of three main modules: the Mask Cue Generation Module (MPGM), the Mask Cue Feature Enhancement Module (MPFEM), and the Availability Prediction Module (APM).

[0086] The method of the present invention includes:

[0087] Step 1: Generating Object Shape Mask Hints

[0088] In this module, the present invention primarily utilizes object shape masks to obtain fine-grained text cues, and removes the background of the original image to generate a mask image containing object instances. Therefore, the present invention first requires generating a refined object shape mask. To reduce model complexity and computational overhead, an off-the-shelf pre-trained image segmentation model, namely SAM, is used. SAM generates high-quality object masks from input cues such as text or bounding boxes and can be used to generate masks for all objects in an image. It has been trained on a dataset of 11 million images and 1.1 billion masks and has strong zero-shot performance on various segmentation tasks. Therefore, it is necessary to detect the target object category, and SAM can accurately segment the object shape mask based on the object category cues. For this purpose, the present invention uses the DETR model, an object detection model based on an end-to-end converter. Unlike traditional candidate box-based methods, DETR does not require pre-generating candidate boxes; it directly generates a fixed number of object queries from the image and infers the entire image through a self-attention mechanism, thereby avoiding the candidate box generation and non-maximum suppression processes and simplifying the process. Specifically, the object detection process is described as follows:

[0089] S obj =DETR(I;Q)

[0090] Where S obj Let I represent the detected object score, I be the input image, and Q be the learnable query parameters initialized in DETR. Since the objects studied in this invention are all included in the pre-trained DETR weights, there is no need to train them separately. Results that maintain a predicted score greater than the threshold T = 0.7 are selected, and those not in the object set O are finally removed. t The objects in the set. Retrieve the final object set O. s The specific process is described as follows:

[0091] O s ={S obj >T|Class(obj)∈O t}

[0092] Where Class(obj) represents the name of the detected object category. The SAM segmentation process is as follows:

[0093] M os =SAM(I;Class(obj)∈O s )

[0094] Where M os This represents a binary object shape mask, where I is the input image and obj is the target object to be detected.

[0095] Furthermore, considering the success of image-text contrastive learning, this invention uses text prompts to guide availability mask prediction. Existing text-prompt-based visual task-related methods utilize visual language models, such as CLIP, to generate text descriptions. However, CLIP generates all text information containing the entire image, but for a specific task, only a portion of this information is valuable, while the rest becomes noise. Therefore, this invention proposes a novel CLIP model, Alpha-CLIP. It takes an object mask as input and outputs a text description of the mask region. The Alpha-CLIP text prompt generation process is described below:

[0096] w mask =Alpha-CLIP(I;M) os )

[0097] Among them, w mask ={sen1,sen2,…,sen n} represents a text hint indicating the area of ​​the object's shape mask.

[0098] Step 2: Masking Cue Feature Enhancement

[0099] Traditional visual feature encoders only encode global information of an image, but for availability segmentation tasks, these features are insufficient to reduce interference from pixels adjacent to the background and target entity. Therefore, this invention proposes adding additional instance features to enhance feature representation.

[0100] Specifically, the object shape mask obtained based on MPGM For the original input image Perform masking. The masked image. Where m represents the number of occluded images, background information is removed, containing only a single object instance, and pixels in other regions are filled with the average pixel value of the image. These images and the original image are fed into the backbone network to obtain features. and F o These features are then concatenated and the number of channels is reduced through convolutional layers to obtain enhanced visual features F. v Specifically, F v The calculation process is as follows:

[0101]

[0102] Where Cat[·] represents the channel concatenation operation, and C(·) represents 1×1 convolution.

[0103] Step 3: Availability Prediction

[0104] A two-stage segmentation pipeline is employed to address the open-world affordance segmentation problem. The first stage generates class-agnostic affordance mask proposals, and the second stage classifies these proposals. This invention first designs a Transformer-based visual-text cross-modal pixel decoder to generate proposals, such as... Figure 2 As shown, it accepts two inputs: visual features and text embeddings, and outputs a set of availability mask proposals.

[0105] First, the text hints in the mask hint generation module are converted to the embedding space. To add more text descriptions, this invention introduces the object's word (w obj ={obj1,obj2,…,obj n}), and all availability (w aff ={aff1,aff2,…,aff c The former provides the features of the instances, while the latter provides all availability classes, as well as categories not present in the training set, thus achieving open-world availability segmentation. These joint text headings are then converted into an embedded space using CLIP's text encoder. The specific process is as follows:

[0106] text join = CLIP.tokenize(w obj ;w aff ;w mask )

[0107] F t = CLIP.encode-text(text join )

[0108] Among them, text join This indicates a joint text embedding via CLIP's "tokenize" function, F t This indicates a combined text hint embedded via CLIP's "encode-text" function.

[0109] Subsequently, visual feature F v and text embedding F t The input is fed into the pixel decoder. First, the visual feature F... v Perform self-attention processing, then combine with text features F t Cross-attention processing is performed, and finally, the fused visual-text attention features are output through a feedforward network (FFN). The FFN consists of two fully connected layers with a non-linear mapping GELU inserted in between. After L iterations, the final refined features are output. The specific operation is as follows:

[0110] Q v ,K v Vv =Linear(F v )

[0111] K t V t =Linear(F t )

[0112] Q t =Norm(SA(Q) v ,K v V v ))

[0113] F vt =FFN(K t +Norm(CA(Q t ,K t V t )))

[0114] in, These are learnable weights. Norm represents L2 normalization. SA and CA represent self-attention and cross-attention, respectively, and they follow a unified formula, namely...

[0115] Finally, F vt The inputs are fed into the MLP layer to generate class-agnostic availability mask proposals. and mask class embedding F cls Then, the dot product between the mask class embedding and the text embedding is used to obtain a set of open classification scores s available for class C. cls The category score is calculated as follows:

[0116]

[0117] Where i∈[1,N] and j∈[1,C] are the indices of object availability and text embedding, respectively, σ(·) is the sigmoid function, and cos(·;·) represents cosine similarity. It is a temperature hyperparameter.

[0118] This invention compares with the latest open-vocabulary-based image segmentation methods and achieves the best results. In addition, ablation experiments were conducted on the proposed mask cue generation module, mask cue feature enhancement module, and availability classification module.

[0119] 1. Experimental data: The present invention was tested on the proposed OVAS-25 dataset, as well as the traditional availability splitting datasets IIT-AFF and UMD.

[0120] 2. Metrics: Mean Intersection over Union (mIoU), Mean Accuracy (mAvg), and F1 score.

[0121] 3. Experimental Setup: The Adam optimizer was selected to optimize the objective function. In all experiments, the learning rate was set to 0.0001, the batch size to 16, and a total of 1000 epochs were trained. A domain adaptive encoder network was added for joint training after 100 epochs. The entire model was built using Python 3.6 and the PyTorch 1.7.1 deep learning framework, and trained using an NVIDIA RTX 3090 24G GPU.

[0122] 4. Since this invention aims to solve the open-vocabulary availability segmentation problem, which is essentially the same as the open-vocabulary component segmentation task, this invention selects the state-of-the-art open-vocabulary component segmentation method, trains it on the proposed OVS-25 dataset, and reports the test results, as shown in Table 1. Compared with the state-of-the-art open-vocabulary component segmentation method OV-PARTS, the proposed method improves performance by 3.53% mIoU, demonstrating the effectiveness of the proposed mask cues. Furthermore, experiments were conducted on the traditional closed-vocabulary availability segmentation datasets IIT-AFF and UMD, and the proposed method can be directly transferred, as shown in Table 1. Compared with traditional availability segmentation methods, the model proposed in this invention remains competitive.

[0123] Table 1. Quantitative comparison with state-of-the-art methods on the OVAS-25, IIT-AFF, and UMD datasets.

[0124]

[0125] 5. Visualization of experimental results: Figure 3 The visualization of the component segmentation results of this invention on the OVAS-25 test dataset is shown. The results, compared with state-of-the-art open-vocabulary component segmentation methods, demonstrate that the proposed method exhibits the best visualization results. As shown in the second and third rows, the proposed method achieves optimal performance when dealing with complex background interference and pixel interference from adjacent components of an object. Furthermore, this invention can correctly segment object components that do not appear in the labels, such as the small bottle shown in the last row. These results demonstrate the effectiveness of the proposed method, particularly for small object components.

[0126] 6. Ablation Experiments: To verify the effectiveness of the different modules designed in the proposed model, extensive experiments were conducted on the proposed dataset, and quantitative results measured on the test dataset are reported, as shown in Table 2. The basic model of this invention utilizes only the original features of the image, while the pixel decoder adopts a structure similar to MaskFormer. As shown in the second row of Table 2, a Mask Cue Feature Enhancement Module (MPFEM) was added, which uses mask cues to add instance features to enhance the visual feature representation, resulting in a 4.9% mIoU improvement. In the third row of Table 2, fine-grained text cues were added to the MPGM module, resulting in a 2.24% mIoU improvement compared to adding instance features only. The last row of Table 2 ablates the proposed pixel decoder. Replacing the MaskFormer pixel decoder with the proposed pixel decoder, which utilizes text cues as prior knowledge, resulted in a 1.87% mIoU improvement. These ablation experiments validate the performance of the proposed modules and demonstrate the effectiveness of object shape mask cues.

[0127] Table 2 shows the ablation experiments of a single module in the OVAS-25 dataset.

[0128]

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An open-vocabulary availability segmentation method based on object shape mask cues, characterized in that: The method includes the following steps: (1) Generate object shape mask cues: Given an input image, a pre-trained object detector is first used to detect the object category; then, the object category and the image are input into an image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output a caption description specific to the mask region, thus obtaining captions specific to the target object region; then, these captions and object categories, as well as open availability words, are used as joint text cues and embedded into the feature space using a text encoder; (2) Mask cue feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object, which is used to extract object instance features; extract the features of the object bounding box, which contains object and background information; (3) Availability prediction: The encoded instance features are concatenated with the global image features and then fed into a Transformer-based pixel decoder along with the text cue features. The decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with the text embeddings to obtain the availability categories of the mask. By combining the availability categories and the mask, the final availability segmentation map is obtained. In step (1), the object detection process is described as follows: （1） in Indicates the score of the detected objects. It is the input image. These are learnable query parameters initialized in DETR, chosen to keep the predicted score above a threshold. =0.7, and finally remove objects not in the collection. Get the final set of objects from the objects in the collection. The specific process is described as follows: （2） in Representing the category name of the detected object, the SAM segmentation process is as follows: （3） in A binary representation of an object shape mask. It is the input image. It is the target object being detected; The Alpha-CLIP model uses text prompts to guide availability mask prediction, taking the object mask as input and outputting a text description of the mask region. The generation process of Alpha-CLIP text prompts is as follows: （4） in, Text hints indicating the masked area of ​​an object's shape; In step (2), the object shape mask obtained based on MPGM For the original input image Masking; the occluded image Background information is removed, containing only a single object instance; pixels in other areas are filled with the average pixel value of the image. This represents the number of occluded images; these images and the original image are fed into the backbone network to obtain features. and These features are then concatenated and the number of channels is reduced through convolutional layers to obtain enhanced visual features. , The calculation process is as follows: （5） in, This indicates a channel splicing operation. This represents a 1×1 convolution.

2. The open vocabulary availability segmentation method based on object shape mask cues according to claim 1, characterized in that: In step (3), a Transformer-based visual-text cross-modal pixel decoder is designed to generate proposals, which accept visual features and text embeddings and output a set of availability mask proposals.

3. The open vocabulary availability segmentation method based on object shape mask cues according to claim 2, characterized in that: In step (3), the text prompts in the mask prompt generation module are converted into the embedding space, and the words of the object are introduced. and all availability The former provides the features of the instance, while the latter provides all availability classes, as well as categories not found in the training set, thus achieving open-world availability segmentation. Then, the CLIP text encoder is used to convert these joint text titles into an embedded space, as detailed below: （6） （7） in, This indicates a joint text embedding embedded via CLIP's tokenize function. This indicates a combined text hint embedded via CLIP's encode-text function; Then, visual features and text embedding The input is fed into the pixel decoder, first processing the visual features. Perform self-attention processing, then combine with text features. Cross-attention processing is performed, and finally, the fused visual-text attention features are output through a feedforward network (FFN). The FFN consists of two fully connected layers with a non-linear mapping (GELU) inserted in between. L After the next iteration, the final refined features will be output. The specific steps are as follows: （8） （9） （10） （11） in, , These are learnable weights. Norm Indicates L2 normalization, SA and CA These represent self-attention and cross-attention, respectively, and they follow a unified formula. ; at last, The inputs are fed into the MLP layer to generate class-agnostic availability mask proposals. and mask-type embedding Then, the dot product between the mask class embedding and the text embedding is used to obtain a set of open classes available for use. Category score The category score is calculated as follows: （12） in, and These are the indexes for object availability and text embedding, respectively. It is the sigmoid function. To represent cosine similarity, It is a temperature hyperparameter.

4. An open vocabulary availability segmentation device based on object shape mask cues, characterized in that: The device includes: The object shape mask cue module, given an input image, first uses a pre-trained object detector to detect object categories; then, the object categories and the image are input into an image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, the mask and the original image are input into Alpha-CLIP to output mask-region-specific caption descriptions, obtaining target object region-specific captions; then, these captions, object categories, and open availability words are used as joint text cues, and a text encoder is used to embed them into the feature space; The mask-based feature enhancement module uses a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract object instance features; it also extracts the features of the object's bounding box, which contains object and background information. The availability prediction module concatenates the encoded instance features with global image features, and then feeds them along with text cue features into a Transformer-based pixel decoder. This decoder outputs category-agnostic availability mask proposals. These proposals are multiplied with text embeddings to obtain the availability categories of the masks. By combining the availability categories and masks, the final availability segmentation map is obtained. The object detection process in the object shape mask prompting module is described as follows: （1） in Indicates the score of the detected objects. It is the input image. These are learnable query parameters initialized in DETR, chosen to keep the predicted score above a threshold. =0.7, and finally remove objects not in the collection. Get the final set of objects from the objects in the collection. The specific process is described as follows: （2） in Representing the category name of the detected object, the SAM segmentation process is as follows: （3） in A binary representation of an object shape mask. It is the input image. It is the target object being detected; The Alpha-CLIP model uses text prompts to guide availability mask prediction, taking the object mask as input and outputting a text description of the mask region. The generation process of Alpha-CLIP text prompts is as follows: （4） in, Text hints indicating the masked area of ​​an object's shape; In the mask cue feature enhancement module, the object shape mask is obtained based on MPGM. For the original input image Masking; the occluded image Background information is removed, containing only a single object instance; pixels in other areas are filled with the average pixel value of the image. This represents the number of occluded images; these images and the original image are fed into the backbone network to obtain features. and These features are then concatenated and the number of channels is reduced through convolutional layers to obtain enhanced visual features. , The calculation process is as follows: （5） in, This indicates a channel splicing operation. This represents a 1×1 convolution.

5. The open vocabulary availability segmentation device based on object shape mask cues according to claim 4, characterized in that: In the availability prediction module, a Transformer-based visual-text cross-modal pixel decoder is designed to generate proposals. It accepts visual features and text embeddings and outputs a set of availability mask proposals.

6. The open vocabulary availability segmentation device based on object shape mask cues according to claim 5, characterized in that: In the availability prediction module, the text prompts from the mask prompt generation module are converted into the embedding space, and the object's words are introduced. and all availability The former provides the features of the instance, while the latter provides all availability classes, as well as categories not found in the training set, thus achieving open-world availability segmentation. Then, the CLIP text encoder is used to convert these joint text titles into an embedded space, as detailed below: （6） （7） in, This indicates a joint text embedding embedded via CLIP's tokenize function. This indicates a combined text hint embedded via CLIP's encode-text function; Then, visual features and text embedding The input is fed into the pixel decoder, first processing the visual features. Perform self-attention processing, then combine with text features. Cross-attention processing is performed, and finally, the fused visual-text attention features are output through a feedforward network (FFN). The FFN consists of two fully connected layers with a non-linear mapping (GELU) inserted in between. L After the next iteration, the final refined features will be output. The specific steps are as follows: （8） （9） （10） （11） in, , These are learnable weights. Norm Indicates L2 normalization, SA and CA These represent self-attention and cross-attention, respectively, and they follow a unified formula. ; at last, The inputs are fed into the MLP layer to generate class-agnostic availability mask proposals. and mask-type embedding Then, the dot product between the mask class embedding and the text embedding is used to obtain a set of open classes available for use. Category score The category score is calculated as follows: （12） in, and These are the indexes for object availability and text embedding, respectively. It is the sigmoid function. To represent cosine similarity, It is a temperature hyperparameter.

Images