Open vocabulary accessibility segmentation method and device based on object shape mask prompt
By generating fine object shape masks and using Alpha-CLIP to generate fine-grained text prompts, combined with Transformer model for affordability prediction, the problems of object adjacent components and small-size object segmentation in complex scenarios are solved, and the openness is achieved High-quality affordability segmentation in the world.
Patent Information
- Application Number
- CN202411023864.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The prior art is difficult to segment the affordability of objects in complex scenarios, especially small-sized objects and the problem of being unable to generalize to the open world. The traditional segmentation model relies on closed data sets and cannot handle the affordability of unknown objects.
Using an open vocabulary affordability segmentation method based on object shape mask hints, we generate fine object shape masks to remove background interference, use Alpha-CLIP to generate fine-grained text hints, and combine it with Transformer model to make affordability prediction. Generate high-quality affordability segmentation maps.
It effectively solves the problems of pixel interference between adjacent parts of the object, small-size object segmentation, and the inability to generalize to the open world in complex scenarios, and improves the quality and accuracy of the segmentation results.
Smart Images

Figure CN118968063A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and in particular to an open vocabulary availability segmentation method based on object shape mask prompts, and an open vocabulary availability segmentation device based on object shape mask prompts. Background Art
[0002] The purpose of object affordance segmentation is to segment the functional parts of an object in a given image. But in real life, many visual tasks require not only segmenting the known affordances of objects in the training dataset, but also segmenting unknown affordances. However, there is no research on these issues. Therefore, the present invention proposes an open vocabulary affordance segmentation task, namely OVAS. OVAS not only segments the affordances of objects in a closed dataset, but also extends to the open world by segmenting affordances that did not appear during training. OVAS plays an important role in many computer vision fields such as scene understanding, human-computer interaction, human behavior recognition, visual question answering, and robotics. In particular, it has received unprecedented attention in the field of robotics. In order to make robots work better for humans and perform specific interactive tasks, it is necessary to segment the affordances of objects.
[0003] Open vocabulary affordance segmentation is a very challenging task. On the one hand, affordance segmentation is an object-part-level image segmentation task, where affordance is a fine-grained version of an object. Unlike traditional object-level semantic segmentation tasks, object parts are complex in structure and usually have more complex boundaries and appearance variations than the object as a whole. In common chaotic scenes, the interference of background and object neighboring part pixels can significantly affect the segmentation performance. Especially for small-sized objects, it is difficult to encode high-quality visual features for them, resulting in unsatisfactory segmentation results. On the other hand, open vocabulary affordance segmentation introduces the challenge of open granularity. OVAS requires segmenting affordances of objects that are not present in the training dataset, which is difficult for data-driven deep learning segmentation models. Traditional affordance segmentation models usually rely on closed datasets where object and affordance classes are predefined. Such datasets provide limited but relatively clear information during training, allowing segmentation models to learn discriminative features of different parts of objects. However, real-world tasks require processing images in the open world, which may contain a variety of different scenes and objects, and many object affordances may not appear in the training dataset. Summary of the invention
[0004] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide an open vocabulary affordance segmentation method based on object shape mask hints, which can solve the problems of interference between pixels of adjacent parts of objects in complex scenes, segmentation of small-sized objects and inability to be generalized to the open world.
[0005] The technical solution of the present invention is: the open vocabulary affordance segmentation method based on object shape mask prompts comprises the following steps:
[0006] (1) Generate object shape mask cues: Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, input the mask and the original image into Alpha-CLIP to output caption descriptions specific to the mask region and obtain captions specific to the target object region; then use these captions and object categories, as well as an open affordance vocabulary, as joint text cues and embed them into the feature space using a text encoder;
[0007] (2) Mask hint feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information.
[0008] (3) Availability prediction: The encoded instance features are concatenated with the global image features and then input into a Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic affordance mask proposals. These proposals are multiplied with the text embedding to obtain the affordance category of the mask. The final affordance segmentation map is obtained by combining the affordance category and the mask.
[0009] The present invention proposes to generate a fine object shape mask as a hint for generating a mask image containing only a single object instance, which can exclude background interference and well compensate for object features, thereby enhancing overall visual features. For the open world affordance segmentation problem, the present invention uses masks to generate fine-grained subtitles as text hints for generating high-quality mask proposals and further accurately classifying masks. It can solve the problems of interference between pixels of adjacent parts of objects in complex scenes, segmentation of small-sized objects, and inability to generalize to the open world.
[0010] A device for segmenting open vocabulary affordance based on object shape mask prompts is also provided, the device comprising:
[0011] Generate object shape mask prompt module. Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text prompts, the mask and the original image are input into Alpha-CLIP to output caption descriptions specific to the mask area and obtain captions specific to the target object area; then use these captions and object categories, as well as an open affordance vocabulary as joint text prompts, and use a text encoder to embed them into the feature space;
[0012] The mask hint feature enhancement module uses a mask to remove the background and obtains a mask image containing only a single object. The mask image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information.
[0013] In the availability prediction module, the encoded instance features are concatenated with the global image features and then input into the Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic availability mask proposals; these proposals are multiplied with the text embedding to obtain the availability category of the mask, and the final availability segmentation map is obtained by combining the availability category and the mask. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 The overall framework diagram of the open vocabulary affordance segmentation method based on object shape mask hints according to the present invention is shown.
[0015] Figure 2 A pixel decoder is shown.
[0016] Figure 3 Figure 2 shows an example visualization of segmentation results on the OVAS-25 dataset. DETAILED DESCRIPTION
[0017] like Figure 1 As shown in FIG. 1 , this open vocabulary affordance segmentation method based on object shape mask cues includes the following steps:
[0018] (1) Generate object shape mask cues: Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, input the mask and the original image into Alpha-CLIP to output caption descriptions specific to the mask region and obtain captions specific to the target object region; then use these captions and object categories, as well as an open affordance vocabulary, as joint text cues and embed them into the feature space using a text encoder;
[0019] (2) Mask hint feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information.
[0020] (3) Availability prediction: The encoded instance features are concatenated with the global image features and then input into a Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic affordance mask proposals. These proposals are multiplied with the text embedding to obtain the affordance category of the mask. The final affordance segmentation map is obtained by combining the affordance category and the mask.
[0021] The present invention proposes to generate a fine object shape mask as a hint for generating a mask image containing only a single object instance, which can exclude background interference and well compensate for object features, thereby enhancing overall visual features. For the open world affordance segmentation problem, the present invention uses masks to generate fine-grained subtitles as text hints for generating high-quality mask proposals and further accurately classifying masks. It can solve the problems of interference between pixels of adjacent parts of objects in complex scenes, segmentation of small-sized objects, and inability to generalize to the open world.
[0022] Preferably, in step (1), the object detection process is described as follows:
[0023] S obj =DETR(I;Q) (1)
[0024] Where S obj represents the detected object score, I is the input image, Q is the learnable query parameter initialized in DETR, and the results with prediction scores greater than the threshold T = 0.7 are selected to be kept, and finally the results that are not in the object set O are removed. t The objects in the final object set O are obtained s The specific process is described as follows:
[0025] O s ={S obj >T|Class(obj)∈O t} (2)
[0026] Where Class(obj) represents the name of the detected object category. The segmentation process of SAM is as follows:
[0027] M os =SAM(I;Class(obj)∈O s ) (3)
[0028] Among them, M osRepresents a binary object shape mask, I is the input image, obj is the target object to be detected;
[0029] Use the Alpha-CLIP model text prompt to guide the affordance mask prediction, taking the object mask as input and outputting the text description of the mask area. The generation process of the Alpha-CLIP text prompt is as follows:
[0030] w mask =Alpha-CLIP(I;M os ) (4)
[0031] Among them, w mask ={sen1,sen2,…,sen n} represents the text hint of the object shape mask area.
[0032] Preferably, in step (2), the object shape mask obtained based on MPGM For the original input image Masking; occluded image Remove the background information, only include a single object instance, and fill the pixels in other areas with the average value of the image pixels, where m represents the number of occluded images; send these images and the original image to the backbone network to obtain the features respectively. and F o , then concatenate these features and reduce the number of channels through a convolutional layer to obtain enhanced visual features F v , F v The calculation process is as follows:
[0033]
[0034] Among them, Cat[·] represents the channel concatenation operation, and C(·) represents a 1×1 convolution.
[0035] Preferably, in step (3), a Transformer-based visual-textual cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of affordance mask proposals.
[0036] Preferably, in step (3), the text prompt in the mask prompt generation module is converted into the embedding space, and the word w of the object is introduced obj = {obj1, obj2, …, obj n}, and all affordances w aff ={aff1,aff2,…,aff c}; the former provides the features of the instance, and the latter provides all affordance classes, as well as categories not present in the training set, thereby achieving open-world affordance segmentation; these joint text titles are then converted into an embedded space using CLIP’s text encoder, as follows:
[0037] text join = CLIP.tokenize(w obj ;w aff ;w mask ) (6)
[0038] F t = CLIP.encode-text(text join ) (7)
[0039] Among them, text join represents the joint text embedding embedded by the tokenize function of CLIP, F t It represents the joint text prompt embedded by the encode-text function of CLIP;
[0040] Then, the visual feature F v and text embedding F t Input to the pixel decoder, first the visual feature F v Perform self-attention processing and then combine it with the text feature F t Perform cross attention processing, and finally output the fused visual-text attention features through the feedforward network FFN. FFN consists of two fully connected layers with a nonlinear mapping GELU inserted in the middle. After L iterations, the final refined features will be output. The specific operations are as follows:
[0041] Q v ,K v ,V v =Linear(F v ) (8)
[0042] K t ,V t =Linear(F t ) (9)
[0043] Q t =Norm(SA(Q v ,K v ,V v )) (10)
[0044] F vt =FFN(K t +Norm(CA(Q t ,K t ,Vt ))) (11)
[0045] in,
[0046] is a learnable weight, Norm represents L2 normalization, SA and CA represent self-attention and cross-attention respectively, which follow a unified formula,
[0047]
[0048] Finally, F vt are fed into the MLP layer to generate class-agnostic affordance mask proposals respectively and mask class embedding F cls , and then use the dot product between the mask class embedding and the text embedding to obtain the classification scores s for a set of open available classes C cls , the classification score is calculated as follows:
[0049]
[0050] where i∈[1,N] and j∈[1,C] are the indices of object affordance and text embedding, respectively, σ(·) is the sigmoid function, and cos(·; ·) represents the cosine similarity. is the temperature hyperparameter.
[0051] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an open vocabulary availability segmentation device based on object shape mask prompts, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:
[0052] A device for segmenting open vocabulary affordance based on object shape mask prompts is also provided, the device comprising:
[0053] Generate object shape mask prompt module. Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text prompts, the mask and the original image are input into Alpha-CLIP to output caption descriptions specific to the mask area and obtain captions specific to the target object area; then use these captions and object categories, as well as an open affordance vocabulary as joint text prompts, and use a text encoder to embed them into the feature space;
[0054] The mask hint feature enhancement module uses a mask to remove the background and obtains a mask image containing only a single object. The mask image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information.
[0055] In the availability prediction module, the encoded instance features are concatenated with the global image features and then input into the Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic availability mask proposals; these proposals are multiplied with the text embedding to obtain the availability category of the mask, and the final availability segmentation map is obtained by combining the availability category and the mask.
[0056] Preferably, in the object shape mask prompting module, the object detection process is described as follows:
[0057] S obj =DETR(I;Q) (1)
[0058] Where S obj represents the detected object score, I is the input image, Q is the learnable query parameter initialized in DETR, and the results with prediction scores greater than the threshold T = 0.7 are selected to be kept, and finally the results that are not in the object set O are removed. t The objects in the final object set O are obtained s The specific process is described as follows:
[0059] O s ={S obj >T|Class(obj)∈O t} (2)
[0060] Where Class(obj) represents the name of the detected object category. The segmentation process of SAM is as follows:
[0061] M os =SAM(I;Class(obj)∈O s ) (3)
[0062] Among them, M osRepresents a binary object shape mask, I is the input image, obj is the target object to be detected;
[0063] Use the Alpha-CLIP model text prompt to guide the affordance mask prediction, taking the object mask as input and outputting the text description of the mask area. The generation process of the Alpha-CLIP text prompt is as follows:
[0064] w mask =Alpha-CLIP(I;M os ) (4)
[0065] Among them, w mask ={sen1,sen2,…,sen n} represents the text hint of the object shape mask area.
[0066] Preferably, in the mask hint feature enhancement module, the object shape mask obtained based on MPGM For the original input image Masking; occluded image Remove the background information, only include a single object instance, and fill the pixels in other areas with the average value of the image pixels, where m represents the number of occluded images; send these images and the original image to the backbone network to obtain the features respectively. and F o , then concatenate these features and reduce the number of channels through a convolutional layer to obtain enhanced visual features F v , F v The calculation process is as follows:
[0067]
[0068] Among them, Cat[·] represents the channel concatenation operation, and C(·) represents a 1×1 convolution.
[0069] Preferably, in the availability prediction module, a Transformer-based visual-textual cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of availability mask proposals.
[0070] Preferably, in the availability prediction module, the text prompts in the mask prompt generation module are converted into the embedding space, and the word w of the object is introduced. obj = {obj1, obj2, …, obj n}, and all affordances w aff ={aff1,aff2,…,aff c}; the former provides the features of the instance, and the latter provides all affordance classes, as well as categories not present in the training set, thereby achieving open-world affordance segmentation; these joint text titles are then converted into an embedded space using CLIP’s text encoder, as follows:
[0071] text join = CLIP.tokenize(w obj ;w aff ;w mask ) (6)
[0072] F t = CLIP.encode-text(text join ) (7)
[0073] Among them, text join represents the joint text embedding embedded by the tokenize function of CLIP, F t It represents the joint text prompt embedded by the encode-text function of CLIP;
[0074] Then, the visual feature F v and text embedding F t Input to the pixel decoder, first the visual feature F v Perform self-attention processing and then combine it with the text feature F t Perform cross attention processing, and finally output the fused visual-text attention features through the feedforward network FFN. FFN consists of two fully connected layers with a nonlinear mapping GELU inserted in the middle. After L iterations, the final refined features will be output. The specific operations are as follows:
[0075] Q v ,K v ,V v =Linear(F v ) (8)
[0076] K t ,V t =Linear(F t ) (9)
[0077] Q t =Norm(SA(Q v ,K v ,V v )) (10)
[0078] F vt =FFN(K t +Norm(CA(Q t ,K t ,Vt ))) (11)
[0079] in,
[0080] is a learnable weight, Norm represents L2 normalization, SA and CA represent self-attention and cross-attention respectively, which follow a unified formula,
[0081]
[0082] Finally, F vt are fed into the MLP layer to generate class-agnostic affordance mask proposals respectively and mask class embedding F cls , and then use the dot product between the mask class embedding and the text embedding to obtain the classification scores s for a set of open available classes C cls , the classification score is calculated as follows:
[0083]
[0084] where i∈[1,N] and j∈[1,C] are the indices of object affordance and text embedding, respectively, σ(·) is the sigmoid function, and cos(·; ·) represents the cosine similarity. is the temperature hyperparameter.
[0085] The present invention proposes an open vocabulary affordance segmentation model based on object shape mask, namely MaskPrompt, the model framework is as follows Figure 1 As shown in Figure 2, MaskPrompt consists of three main modules, namely, the Mask Prompt Generation Module (MPGM), the Mask Prompt Feature Enhancement Module (MPFEM), and the Availability Prediction Module (APM).
[0086] The method of the present invention comprises:
[0087] Step 1: Object shape mask hint generation
[0088] In this module, the present invention mainly uses object shape masks to obtain fine-grained text prompts, as well as to remove the background of the original image and generate a mask image containing an object instance. Therefore, the present invention first needs to generate a refined object shape mask. In order to reduce the complexity and computational overhead of the model, a ready-made pre-trained image segmentation model, namely SAM, is used. SAM generates high-quality object masks from input prompts such as text or rectangular boxes, and can be used to generate masks for all objects in the image. It has been trained on a dataset of 11 million images and 1.1 billion masks, and has strong zero-sample performance on various segmentation tasks. Therefore, it is necessary to detect the target object category, and SAM can accurately segment the object shape mask based on the object category prompt. To this end, the present invention uses the DETR model, which is an object detection model based on an end-to-end converter. Unlike traditional candidate box-based methods, DETR does not require pre-generation of candidate boxes. It directly generates a fixed number of object queries from the image and infers the entire image through a self-attention mechanism, thereby avoiding the process of candidate box generation and non-maximum suppression and simplifying the process. Specifically, the process of object detection is described as follows:
[0089] S obj =DETR(I;Q)
[0090] Where S obj represents the detected object score, I is the input image, and Q is the learnable query parameter initialized in DETR. Since the objects studied in this paper are all included in the pre-trained DERT weights, there is no need to train them separately. The results with prediction scores greater than the threshold T = 0.7 are selected and finally removed from the object set O. t Get the final object set O s The specific process is described as follows:
[0091] O s ={S obj >T|Class(obj)∈O t}
[0092] Where Class(obj) represents the name of the detected object category. The segmentation process of SAM is as follows:
[0093] M os =SAM(I;Class(obj)∈O s )
[0094] Among them, M os Represents a binary object shape mask, I is the input image, and obj is the target object to be detected.
[0095] In addition, considering the success of image-text contrastive learning, the present invention uses textual cues to guide affordance mask prediction. Existing visual task-related methods based on textual cues all use visual language models, such as CLIP, to generate textual descriptions. However, CLIP generates all textual information containing the entire image, but for specific tasks, only part of the information is valuable, and the other information becomes noise. To this end, the present invention proposes to use a new CLIP model, Alpha-CLIP. It takes the object mask as input and outputs a textual description of the mask area. The generation process of Alpha-CLIP text cues is described as follows:
[0096] w mask =Alpha-CLIP(I;M os )
[0097] Among them, w mask ={sen1,sen2,…,sen n} represents the text hint of the object shape mask area.
[0098] Step 2: Mask hint feature enhancement
[0099] Traditional visual feature encoders only encode the global information of the image, but for the task of affordance segmentation, these features are not enough to reduce the interference of pixels in the background and adjacent parts of the target entity. Therefore, the present invention proposes to add additional instance features to enhance feature representation.
[0100] Specifically, the object shape mask obtained based on MPGM For the original input image Masking. The occluded image Where m represents the number of occluded images) removes background information and only contains a single object instance. The pixels in other areas are filled with the average value of the image pixels. These images and the original image are fed into the backbone network to obtain the features and F o These features are then concatenated and the number of channels is reduced through a convolutional layer to obtain the enhanced visual feature F v Specifically, F v The calculation process is as follows:
[0101]
[0102] Among them, Cat[·] represents the channel concatenation operation, and C(·) represents a 1×1 convolution.
[0103] Step 3: Availability Forecast
[0104] A two-stage segmentation pipeline is adopted to solve the open-world affordance segmentation problem. The first stage generates class-agnostic affordance mask proposals, and then the second stage classifies these proposals. We first design a Transformer-based visual-textual cross-modal pixel decoder to generate proposals, such as Figure 2 It takes two inputs, visual features and text embeddings, and outputs a set of affordance mask proposals.
[0105] First, the text prompts in the mask prompt generation module are converted into the embedding space. In order to add more text descriptions, the present invention introduces the words of the object (w obj = {obj1, obj2, …, obj n}), and all affordances (w aff ={aff1,aff2,…,aff c}). The former provides the features of the instance, and the latter provides all affordance classes, as well as those not present in the training set, thus achieving open-world affordance segmentation. These joint text captions are then converted into an embedded space using CLIP’s text encoder. The specific process is as follows:
[0106] text join = CLIP.tokenize(w obj ;w aff ;w mask )
[0107] F t = CLIP.encode-text(text join )
[0108] Among them, text join represents the joint text embedding embedded by the “tokenize” function of CLIP, F t This represents the joint text hint embedded by the CLIP "encode-text" function.
[0109] Then, the visual feature F v and text embedding F t Input to the pixel decoder. First, the visual feature F v Perform self-attention processing and then combine it with the text feature F t Perform cross attention processing and finally output the fused visual-textual attention features through the feedforward network (FFN). FFN consists of two fully connected layers with a nonlinear mapping GELU inserted in the middle. After L iterations, the final refined features will be output. The specific operations are as follows:
[0110] Q v ,K v ,Vv =Linear(F v )
[0111] K t ,V t =Linear(F t )
[0112] Q t =Norm(SA(Q v ,K v ,V v ))
[0113] F vt =FFN(K t +Norm(CA(Q t ,K t ,V t )))
[0114] in, is a learnable weight. Norm represents L2 normalization. SA and CA represent self-attention and cross-attention, respectively, which follow a unified formula, namely
[0115] Finally, F vt are fed into the MLP layer to generate class-agnostic affordance mask proposals respectively and mask class embedding F cls The dot product between the mask class embedding and the text embedding is then used to obtain the classification scores s for a set of open available classes C cls , the classification score is calculated as follows:
[0116]
[0117] where i∈[1,N] and j∈[1,C] are the indices of object affordance and text embedding, respectively, σ(·) is the sigmoid function, and cos(·; ·) represents the cosine similarity. is the temperature hyperparameter.
[0118] The present invention conducted comparative experiments on the latest open vocabulary-based image segmentation method and achieved the best effect. In addition, ablation experiments were performed on the proposed mask hint generation module, mask hint feature enhancement module and affordance classification module.
[0119] 1. Experimental data: This paper conducts experiments on the proposed OVAS-25 dataset, as well as the traditional affordance segmentation datasets IIT-AFF and UMD.
[0120] 2. Metrics: mean intersection over union (mIoU), average precision (mAvg), and F1 score.
[0121] 3. Experimental settings: Adam optimizer is selected to optimize the objective function. In all experiments, the learning rate is set to 0.0001, the batch size is set to 16, a total of 1000 epochs are trained, and the domain adaptive encoder network is added for joint training after 100 epochs. The entire model is built based on the programming language Python3.6 and the deep learning framework Pytorch1.7.1, and trained using the NVIDIA RTX 3090 24G GPU.
[0122] 4. Since the present invention aims to solve the open vocabulary availability segmentation problem, which is essentially the same as the open vocabulary component segmentation task. Therefore, the present invention selects the most advanced existing open vocabulary component segmentation method, trains it on the proposed OVS-25 dataset, and reports the test results, as shown in Table 1. Compared with the most advanced open vocabulary component segmentation method OV-PARTS, the performance of the method proposed in the present invention is improved by 3.53% mIoU, which proves the effectiveness of the proposed mask hint. In addition, experiments were conducted on the traditional closed vocabulary availability segmentation datasets IIT-AFF and UMD, and the method of the present invention can be directly migrated, as shown in Table 1. Compared with traditional availability segmentation methods, the model proposed in the present invention is still competitive.
[0123] Table 1 Quantitative comparison with state-of-the-art methods on OVAS-25, IIT-AFF and UMD datasets
[0124]
[0125] 5. Visualization of experimental results: Figure 3 The visualization of the component segmentation results of the present invention on the OVAS-25 test dataset is shown in . The results compared with the existing state-of-the-art open vocabulary component segmentation methods show that the method proposed in the present invention shows the best visualization results. As shown in the second and third rows, the method proposed in the present invention can achieve the best performance when solving the problems of complex background interference and pixel interference of adjacent components of objects. In addition, the present invention can also correctly segment object components that do not appear in the label, such as the small bottle shown in the last row. These results demonstrate the effectiveness of the proposed method, especially for tiny object components.
[0126] 6. Ablation Experiments: To examine the effectiveness of the different modules designed in the proposed model, extensive experiments are conducted on the proposed datasets and quantitative results measured on the test datasets are reported, as shown in Table 2. The base model of the present invention only utilizes the original features of the image, while the pixel decoder adopts a structure similar to MaskFormer. As shown in the second row of Table 2, a mask-cued feature enhancement module (MPFEM) is added, which enhances the visual feature representation by adding instance features using mask cues, and the performance is improved by 4.9% mIoU. In the third row of Table 2, adding fine-grained text cues in the MPGM module improves the performance by 2.24% mIoU compared to adding only instance features. The last row of Table 2 ablates the proposed pixel decoder. Replacing the pixel decoder of MaskFormer with the proposed pixel decoder, which utilizes text cues as knowledge priors, improves the performance by 1.87% mIoU. These ablation experiments validate the performance of the proposed modules and demonstrate the effectiveness of object shape mask cues.
[0127] Table 2 Ablation experiments of a single module in the OVAS-25 dataset
[0128]
[0129] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.
Claims
1. An open vocabulary affordance segmentation method based on object shape mask cues, characterized by: The method comprises the following steps: (1) Generate object shape mask cues: Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; to obtain fine-grained text cues, input the mask and the original image into Alpha-CLIP to output caption descriptions specific to the mask region and obtain captions specific to the target object region; then use these captions and object categories, as well as an open affordance vocabulary, as joint text cues and embed them into the feature space using a text encoder; (2) Mask hint feature enhancement: Use a mask to remove the background and obtain a mask image containing only a single object. This image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information. (3) Availability prediction: The encoded instance features are concatenated with the global image features and then input into a Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic affordance mask proposals. These proposals are multiplied with the text embedding to obtain the affordance category of the mask. The final affordance segmentation map is obtained by combining the affordance category and the mask.
2. The open vocabulary affordance segmentation method based on object shape mask hint according to claim 1, characterized in that: In step (1), the object detection process is described as follows: S obj =DETR(I;Q) (1) Where S obj represents the detected object score, I is the input image, Q is the learnable query parameter initialized in DETR, and the results with prediction scores greater than the threshold T = 0.7 are selected to be kept, and finally the results that are not in the object set O are removed. t The objects in the final object set O are obtained s The specific process is described as follows: O s ={S obj >T|Class(obj)∈O t } (2) Where Class(obj) represents the name of the detected object category. The segmentation process of SAM is as follows: M os =SAM(I;Class(obj)∈O s ) (3) Among them, M os Represents a binary object shape mask, I is the input image, obj is the target object to be detected; Use the Alpha-CLIP model text prompt to guide the affordance mask prediction, taking the object mask as input and outputting the text description of the mask area. The generation process of the Alpha-CLIP text prompt is as follows: w mask =Alpha-CLIP(I;M os ) (4) Among them, w mask ={sen1,sen2,…,sen n } represents the text hint of the object shape mask area.
3. The open vocabulary affordance segmentation method based on object shape mask hint according to claim 2, characterized in that: In the step (2), the object shape mask obtained based on MPGM For the original input image Masking; occluded image Remove the background information, only include a single object instance, and fill the pixels in other areas with the average value of the image pixels, where m represents the number of occluded images; send these images and the original image to the backbone network to obtain the features respectively. and F o , then concatenate these features and reduce the number of channels through a convolutional layer to obtain enhanced visual features F v , F v The calculation process is as follows: Among them, Cat[·] represents the channel concatenation operation, and C(·) represents a 1×1 convolution.
4. The method for segmenting open vocabulary affordance based on object shape mask hints according to claim 3, characterized in that: In step (3), a Transformer-based visual-textual cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of affordance mask proposals.
5. The method for segmenting open vocabulary affordance based on object shape mask hints according to claim 4, characterized in that: In step (3), the text prompt in the mask prompt generation module is converted into the embedding space, and the word w of the object is introduced obj = {obj1, obj2, …, obj n }, and all affordances w aff ={aff1,aff2,…,aff c }; the former provides the features of the instance, and the latter provides all affordance classes, as well as categories not present in the training set, thereby achieving open-world affordance segmentation; these joint text titles are then converted into an embedded space using CLIP’s text encoder, as follows: text join =CLIP.tokenize(w obj ;w aff ;w mask ) (6) F t =CLIP.encode-text(text join ) (7) Among them, text join represents the joint text embedding embedded by the tokenize function of CLIP, F t It represents the joint text prompt embedded by the encode-text function of CLIP; Then, the visual feature F v and text embedding F t Input to the pixel decoder, first the visual feature F v Perform self-attention processing and then combine it with the text feature F t Perform cross attention processing, and finally output the fused visual-text attention features through the feedforward network FFN. FFN consists of two fully connected layers with a nonlinear mapping GELU inserted in the middle. After L iterations, the final refined features will be output. The specific operations are as follows: Q v ,K v ,V v =Linear(F v ) (8) K t ,V t =Linear(F t ) (9) Q t =Norm(SA(Q v ,K v ,V v )) (10) F vt =FFN(K t +Norm(CA(Q t ,K t ,V t ))) (11) in, is a learnable weight, Norm represents L2 normalization, SA and CA represent self-attention and cross-attention respectively, which follow a unified formula, Finally, F vt are fed into the MLP layer to generate class-agnostic affordance mask proposals respectively and mask class embedding F cls , and then use the dot product between the mask class embedding and the text embedding to obtain the classification scores s for a set of open available classes C cls , the classification score is calculated as follows: where i∈[1,N] and j∈[1,C] are the indices of object affordance and text embedding, respectively, σ(·) is the sigmoid function, and cos(·; ·) represents the cosine similarity. is the temperature hyperparameter.
6. An open vocabulary affordance segmentation device based on object shape mask hints, characterized in that: The device includes: Generate object shape mask prompt module. Given an input image, first use a pre-trained object detector to detect the object category; then, input the object category and image into the image segmentation model to obtain a refined target object shape mask; in order to obtain fine-grained text prompts, the mask and the original image are input into Alpha-CLIP to output caption descriptions specific to the mask area and obtain captions specific to the target object area; then use these captions and object categories, as well as an open affordance vocabulary as joint text prompts, and use a text encoder to embed them into the feature space; The mask hint feature enhancement module uses a mask to remove the background and obtains a mask image containing only a single object. The mask image is used to extract the object instance features. The features of the object rectangular box are extracted. The rectangular box contains the object and background information. In the availability prediction module, the encoded instance features are concatenated with the global image features and then input into the Transformer-based pixel decoder together with the textual hint features, which outputs category-agnostic availability mask proposals; these proposals are multiplied with the text embedding to obtain the availability category of the mask, and the final availability segmentation map is obtained by combining the availability category and the mask.
7. The device for segmenting open vocabulary affordance based on object shape mask prompts according to claim 6, characterized in that: In the object shape mask prompting module, the object detection process is described as follows: S obj =DETR(I;Q) (1) Where S obj represents the detected object score, I is the input image, Q is the learnable query parameter initialized in DETR, and the results with prediction scores greater than the threshold T = 0.7 are selected to be kept, and finally the results that are not in the object set O are removed. t The objects in the final object set O are obtained s The specific process is described as follows: O s ={S obj >T|Class(obj)∈O t } (2) Where Class(obj) represents the name of the detected object category. The segmentation process of SAM is as follows: M os =SAM(I;Class(obj)∈O s ) (3) Among them, M os Represents a binary object shape mask, I is the input image, obj is the target object to be detected; Use the Alpha-CLIP model text prompt to guide the affordance mask prediction, taking the object mask as input and outputting the text description of the mask area. The generation process of the Alpha-CLIP text prompt is as follows: w mask =Alpha-CLIP(I;M os ) (4) Among them, w mask ={sen1,sen2,…,sen n } represents the text hint of the object shape mask area.
8. The device for segmenting open vocabulary affordance based on object shape mask prompts according to claim 7, characterized in that: In the mask hint feature enhancement module, the object shape mask obtained based on MPGM For the original input image Masking; occluded image Remove the background information, only include a single object instance, and fill the pixels in other areas with the average value of the image pixels, where m represents the number of occluded images; send these images and the original image to the backbone network to obtain the features respectively. and F o , then concatenate these features and reduce the number of channels through a convolutional layer to obtain enhanced visual features F v , F v The calculation process is as follows: Among them, Cat[·] represents the channel concatenation operation, and C(·) represents a 1×1 convolution.
9. The device for segmenting open vocabulary affordance based on object shape mask prompts according to claim 8, characterized in that: In the affordance prediction module, a Transformer-based visual-textual cross-modal pixel decoder is designed to generate proposals, which accepts visual features and text embeddings and outputs a set of affordance mask proposals.
10. The device for segmenting open vocabulary affordance based on object shape mask prompts according to claim 9, characterized in that: In the availability prediction module, the text prompts in the mask prompt generation module are converted into the embedding space, and the word w of the object is introduced obj = {obj1, obj2, …, obj n }, and all affordances w aff ={aff1,aff2,…,aff c }; the former provides the features of the instance, and the latter provides all affordance classes, as well as categories not present in the training set, thereby achieving open-world affordance segmentation; these joint text titles are then converted into an embedded space using CLIP’s text encoder, as follows: text join =CLIP.tokenize(w obj ;w aff ;w mask ) (6) F t =CLIP.encode-text(text join ) (7) Among them, text join represents the joint text embedding embedded by the tokenize function of CLIP, F t It represents the joint text prompt embedded by the encode-text function of CLIP; Then, the visual feature F v and text embedding F t Input to the pixel decoder, first the visual feature F v Perform self-attention processing and then combine it with the text feature F t Perform cross attention processing, and finally output the fused visual-text attention features through the feedforward network FFN. FFN consists of two fully connected layers with a nonlinear mapping GELU inserted in the middle. After L iterations, the final refined features will be output. The specific operations are as follows: Q v ,K v ,V v =Linear(F v ) (8) K t ,V t =Linear(F t ) (9) Q t =Norm(SA(Q v ,K v ,V v )) (10) F vt =FFN(K t +Norm(CA(Q t ,K t ,V t ))) (11) in, is a learnable weight, Norm represents L2 normalization, SA and CA represent self-attention and cross-attention respectively, which follow a unified formula, Finally, F vt are fed into the MLP layer to generate class-agnostic affordance mask proposals respectively and mask class embedding F cls , and then use the dot product between the mask class embedding and the text embedding to obtain the classification scores s for a set of open available classes C cls , the classification score is calculated as follows: where i∈[1,N] and j∈[1,C] are the indices of object affordance and text embedding, respectively, σ(·) is the sigmoid function, and cos(·; ·) represents the cosine similarity. is the temperature hyperparameter.
Citation Information
Patent Citations
Image processing method and device, storage medium and electronic equipment
CN116664604A
Visual positioning method based on multi-modal feature alignment
CN117934803A
Space ranging method based on fine segmentation
CN118351189A
Open vocabulary instance segmentation with noise estimation and robust student
US20230401827A1
