A cross-task general foreground segmentation method

Abstract

The application relates to the field of computer vision and discloses a foreground segmentation method which is universal across tasks. First, a unified foreground segmentation framework is constructed based on a multi-scale strategy and a mask attention mechanism. A binary query is introduced to represent foreground and background features in an image through foreground and background queries. An edge enhancement module is used to extract edge information of the image by using a convolutional neural network, and the edge features are fused with multi-scale features of the image by combining a multi-scale deformation attention mechanism. The multi-scale features and the binary query are input into a Transformer decoder, a mask attention mechanism is applied, the binary query is updated, and accurate foreground and background segmentation masks are obtained. A multi-modal contrast learning strategy is used to refine the segmentation results of the foreground and the background, so that the accuracy of the segmentation boundary and the detail retention effect are improved. The method can be widely applied to different types of foreground segmentation tasks and can achieve high-precision segmentation results in multiple complex scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a cross-task general foreground segmentation method based on multi-modal learning. BACKGROUND

[0002] Foreground segmentation is an important task in computer vision, aiming to distinguish significant objects (foreground) from the rest (background) in an image. This task encompasses multiple subfields such as salient object detection, camouflage object detection, shadow detection, focus blur detection, and forgery detection. Traditionally, these tasks often employ specially designed architectures, lacking generalizability, making it difficult to share and extend between different tasks. Moreover, these methods mainly focus on the identification of foreground, often neglecting the background and the relationship between foreground and background, resulting in poor segmentation results.

[0003] In the field of computer vision, some general segmentation tasks such as instance segmentation, semantic segmentation, and panoptic segmentation have made significant progress, with many advanced models performing well in these tasks. However, these models often lack specific optimization for certain foreground segmentation tasks. For example, in camouflage object detection, existing general models perform poorly in distinguishing camouflage objects from the background. At the same time, traditional segmentation methods often generate multiple redundant masks for the same image, while users in practical applications often only need specific foreground masks, such as image background removal, making these methods perform poorly in real-world scenarios.

[0004] Although there have been some attempts in recent years to achieve generalization between salient object detection and camouflage object detection, the scope of these methods is still limited, especially when facing different types of foreground segmentation tasks, they still cannot compare with task-specific models. In addition, most foreground segmentation models mainly focus on foreground objects, without fully utilizing background information and its relationship with foreground. In traditional segmentation tasks, background is often a key category, therefore, in foreground segmentation tasks, how to effectively utilize background information to optimize segmentation results is a problem that needs to be solved.

[0005] Therefore, based on the challenges faced by existing technology, it is urgent to develop a framework that can uniformly handle multiple foreground segmentation tasks and fully utilize background information to improve the accuracy and robustness of segmentation. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a cross-task general foreground segmentation method, which effectively realizes the uniform processing of multiple foreground segmentation tasks by combining multi-scale strategy, mask attention mechanism, edge enhancement module and multi-modal contrast learning of CLIP model, and significantly improves the accuracy and detail expression of segmentation boundaries.

[0007] The object of the present application can be achieved by the following technical solutions:

[0008] A foreground segmentation method for cross-task generalization, the method steps comprising:

[0009] A unified foreground segmentation model is constructed, and binary queries are introduced to represent foreground and background features in the image through foreground queries and background queries;

[0010] Multi-scale image features are extracted through a visual backbone network;

[0011] Edge features of the image are extracted through an edge enhancement module, and multi-scale deformation attention mechanisms are combined to fuse the edge features with the multi-scale image features to obtain edge-enhanced multi-scale image features;

[0012] The edge-enhanced multi-scale features are input into a Transformer decoder together with the binary queries, and the Transformer decoder applies a mask attention mechanism to update the binary queries, so that the binary queries focus on features related to the foreground and the background;

[0013] The binary queries are decoded to generate foreground and background segmentation masks and class predictions.

[0014] As a preferred technical solution, the edge enhancement module includes multiple pairs of edge feature injection modules and backbone feature extraction modules, and each level block of the backbone network corresponds to an edge injection module and a backbone feature extraction module;

[0015] The edge feature injection module extracts edge features of the image through a convolutional neural network, and the convolutional neural network uses ResNet50, including a STEM and multiple convolutional blocks, wherein the STEM is used for preliminary feature extraction; the feature maps output by the convolutional blocks are flattened and projected to the same dimension through 1x1 convolution, and are connected to form a feature pyramid of edge features An edge feature injection structure based on cross-attention is used to fuse the image features extracted by the backbone network DINOv2 with the edge features;

[0016] The backbone feature extraction module applies multi-scale deformation attention and further processes the features fused by the edge feature injection module through multiple convolutional blocks to obtain enhanced foreground and background feature representations.

[0017] As a preferred technical solution, the edge feature injection is represented as:

[0018]

[0019] wherein MSDA represents multi-scale deformable attention, and the normalized edge feature of the i-th layer of the backbone network is taken as the query, the output feature of the next layer of the backbone network is taken as the key and the value; and γ is a learnable parameter for balancing the backbone feature and the fused feature. wherein the normalized edge feature is taken as the query, and the output feature of the next layer of the backbone network is taken as the key and the value. wherein the normalized edge feature is taken as the query, and the output feature of the next layer of the backbone network is taken as the key and the value. i is a learnable parameter for balancing the backbone feature and the fused feature.

[0020] As a preferred technical solution, the backbone feature extraction module is represented as:

[0021]

[0022] wherein MSDA represents multi-scale deformable attention, and the normalized edge feature of the i-th layer of the backbone network is taken as the query, the output feature of the next layer of the backbone network is taken as the key and the value; and γ is a learnable parameter for balancing the backbone feature and the fused feature. wherein the normalized edge feature is taken as the query, and the output feature of the next layer of the backbone network is taken as the key and the value. wherein the normalized edge feature is taken as the query, and the output feature of the next layer of the backbone network is taken as the key and the value. The output of the next layer of the edge feature is injected into the input of the module.

[0023] As a preferred technical solution, the edge enhancement module performs multi-scale feature fusion on the edge feature and the image feature and outputs, specifically as follows:

[0024] The features output by different blocks of the backbone network are upsampled to resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32, the output of the backbone feature extraction module is split and restored to the original size, then the upsampled backbone features are added to the corresponding split output from the backbone feature extraction module and the STEM output, to obtain the multi-scale image features enhanced by the edge.

[0025] As a preferred technical solution, the Transformer decoder applies a masked attention mechanism to update the binary query, specifically as follows:

[0026] The multi-scale image features extracted by the edge enhancement module are input into a pixel decoder based on multi-scale deformable attention, and the pixel decoder performs dense pixel-level prediction in combination with the feature maps generated by the backbone network, to generate pixel-level output features;

[0027] The pixel-level output features are input into the Transformer decoder together with the binary query, and the binary query is updated through the masked attention mechanism, and the process is represented as:

[0028]

[0029] wherein K l and V l are image features linearly transformed from the l-th layer block of the pixel decoder, and both belong to space. denotes the query feature extracted from the l-th Transformer decoder block, X0 is initialized by the input query feature of the Transformer decoder; is the binary query of the l-th layer, is the attention mask of the previous layer, defined as follows:

[0030]

[0031] wherein, by decoding the binary query GQ l-1 and performing binary processing, and the dimension size is consistent with K l .

[0032] As a preferred technical solution, the method initializes the attention mask by performing principal component analysis and binary processing on the feature map of the last block of the backbone network

[0033]

[0034] wherein, F DINOv2 denotes the binary feature map from the last backbone network block of DINOv2, which is adjusted to the same resolution as K1.

[0035] As a preferred technical solution, the foreground segmentation model further adopts a multi-modal refinement module to refine the segmentation results of the foreground and background, specifically as follows:

[0036] decode the binary query to obtain foreground and background masks, and adjust the size of the foreground and background masks to superimpose them on the original image to form a mask fusion image;

[0037] use text descriptions to represent the foreground and background, respectively;

[0038] use the image encoder and text encoder of the CLIP model to encode the mask fusion image and the text description, respectively;

[0039] calculate the contrastive loss between the mask fusion image and the text description in the multi-modal feature space refine the segmentation results of the foreground and background by minimizing the contrastive loss.

[0040] As a preferred technical solution, the contrastive loss between the mask fusion image and the text features is specifically as follows:

[0041]

[0042]

[0043]

[0044] where, is the contrastive loss from image to text; is the contrastive loss from text to image;I f ,I b ,T f ,T b ∈R N×C and represent the image and text features of foreground and background obtained by CLIP, respectively; τ is the temperature parameter for controlling the smoothness of the softmax function.

[0045] As a preferred technical solution, the multi-modal refinement module optimizes the boundaries of the foreground and background masks through multiple iterations in the training stage to achieve the best effect; in the inference stage, the multi-modal refinement module is removed, and only the optimized foreground segmentation model is retained.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] 1) The present application proposes a unified foreground segmentation model architecture, which represents the features of foreground and background as foreground and background queries, respectively, and inputs the visual features extracted from the image by the visual feature extraction module through a multi-scale strategy into the Transformer decoder together with the foreground and background queries, applies the mask attention mechanism, so that the foreground and background queries focus on the foreground and background information in the image, respectively. Then, the multi-layer perceptron is used to decode these query features to generate the final foreground and background masks and the corresponding class predictions. The binary query is a general feature with strong generalization ability, which can learn the corresponding foreground and background information according to the context of different tasks.

[0048] 2) The present application constructs an edge enhancement module to perform multi-scale edge enhancement features, extracts the edge information of the image through the convolutional neural network, combines the multi-scale deformation attention mechanism, and fuses the edge features with the multi-scale features of the image, thereby enhancing the expression ability of the foreground and background in the image. It can effectively fuse the global features and edge information of the image, thereby significantly improving the clarity of the boundary and the accuracy of the segmentation in the foreground segmentation task.

[0049] 3) The present application also proposes a multi-modal refinement module, which iteratively refines the generated mask edges to ensure that only appropriate pixels are included in the foreground and background. This process makes the mask fused image closer to the corresponding text in the feature space while maintaining a distance from the mismatched text features. This not only makes the mask edges more accurate, but also further increases the gap between the foreground and the background. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1A flowchart of a cross-task general foreground segmentation method of the present application is shown in the figure.

[0051] Figure 2 A structural diagram of an edge enhancement module for extracting and fusing image edge features of the present application is shown in the figure.

[0052] Figure 3 A workflow diagram of a multi-modal refinement module of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation methods and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0054] Embodiment 1

[0055] The present application aims to solve the problem that the existing foreground segmentation method lacks uniformity and is difficult to effectively distinguish foreground and background. The proposed general multi-modal foreground segmentation method has an overall process as shown in the figure. Figure 1

[0056] First, a unified foreground segmentation framework is constructed based on a multi-scale strategy and a mask attention mechanism. By introducing the concept of binary query (GQ), the features of foreground and background are represented as foreground query and background query, respectively. The binary query is a pair of learnable tensors, which is input into the Transformer decoder together with the visual features extracted from the image by the multi-scale strategy of the visual feature extraction module, and the mask attention mechanism is applied to make the foreground query and the background query focus on the foreground and background information in the image, respectively. The binary query is randomly initialized, and the binary query learns the corresponding features during the forward propagation process. Then, the multi-layer perceptron is used to decode these query features to generate the final foreground and background masks and the corresponding class predictions. The binary query is a general feature with strong generalization ability, which can learn the corresponding foreground and background information according to the context of different tasks. The constructed unified foreground segmentation model can handle different foreground segmentation tasks including salient object detection, camouflage object detection, shadow detection, focal blur detection, and forgery detection.

[0057] Secondly, an edge enhancement module is proposed, which extracts the edge information of the image through a convolutional neural network (CNN), combines a multi-scale deformation attention mechanism, and fuses the edge features with the multi-scale features of the image to enhance the edge features of the image. This fusion can improve the accuracy of the segmentation boundary.

[0058] ​Finally, the application also proposes a multi-modal refinement module, which further refines the generated foreground and background segmentation results through multi-modal contrast learning of the CLIP model. The foreground and background masks are fused with the image to generate a mask fusion image, and the multi-modal contrast learning strategy is used to optimize the segmentation boundaries of the foreground and background. This refinement process significantly improves the detail expression and boundary clarity of the segmentation results, improves the accuracy and detail preservation effect of the segmentation boundaries, and can be selectively removed during the inference stage to improve the segmentation efficiency.

[0059] The method of the application can be widely applied to different types of foreground segmentation tasks and achieve high-precision segmentation results in multiple complex scenes.

[0060] The application proposes a FOCUS architecture based on a multi-scale strategy and a mask attention mechanism, which can handle various foreground segmentation tasks and is suitable for tasks such as salient object detection, camouflage object detection, shadow detection, focal blur detection, and forgery detection. The concept of binary query (GQ) is introduced. Binary query consists of two independent vectors, foreground query and background query, which are used to embed and represent foreground and background in images in a specific task context.

[0061] (S1.1) extracting image features from a DINOv2-based backbone network;

[0062] (S1.2) correcting the image features extracted by the backbone network using the edge information of the foreground object through an edge enhancement module, the edge enhancement module extracts the edge information of the image through a convolutional neural network, and combines a multi-scale deformation attention mechanism to fuse the edge features with the image features extracted by the backbone network to obtain multi-scale features;

[0063] (S1.3) inputting the multi-scale features into a Transformer decoder, applying a mask attention mechanism in the Transformer decoder to focus the binary query on features related to the foreground and background, and obtaining accurate foreground and background segmentation masks;

[0064] (S1.4) decoding the binary query through a multi-layer perceptron to generate foreground and background segmentation masks and class predictions;

[0065] (S1.5) comparing the generated segmentation masks with the real labels, and optimizing the segmentation masks using a binary cross-entropy loss function and a Dice loss function.

[0066] The step of optimizing the image features through the edge enhancement module is as follows:

[0067] (S2.1) using a ResNet50 convolutional neural network to extract edge information of the image, the convolutional neural network is composed of a STEM and a plurality of convolutional blocks, wherein the STEM is used for preliminary feature extraction, and the convolutional blocks are used for extracting multi-scale edge features;

[0068] (S2.2) the extracted edge features are fused with the image features generated by the DINOv2 backbone network through a multi-scale deformation attention mechanism, and the deformation attention mechanism enhances the expression ability of the foreground and background features through a multi-scale spatial transformation operation;

[0069] (S2.3) the fused features are further processed through a plurality of convolutional blocks to obtain enhanced foreground and background feature expressions, thereby improving the accuracy of foreground segmentation.

[0070] (S2.4) in the fused features, through the iterative fusion operation of the multi-scale spatial dimension and the channel dimension, the local spatial information and the global semantic information of the image features are further optimized, so that the segmentation boundary between the foreground and the background is clearer and more accurate.

[0071] The application also refines the segmentation results of the foreground and the background through a multi-modal refinement module, and the steps are as follows:

[0072] (S3.1) fuse the foreground and background masks generated by the binary query with the original image to form a mask fusion image;

[0073] (S3.2) using the image encoder and the text encoder of the CLIP model, the mask fusion image and the preset text description are encoded respectively, and the text description can be replaced according to different processing tasks;

[0074] (S3.3) calculate the contrastive loss of the mask fusion image and the text description in the multi-modal feature space, and refine the segmentation results of the foreground and the background by minimizing the contrastive loss, thereby improving the accuracy and detail performance of the segmentation boundary.

[0075] In the training stage, the multi-modal refinement module optimizes the boundaries of the foreground and background masks through multiple iterations to achieve the best effect; in the inference stage, the multi-modal refinement module can be removed, and only the optimized foreground segmentation model is retained to improve the inference efficiency.

[0076] Wherein, the implementation of the unified foreground segmentation model proposed by the application is as follows:

[0077] The application selects DINOv2 as the backbone network of the FOCUS architecture to extract image features. DINOv2 is a Transformer-based self-supervised learning model that uses multi-head self-attention mechanisms to analyze global dependencies in images, effectively capturing rich semantic information. Visualization of its feature maps shows that the model has been able to focus on salient objects in images without supervision. In the application, we use the features of each level block of DINOv2 to extract multi-scale image features, and initialize the mask attention mechanism of the first layer using the binarized feature map of the last backbone network block. This processing strategy effectively enhances the model's ability to recognize key visual elements.

[0078] As Figure 2 shown, the edge enhancement module of the application aims to use edge information of foreground objects to correct image features extracted by the backbone network, thereby improving the accuracy of foreground segmentation. The edge enhancement module includes two key parts: edge feature injection module and backbone feature extraction module, and each DINOv2 backbone network block block corresponds to an edge injection module and a backbone feature extraction module.

[0079] 2.1 Edge feature injection module

[0080] First, the edge features of the image are extracted by the ResNet50 convolutional neural network. To reduce the confusion caused by color, the image is converted to a grayscale image, and Gaussian smoothing is applied to reduce noise. Then, the Canny edge detector is used to extract the gradient map, which is superimposed on the original image. The ResNet50 network is divided into STEM and the rest, where STEM is used for preliminary feature extraction, including a series of convolution, batch normalization and ReLU activation layers. The feature maps output by the remaining convolution blocks are flattened and projected to the same dimension D by 1x1 convolution, and are connected into a feature pyramid of edge features

[0081] Next, the edge feature injection structure based on cross-attention is used to fuse the image features extracted by the backbone network DINOv2 with the edge features. The formula of the edge feature injection part is:

[0082]

[0083] where MSDA (Multi-Scale Deformable Attention) takes the normalized output features of the i-th layer of the backbone network as the query (query), and the normalized edge features as the key (key) and value (value). γ iis a learnable parameter for balancing backbone features and fusion features.

[0084] 2.2 Backbone feature extraction module

[0085] After the edge feature injection module is completed, the fused features are further processed using the backbone feature extraction module. The formula of the backbone feature extraction module is:

[0086]

[0087] Here, MSDA is applied again, however, this time with normalized edge features as queries, the output features of the next layer of DINOv2 as keys and values. The ConvFFN structure includes two fully connected layers and a depthwise separable convolution layer. The output will be the input of the next edge feature injection module.

[0088] 2.3 Multi-scale feature fusion and output

[0089] Finally, the features output by different blocks of the backbone network are upsampled to resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. The output of the backbone feature extraction module is split and restored to the original size, and then the upsampled backbone features are added to the corresponding split output from the backbone feature extraction module and the STEM output, obtaining edge-enhanced multi-scale image features. These features will be input to a pixel decoder based on multi-scale deformation attention for dense pixel-level prediction.

[0090] In this way, the edge enhancement module can effectively fuse the global features and edge information of the image, thereby significantly improving the clarity of the boundary and the accuracy of the segmentation in the foreground segmentation task.

[0091] After obtaining the multi-scale edge-enhanced features extracted by the backbone network and the edge enhancement module, these pixel-level features will be input into the pixel decoder to generate pixel-level output. These output features will be input into the Transformer decoder together with the binary query, which will be updated through the masked attention mechanism. This process can be represented as:

[0092]

[0093] where K l and V l are image features obtained by linear transformation from the lth block of the pixel decoder, both belonging to space; denotes the query feature extracted from the l-th Transformer decoder block, X0 is initialized by the input query feature of the Transformer decoder; is the binary query of the l-th layer, is the attention mask of the last layer, defined as follows:

[0094]

[0095] where, is obtained by decoding the binary query GQ l-1 and performing binary processing, and its dimension size is consistent with K l .

[0096] The application initializes the attention mask by principal component analysis and binary processing of the feature map of the last block of the backbone network

[0097]

[0098] where, F DINOv2 denotes the binary feature map from the last backbone network block of DINOv2, which is adjusted to the same resolution as K0. This new initialization method can utilize the positioning prior knowledge learned by DINOv2 on large-scale data, thereby further improving the computational efficiency of the mask attention mechanism.

[0099] The implementation of the multi-modal refinement module in the application is shown in Figure 3 , which first decodes the binary query through a multi-layer perception to obtain the masks of the foreground and background, and then superimposes these masks on the original image after adjusting their sizes. Then, we use text descriptions to represent the foreground and background, such as "this is a salient object image without background" and "this is a background image after removing the salient object". It should be noted that these text descriptions can be adjusted according to specific tasks. For example, in the shadow detection task, the text can be replaced with "this is a shadow image without background" and "this is a background image after removing the shadow", thereby extending the multi-modal refinement module to other foreground segmentation tasks.

[0100] In the encoding process of images and texts, the image encoder and text encoder of CLIP are adopted to encode the fusion image and text, respectively. Then, the contrastive loss between the mask fusion image and the text features is calculated.

[0101] 3.1. Image-to-text contrastive loss

[0102]

[0103] 3.2. Text-to-image contrastive loss

[0104]

[0105] The final contrastive loss is the average of the above two contrastive losses:

[0106]

[0107] where I f ,I b ,T f ,T b ∈R N×C denote the image features and text features of the foreground and background obtained by CLIP, respectively, and τ is the temperature parameter used to control the smoothness of the softmax function.

[0108] The multi-modal refinement module refines the mask edges generated by the previous module iteratively, ensuring that only appropriate pixels are included in the foreground and background. This process makes the mask-fused image closer to the corresponding text in the feature space while maintaining a distance from the mismatched text features. This not only makes the mask edges more accurate, but also further increases the gap between the foreground and background.

[0109] It is worth noting that the multi-modal refinement module is only used to distill knowledge from CLIP and will be discarded during the inference stage. In addition, in order to fully utilize the multi-modal capabilities of CLIP, we keep the parameters of the image and text encoders completely frozen, avoiding potential performance degradation due to fine-tuning.

[0110] The above detailed description of the preferred embodiments of the present application. It should be understood that those skilled in the art without creative labor can make many modifications and changes according to the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment on the basis of the prior art according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. A foreground segmentation method that is task-agnostic, characterized in that, The method steps include: A unified foreground segmentation model is constructed, binary queries are introduced, and foreground and background features in the image are represented through foreground queries and background queries; Multi-scale image features are extracted through a visual backbone network; Edge features of the image are extracted through an edge enhancement module, and the edge features are fused with the multi-scale image features to obtain edge-enhanced multi-scale image features through a multi-scale deformation attention mechanism; The edge-enhanced multi-scale features and the binary queries are input into a Transformer decoder, and the Transformer decoder updates the binary queries through a mask attention mechanism, so that the binary queries focus on features related to the foreground and the background; The binary queries are decoded to generate foreground and background segmentation masks and class predictions.

2. The task-agnostic foreground segmentation method of claim 1, wherein, The edge enhancement module includes multiple pairs of edge feature injection modules and backbone feature extraction modules, and each level block of the backbone network corresponds to an edge injection module and a backbone feature extraction module; The edge feature injection module extracts edge features of an image through a convolutional neural network, the convolutional neural network adopts ResNet50, and includes a STEM and a plurality of convolutional blocks, wherein the STEM is used for preliminary feature extraction; feature maps output by the convolutional blocks are flattened and projected to the same dimension through 1*1 convolution, and are connected into a feature pyramid of edge features An edge feature injection structure based on cross attention is used to fuse image features extracted by a backbone network DINOv2 and edge features. The backbone feature extraction module applies a multi-scale deformation attention and further processes the features fused by the edge feature injection module through multiple convolution blocks to obtain enhanced foreground and background feature expressions.

3. The task-agnostic foreground segmentation method of claim 2, wherein, The edge feature injection is represented as: wherein MSDA denotes a multi-scale deformation attention to the output feature of the i-th layer of the normalized backbone network as the query, to the normalized edge feature as the key and value; γ i is a learnable parameter for balancing the backbone feature and the fused feature.

4. The task-agnostic foreground segmentation method of claim 2, wherein, The backbone feature extraction module is represented as: wherein MSDA denotes multi-scale deformation attention to normalized edge features as the query, the output features of the backbone network next layer as the key and value; the ConvFFN structure includes two fully connected layers and a depthwise separable convolution layer, and the output will be injected as the input of the next layer edge features into the module.

5. The task-agnostic foreground segmentation method of claim 2, wherein, The edge enhancement module performs multi-scale feature fusion and output on the edge features and the image features, specifically as follows: The features output by different blocks of the backbone network are upsampled to resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, the output of the backbone feature extraction module is split and restored to the original size, then the upsampled backbone features are added to the corresponding split output and the STEM output from the backbone feature extraction module to obtain edge-enhanced multi-scale image features.

6. The task-agnostic foreground segmentation method of claim 1, wherein, The Transformer decoder updates the binary queries through a mask attention mechanism, specifically as follows: The multi-scale image features extracted by the edge enhancement module are input into a pixel decoder based on multi-scale deformation attention, and the pixel decoder generates dense pixel-level predictions combined with the feature maps generated by the backbone network to generate pixel-level output features; The pixel-level output features and the binary queries are input into the Transformer decoder, and the binary queries are updated through a mask attention mechanism, and the process is represented as: where K l and V l are image features linearly transformed from the i-th layer block of the pixel decoder, both belong to space; denotes the query feature extracted from the i-th layer Transformer decoder block, X0is initialized by the input query feature of the Transformer decoder; is the binary query of the i-th layer, is the attention mask of the previous layer, defined as follows: wherein, By decoding the binary query GQ l-1 And binary processing is obtained, and the dimension size is consistent with K l Keep.

7. The task-agnostic foreground segmentation method of claim 6, wherein, The method initializes the attention mask by principal component analysis and binarizing feature maps of a last block of the backbone network where F DINOv2 denotes the binarized feature map from the last backbone network block of DINOv2, which is adjusted to the same resolution as K1.

8. The task-agnostic foreground segmentation method of claim 1, wherein, The foreground segmentation model also uses a multi-modal refinement module to refine the segmentation results of the foreground and the background, specifically as follows: The binary queries are decoded to obtain foreground and background masks, and the foreground and background masks are resized and superimposed on the original image to form a mask fusion image; Text descriptions are used to represent the foreground and the background respectively; Image encoders and text encoders of a CLIP model are used to encode the mask fusion image and the text descriptions respectively; Computing a contrastive loss between the masked-fusion image and the textual description in a multi-modal feature space The segmentation results of the foreground and the background are refined by minimizing the contrastive loss.

9. The task-agnostic foreground segmentation method of claim 8, wherein, The contrast loss between the mask fusion image and the text features As follows: wherein, is the contrastive loss for image-to-text; is the contrastive loss for text-to-image; f ,I b ,T f ,T b ∈R N ×C and denote the image features and text features of foreground and background obtained by CLIP, respectively; τ is a temperature parameter used to control the smoothness of the softmax function.

10. The task-agnostic foreground segmentation method of claim 8, wherein, In the training stage, the multi-modal refinement module optimizes the boundaries of the foreground and background masks through multiple iterations to achieve the best effect; in the inference stage, the multi-modal refinement module is removed, and only the optimized foreground segmentation model is retained.

Images