Camouflage target segmentation method and system based on mask separable attention mechanism

Abstract

The application provides a camouflage target segmentation method and system based on a mask separable attention mechanism, relates to the technical field of image processing, and specifically includes the following steps: constructing a camouflage target segmentation model and training the camouflage target segmentation model; encoding an input image by using an encoder to obtain initial features; inputting the initial features into a decoder to decode and generate initial segmentation of a camouflage target; and based on a mask separable attention mechanism, gradually optimizing the initial segmentation to obtain a final camouflage target segmentation result; the mask separable attention mechanism is provided, the attention mechanism is divided into foreground attention, background attention and global attention by using a mask, the foreground and background attention respectively focus on the foreground and background regions, the global attention mechanism extracts global information, the foreground attention score is used to find a camouflage object in the global information, the background attention is used to determine the background, and then an accurate camouflage target is segmented.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a camouflage target segmentation method and system based on a mask separable attention mechanism. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In recent years, deep learning algorithms have brought rapid development to the task of camouflage target detection; however, the high similarity between camouflage targets and their surroundings makes camouflage target detection more challenging than traditional target detection; therefore, in order to improve the detection accuracy of camouflage targets, many methods using more complex deep learning techniques have appeared in recent years to solve this problem; In the training of camouflage target detection, pixel-level labeling is used as supervision to train the network; Early traditional camouflage target detection extracts hand-crafted features between camouflage targets and backgrounds for prediction, which can only handle relatively simple scenes; Later, methods based on convolutional neural networks became the mainstream, which can be divided into three categories: multi-scale feature fusion, multi-stage training and detection, and multi-task collaborative training; Multi-scale feature fusion, Mingcheng Zhuge et al. in "Cubenet: X-shape connection for camouflaged object detection" combines attention fusion and X-type connection to effectively aggregate multi-level features; Youwei Pang et al. in "Zoom in and out: A mixed-scale triplet network for camouflaged object detection" input three different scale pictures, process them at different scales respectively, then aggregate the features according to the middle scale, and then group the features for interactive fusion; Multi-stage training and detection, Deng-Ping Fan et al. in "Camouflaged object detection" propose to first locate the target and then detect it to optimize the model performance; Qi Jia et al. in "Segment, magnify and reiterate: Detecting camouflaged objects the hard way" construct a multi-stage training and detection framework, which locates the target and then further magnifies it to detect the camouflage target; Miao Zhang et al. in "Preynet: Preying on camouflaged objects" simulate the process of predation, dividing the detection process of camouflage targets into initial detection and predator learning, thereby increasing the ability to capture camouflage targets; Deng-Ping Fan et al. in "Concealed object detection" use group inversion attention and surrounding connection decoder to improve performance; Multi-task collaborative training, Aixuan Li in "Uncertainty-aware joint salient object and camouflaged object detection" improves the performance of the model in both tasks by comparing salient targets and camouflage targets;Yunqiu Lv et al. in "Simultaneously localize, segment and rank the camouflaged objects" combine camouflage ranking and camouflage object detection to build a joint training framework.

[0004] Recently, with the rapid development of Transformer in vision tasks, Transformer-based methods become more and more popular; compared with traditional convolutional neural network methods, Transformer can effectively encode global information, so it is widely used in a large number of vision tasks; in camouflage object detection, Transformer-based methods have become a new trend; under the Transformer framework, Fan Yang et al. in "Uncertainty-guided transformer reasoning for camouflaged object detection" explicitly use probabilistic representation models to learn the uncertainty of camouflage objects to improve the segmentation quality of the target; Zhengyi Liu et al. in "Boosting camouflaged object detection with dual-task interactive transformer." designed a dual-task interactive Transformer to segment the camouflage object and its detailed information boundary to improve the detection accuracy of both tasks; Qiao Zhang et al. in "Tprnet: camouflaged object detection via transformer-induced progressive refinement network" proposed a Transformer-based progressive refinement network that uses semantic information from high-level features to guide the detection of camouflage objects; in addition, under the Transformer-based framework, Xuelian Cheng et al. in "Implicit motion handling for video camouflaged object detection." use short-term dynamics and long-term temporal consistency to capture camouflage objects in dynamic videos.

[0005] The above methods complete the task of refining the attention image from a global perspective, and the problems are that: the existing methods process the foreground and background information without distinction, and in the camouflage target detection scene, the surrounding background is often highly similar to the target, which makes it difficult for the network to distinguish the camouflage target from the highly similar background; therefore, there is an urgent need for a camouflage target detection method that separately processes foreground and background information to improve the accuracy of camouflage target segmentation. SUMMARY

[0006] To overcome the above-mentioned deficiencies of the prior art, the present application provides a camouflage target segmentation method and system based on a mask separable attention mechanism, which sets up a mask separable attention mechanism, uses a mask to separate the attention mechanism into foreground attention, background attention and global attention, the foreground and background attention respectively focus on the foreground and background areas, the global attention mechanism extracts global information, then uses the foreground attention score to find the camouflage object in the global information, uses the background attention to determine the background, and further segments the accurate camouflage target.

[0007] To achieve the above object, one or more embodiments of the present application provide the following technical solutions:

[0008] The first aspect of the present application provides a camouflage target segmentation method based on a mask separable attention mechanism;

[0009] The camouflage target segmentation method based on the mask separable attention mechanism comprises:

[0010] A camouflage target segmentation model composed of an encoder and a decoder is constructed, and the camouflage target segmentation model is trained;

[0011] The trained encoder is used to encode the input image to obtain initial features;

[0012] The initial features are input into the trained decoder to decode and generate an initial segmentation of the camouflage target;

[0013] In the trained decoder, based on the mask separable attention mechanism, the initial segmentation is progressively optimized and segmented to obtain a final camouflage target segmentation result;

[0014] The mask separable attention mechanism is used to focus on different parts of the image using different attention.

[0015] Further, the mask separable attention mechanism is used to separate the attention mechanism into foreground attention, background attention and global attention by using a mask, and to extract foreground region information, background region information and global information respectively.

[0016] Further, the progressive optimization and segmentation of the initial segmentation are as follows:

[0017] The previous generated intermediate segmentation is taken as a mask to limit foreground attention and background attention on the foreground and background, combined with global information and larger resolution features to generate the next intermediate segmentation, and the above process is repeated to finally generate the final camouflage target segmentation result.

[0018] Further, in the decoder, the feature fusion mode of element multiplication and sum connection is adopted.

[0019] Further, the camouflage target segmentation model is trained by using the real annotation of the image to supervise the intermediate segmentation and the final segmentation until the training stopping condition is met.

[0020] The second aspect of the present application provides an evaluation method for segmentation quality.

[0021] An evaluation method for segmentation quality is used to evaluate the segmentation quality of the camouflage target segmentation method based on the mask separable attention mechanism provided in the first aspect, comprising:

[0022] The edge of the target is extracted from the real annotation of the image to obtain an edge region;

[0023] The average absolute error and the F-measure between the real annotation and the prediction in the edge region are calculated, and the obtained index value is taken as the evaluation result of the segmentation quality.

[0024] Further, the edge region is obtained by using an edge search method to determine the edge of the target on the real annotation, and then using a dilation operation to dilate the edge into an edge region.

[0025] The third aspect of the present application provides a camouflage target segmentation system based on a mask separable attention mechanism.

[0026] The camouflage target segmentation system based on the mask separable attention mechanism comprises a model construction module, an image encoding module, an initial segmentation module and an optimized segmentation module.

[0027] The model construction module is configured to construct a camouflage target segmentation model composed of an encoder and a decoder, and to train the camouflage target segmentation model;

[0028] The image encoding module is configured to encode the input image using the trained encoder to obtain initial features;

[0029] The initial segmentation module is configured to input the initial features into the trained decoder to decode and generate the initial segmentation of the camouflage target;

[0030] The optimization segmentation module is configured to: in the trained decoder, based on the mask separable attention mechanism, gradually optimize the initial segmentation to obtain a final camouflage target segmentation result.

[0031] The mask separable attention mechanism is used to focus on different parts of the image.

[0032] The fourth aspect of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the steps in the camouflage target segmentation method based on the mask separable attention mechanism according to the first aspect of the present application.

[0033] The fifth aspect of the present application provides an electronic device, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to realize the steps in the camouflage target segmentation method based on the mask separable attention mechanism according to the first aspect of the present application.

[0034] The above one or more technical solutions have the following beneficial effects:

[0035] The present application designs a mask separable attention mechanism, in which background and foreground attention are used to obtain foreground and background area information respectively, and global attention is used to extract global information, the model enhances the resolution ability for foreground and background by capturing foreground and background information in the global information, and then generates accurate camouflage target segmentation.

[0036] The present application fuses different attentions, deeply explores the foreground and background clues in the image, captures foreground and background information in the global feature, and promotes the model to have strong resolution ability for foreground and background.

[0037] The present application generates an edge region mask to calculate two edge region limited indicators, and then evaluates the segmentation effect of the method on the edge.

[0038] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application.

[0040] Figure 1 The method flowchart of the first embodiment.

[0041] Figure 2This is a structural diagram of the camouflaged target segmentation model for the first embodiment.

[0042] Figure 3 This is a schematic diagram of foreground attention in the mask-separable attention mechanism of the first embodiment.

[0043] Figure 4 This is a schematic diagram of the Mask Separable Attention (MSA) operation in the first embodiment.

[0044] Figure 5 This is a diagram showing the camouflaged target segmentation effect of the first embodiment.

[0045] Figure 6 This is a diagram illustrating the polyp segmentation effect of the first embodiment.

[0046] Figure 7 This is a flowchart of the method in the second embodiment.

[0047] Figure 8 This is a schematic diagram of the expansion operation in the second embodiment.

[0048] Figure 9 This is a partial image of the second embodiment, along with its annotations, edges, and edge regions.

[0049] Figure 10 This is a system architecture diagram of the third embodiment. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] Example 1

[0052] This embodiment discloses a camouflaged target segmentation method based on a mask-separable attention mechanism;

[0053] like Figure 1 As shown, the camouflaged target segmentation method based on the mask-separable attention mechanism includes:

[0054] Step S101: Construct a camouflaged target segmentation model consisting of an encoder and a decoder, and train the camouflaged target segmentation model.

[0055] The structure of the camouflaged target segmentation model, such as Figure 2 As shown, it consists of an encoder and a decoder. The encoder encodes the input image and extracts multi-level initial features. The decoder uses the multi-level initial features to generate a coarse initial segmentation. Then, this initial segmentation is used as a mask. The mask can be used to separate attention mechanism to extract foreground and background information respectively, and to retrieve the target in global information. Then, multi-scale features are fused to progressively optimize the segmentation result.

[0056] A training sample set is constructed to train the camouflaged target segmentation model. The training sample set includes images and their ground truth annotations. The camouflaged target segmentation model takes images as input and the segmentation results of the images as output. During the training process, the ground truth annotations of the images are used to supervise the intermediate and final segmentation until the training stopping condition is met.

[0057] The training stopping condition here can be determined by the error represented by the loss function, or by the evaluation result obtained by the segmentation quality evaluation method disclosed in Example 2.

[0058] Step S102: Encode the input image using the trained encoder to obtain initial features.

[0059] like Figure 2 The encoder shown is built on a backbone network. In this embodiment, the Transformer, which is one of the backbone networks, is used to extract initial features. The encoder consists of four stacked Transformers and one convolutional module.

[0060] The original image I∈R H×W×3 The input is fed into the encoder, where it is first downsampled by four Transformers to generate four multi-level features, represented as follows: Then, the convolution module performs convolution on the combined feature matrices E2, E3, and E4 to generate feature E5. Feature E5 provides a more comprehensive understanding of the image content. Finally, E5 is used to enhance E2 to E4, and the encoder outputs the feature.

[0061] Step S103: Input the initial features into the trained decoder to decode and generate the initial segmentation of the camouflaged target.

[0062] Step S104: In the trained decoder, based on the mask separable attention mechanism, the initial segmentation is progressively optimized to obtain the final camouflaged target segmentation result.

[0063] In the decoder, the initial segment P5 of the camouflaged target is generated by E5 through a single convolutional operation. A progressive approach is used to optimize the initial segmentation, which consists of four stages. Each stage employs a mask-separable attention mechanism to enhance the model's ability to distinguish between foreground and background. The feature fusion in the first stage is represented as follows:

[0064] D4 = MSA(E5)·F up (E4)+F up (E4) (1)

[0065] Where MSA(·) represents the masked separable attention operation, F up(·) represents an up-sampling operation for shape matching, and the feature fusion of the last three stages is represented as:

[0066] D i = F up (MSA(D i+1 ))·E i +E i (2)

[0067] The embodiment adopts the feature fusion mode of element multiplication and matrix addition, specifically: first, calculate the element-by-element product between the features, and then use addition; compared with the existing method of using element summation operation or merging operation to fuse features from different feature levels, the feature fusion method of the embodiment will bring a relative improvement of 0.2%.

[0068] The segmentation of each stage is generated by the feature fusion generated in this stage through a convolution operation, that is, the intermediate segmentation P i (i=4, 3, 2) and the final camouflage target segmentation result P1 are generated by the feature fusion D i (i=4, 3, 2) and D1 through a convolution operation.

[0069] The mask separable attention mechanism is to use different attention to focus on different parts of the image, specifically: the mask is used to divide the attention mechanism into foreground attention, background attention and global attention, and the foreground region information, background region information and global information are extracted respectively; the mask is used to limit the foreground attention and background attention to the foreground and background, and the global information and the feature with larger resolution are combined to generate the next intermediate segmentation, and the above process is repeated, and finally the final camouflage target segmentation result is generated.

[0070] The mask separable attention mechanism uses different heads to be responsible for different functions, respectively perceives foreground and background information, and compares in the global, which can better distinguish the target. The embodiment adopts a lightweight variant of self-attention mechanism Multi-Dconv Head Transposed Attention, abbreviated as TA, which can be represented as:

[0071]

[0072] Wherein, Q, K, V are query matrix Query keyword matrix Key and value matrix Value, T is a matrix transpose operation, and a is a learnable parameter in the training process.

[0073] Figure 3 The foreground attention (F-TA) in the mask separable attention mechanism is given, which is defined as:

[0074]

[0075] Where, α F Q is a parameter that can be learned during training. F K F V F The foreground attention query matrix, keyword matrix, and value matrix are obtained by passing the input D through three different sets of two-layer convolutions, and then using the intermediate segments of the same stage as the foreground mask M. F , for Q F and K F Cover it up.

[0076] Similarly, given a background mask M B =1-M F M F The values ​​in the range are composed of 0 and 1, 1-M F This involves inverting the foreground values, i.e., negating them, to obtain the background mask M. B ; after background mask M B Due to occlusion, background attention will focus more on the background area; therefore, background attention (B-TA) is represented as:

[0077]

[0078] Where, α B Q is a parameter that can be learned during training. B K B V B The background attention matrix (Query), keyword matrix (Key), and value matrix (Value) are obtained by passing the input D through three different sets of two-layer convolutions. Then, Q is processed... B and K B Use background mask M B Cover it up.

[0079] Masking can separate attention operations such as MSA(·). Figure 4 As shown, features generated by three different attention mechanisms—foreground attention, background attention, and panoramic attention—are merged and then fused through a convolution operation. Figure 5 A partial segmentation effect diagram is shown.

[0080] The segmentation method in this embodiment can be extended to various segmentation tasks, including but not limited to: polyp segmentation, transparent object segmentation, and semantic segmentation; taking polyp segmentation as an example... Figure 6 The image shows the effect of polyp segmentation.

[0081] Example 2

[0082] The embodiment discloses an evaluation method for segmentation quality.

[0083] As shown in the drawings, an evaluation method for segmentation quality is used to evaluate the segmentation quality of the camouflage target segmentation method based on the mask separable attention mechanism disclosed in embodiment one, and comprises the following steps. Figure 7

[0084] Step S701: edges of the target are extracted from the real annotation of the image to obtain an edge region.

[0085] The boundary is determined by an edge search algorithm of a binary image, and then the edge is dilated by a dilation operation to obtain the edge region.

[0086] ∈{0, 1} H×W The dilation is an edge region BR∈{0, 1} H×W , Figure 8 A schematic diagram of the dilation operation is shown, Figure 9 A schematic diagram of the edge and the edge region (different dilation kernel sizes) is shown, and the dilation operation at point (i, j) can be defined as:

[0087] BR(i,j)=Max(B((i,j),r)) (6)

[0088] Wherein, r is the kernel size of the dilation operation, and the size of the edge region depends on the kernel size r of the dilation operation.

[0089] Step S702: the average absolute error and the F-measure between the real annotation and the prediction in the edge region are calculated, and the obtained index value is taken as the evaluation result of the segmentation quality.

[0090] The two indexes are the edge region defined average absolute error (BR-M) and the edge region defined F-measure (BR-F), which can evaluate the performance of the model on the edge region.

[0091] The average absolute error (BR-M) is defined as:

[0092]

[0093] Wherein, P and G are the prediction result and the real annotation of the image, BR is the edge region, P, G and BR are all pictures with H*W resolution, the points in BR belonging to the edge region are set to 1, and the others are 0; |P-G| is the absolute value difference between P and G, and the sum of |P-G| multiplied by BR on H*W is a number, and the denominator is the area of BR region.

[0094] The edge region defined F-measure (BR-F) is to calculate the accuracy and recall rate in the boundary region, and the BR-F is defined as: ​

[0095]

[0096] wherein Precision BR is the accuracy of the segmentation on the boundary region, Recall BR is the recall on the boundary region, and β is a parameter to balance the accuracy and the recall, which is usually set to 1. The above two indicators can well reflect the segmentation effect of the segmentation result on the boundary region.

[0097] Embodiment Three

[0098] The embodiment discloses a camouflage target segmentation system based on a mask separable attention mechanism.

[0099] As shown in Figure 10 , the camouflage target segmentation system based on the mask separable attention mechanism comprises a model construction module, an image encoding module, an initial segmentation module, and an optimized segmentation module:

[0100] The model construction module is configured to construct a camouflage target segmentation model composed of an encoder and a decoder, and train the camouflage target segmentation model.

[0101] The image encoding module is configured to encode an input image by using the trained encoder to obtain initial features.

[0102] The initial segmentation module is configured to input the initial features into the trained decoder to decode and generate an initial segmentation of a camouflage target.

[0103] The optimized segmentation module is configured to perform progressive optimized segmentation on the initial segmentation based on a mask separable attention mechanism in the trained decoder to obtain a final camouflage target segmentation result.

[0104] The mask separable attention mechanism is to use different attention to focus on different parts of the image.

[0105] Embodiment Four

[0106] The purpose of the embodiment is to provide a computer-readable storage medium.

[0107] The computer-readable storage medium stores a computer program, which is executed by a processor to implement the steps in the camouflage target segmentation method based on the mask separable attention mechanism according to Embodiment One of the present disclosure.

[0108] Embodiment Five

[0109] The purpose of the embodiment is to provide an electronic device.

[0110] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor implements the steps in the camouflage target segmentation method based on the mask separable attention mechanism as described in the embodiment one of the present disclosure when executing the program.

[0111] The above merely provides preferred embodiments of the present application, but not for limiting the present application. For the person skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A camouflage target segmentation method based on mask separable attention mechanism, characterized in that, The method comprises the steps of: constructing a camouflage target segmentation model composed of an encoder and a decoder, and training the camouflage target segmentation model; encoding an input image using the trained encoder to obtain initial features; inputting the initial features into the trained decoder to decode and generate an initial segmentation of the camouflage target; in the trained decoder, based on a mask separable attention mechanism, the initial segmentation is progressively optimized and segmented to obtain a final camouflage target segmentation result; wherein the mask separable attention mechanism uses different attention to focus on different parts of the image; the progressive optimization and segmentation of the initial segmentation is specifically: using the previously generated intermediate segmentation as a mask to limit foreground attention and background attention on the foreground and background, combining global information and larger resolution features to generate the next intermediate segmentation, and repeating the above process to finally generate the final camouflage target segmentation result.

2. The camouflage target segmentation method based on the mask separable attention mechanism according to claim 1, wherein, The mask separable attention mechanism uses a mask to separate the attention mechanism into foreground attention, background attention, and global attention, and extracts foreground region information, background region information, and global information, respectively.

3. The camouflage target segmentation method based on the mask separable attention mechanism according to claim 1, wherein, In the decoder, an element multiplication and sum connection feature fusion method is used.

4. The camouflage target segmentation method based on the mask separable attention mechanism according to claim 1, wherein, The training of the camouflage target segmentation model is supervised by the real annotation of the image to train the intermediate segmentation and the final segmentation until the training stopping condition is met.

5. A method of evaluating segmentation quality, characterized by, The evaluation method for the segmentation quality of the camouflage target segmentation method based on the mask separable attention mechanism according to any one of claims 1-4 comprises the following steps: extracting the edge of the target from the real annotation of the image to obtain an edge region; calculating the average absolute error and F-measure between the real annotation and the prediction in the edge region, and taking the obtained index value as the evaluation result of the segmentation quality.

6. The method of claim 5, wherein the quality of the segmentation is evaluated by, The edge region is obtained by using an edge search method to determine the edge of the target on the real annotation, and then using a dilation operation to expand the edge into an edge region.

7. A camouflage target segmentation system based on mask separable attention mechanism, characterized in that, The method comprises a model construction module, an image encoding module, an initial segmentation module, and an optimization segmentation module: The model construction module is configured to construct a camouflage target segmentation model composed of an encoder and a decoder, and train the camouflage target segmentation model; The image encoding module is configured to encode an input image using the trained encoder to obtain initial features; The initial segmentation module is configured to input the initial features into the trained decoder to decode and generate an initial segmentation of the camouflage target; The optimization segmentation module is configured to, in the trained decoder, based on a mask separable attention mechanism, progressively optimize and segment the initial segmentation to obtain a final camouflage target segmentation result; wherein the mask separable attention mechanism uses different attention to focus on different parts of the image; the progressive optimization and segmentation of the initial segmentation is specifically: The previous generated intermediate segmentation is taken as a mask to limit the foreground attention and the background attention on the foreground and the background, combined with global information and larger resolution features to generate the next intermediate segmentation, and the above process is repeated to finally generate the final camouflage target segmentation result.

8. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the camouflage target segmentation method based on the mask separable attention mechanism according to any one of claims 1-4.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized by The processor executes the program to implement the steps in the camouflage target segmentation method based on the mask separable attention mechanism according to any one of claims 1-4.

Images