Visual target segmentation method and device for noise image

Abstract

The invention relates to the technical field of image segmentation, and discloses a visual target segmentation method for a noise image, which comprises the following steps: acquiring a noise image, inputting the noise image into a backbone network, learning noise distribution and noise level information, outputting a de-noised image, and inputting the de-noised image into an improved MaskFormer model to obtain enhanced features; and inputting the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noise image. According to the de-noised image, acquiring continuous feature mapping by using a pixel-level decoder, and embedding image features and a query sequence pixel by pixel; after multiple continuous features are mapped, input into a GAMS, fused and spliced with features of different scales and output fused features, the fused features and a query sequence are input into an improved decoder, after downsampling is conducted on keys and values of the fused features, lightweight mask attention and standard cross attention are sequentially utilized, attention weights are calculated, and the attention weights are calculated; and carrying out weighted summation on the downsampled values according to the attention weight to obtain enhanced features.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and device for visual target segmentation of a noise image. Background Art

[0002] Noise-corrupted images refer to unclear images caused by the loss of pixel features or noise pollution during transmission and storage. In recent years, although some noise-corrupted image segmentation algorithms have been proposed, most of them rely on manual experience, and there are certain problems with accuracy and robustness. With the continuous improvement of the computing power of computer hardware equipment and the continuous advancement of artificial intelligence algorithms, deep learning models represented by Transformer have achieved great success in many fields of image processing, and also provided new ideas and directions for the research of noise-corrupted image segmentation algorithms.

[0003] Xu et al. proposed Attention U-Net, which added attention modules in the encoding and decoding paths to select areas of interest to reduce the influence of useless features and reduce the amount of network computation, thereby improving the segmentation effect. Wan et al. proposed a network XFi nger-Net for pixel-level segmentation, which replaced the cascade network with an attention gate with fewer parameters to suppress non-defiled parts. Xi e et al. combined a hierarchical Transformer-based encoder with a lightweight multi-layer perceptron decoder to aggregate feature maps from different layers. Liu et al. proposed the SwinTransformer in 2021 to construct hierarchical feature maps and generalized it as a general backbone network for downstream visual tasks. In the same year, Ali et al. proposed a method to calculate self-attention through feature channels. Wu et al. proposed an encoder-decoder based on the full transformer network (FTN) for image segmentation in 2021. This simple baseline achieved excellent results in multiple challenging semantic segmentation. Chen et al. proposed a new feature adaptive transformer network based on the classic encoder-decoder architecture and named it FAT-Net. The network integrates an additional transformer branch to effectively capture long-range dependencies and global context information. Jain et al. used the semantic attention mechanism to incorporate semantic information into the encoder to capture the semantic context in the hierarchical encoder network. Xia et al. proposed Uformer, a medical image segmentation network based on the self-attention mechanism. It adopts a structure similar to ViT, but unlike ViT, Uformer considers both local context information and global context information, so that the network can capture local features and pay attention to the global prospect. Shen et al. proposed a multi-branch network model of TransCUNet, which realizes the effective extraction of features at different levels of the image and shows high robustness on different data sets. Cheng et al. proposed a mask attention network in 2022. The basic idea of ​​this method is to extract local feature information by restricting cross attention within the predicted mask area. In addition, Dhamija et al. proposed an end-to-end deep learning framework called USegTransformer, which extracts image features of different scales by combining Transformer and convolution methods, and has achieved good results on different data sets.

[0004] In summary, although the above networks integrated with Transformer have achieved excellent performance in the field of image segmentation, in image segmentation, since objects may appear in different positions in the picture, the model needs to be able to recognize long-distance dependencies and global information. MaskFormer provides a new image segmentation method. MaskFormer integrates the advantages of the DETR model and the Transformer architecture. On the basis of DETR, it removes the prediction of the target box and the Transformer encoder, and instead adds a pixel decoder and directly outputs the mask classification, saving a certain amount of time and space overhead.

[0005] However, although the existing MaskFormer can effectively extract image features, small targets occupy fewer pixels in the image. At the same time, due to the limitation of small target size, the details of the object are not obvious enough, which increases the difficulty of retrieval and segmentation. The model structure design of MaskFormer is not specially optimized for small targets, so it is unable to capture the subtle differences and details of some images, resulting in the MaskFormer-based image segmentation method not being fine enough for the segmentation of small targets. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems of low training efficiency and insufficient fineness of small target segmentation in the prior art image segmentation method based on MaskFormer.

[0007] In order to solve the above technical problems, the present invention provides a visual target segmentation method for a noisy image, comprising:

[0008] Obtain a noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output a denoised image;

[0009] The denoised image is input into the improved MaskFormer model to obtain enhanced features, which include:

[0010] After the pixel-level decoder, multiple continuous feature maps, a pixel-by-pixel embedded image feature and query sequence are obtained;

[0011] Input multiple continuous feature maps into the multi-scale feature fusion module GAMS, and obtain a weight map after channel downsampling and channel splicing; divide the weight map into multiple sub-weight maps, weight the multiple continuous feature maps respectively, obtain multiple weighted features, perform convolution aggregation, and output fusion features;

[0012] The fused features and the query sequence are input into the improved TransFormer decoder, the query sequence is masked, and a binary mask prediction value is obtained; the keys and values ​​of the fused features are downsampled using a depthwise separable convolution operation; the attention weights between the query and the downsampled keys are calculated using lightweight mask attention and standard cross attention respectively, and the downsampled values ​​are weighted summed according to the attention weights to obtain enhanced features;

[0013] The enhanced features are input into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

[0014] In one embodiment of the present invention, the key and value of the fused feature are downsampled by using a depthwise separable convolution operation; the attention weights between the query and the downsampled key are calculated by using a lightweight mask attention and a standard cross attention respectively, and the downsampled value is weighted summed according to the attention weight to obtain an enhanced feature, including:

[0015] Key K for fusion features l Downsampling is performed, expressed as:

[0016] The value V of the fusion feature l Downsampling is performed, expressed as:

[0017] Compute query Q and downsampled key K using lightweight masked attention l The attention weight between them is used to calculate the downsampled value V according to the attention weight. l Perform weighted summation, expressed as:

[0018]

[0019] Calculate the attention weights between the query and the downsampled key according to the standard cross attention, and perform weighted summation of the downsampled values ​​according to the attention weights to obtain the enhanced feature X l ", expressed as:

[0020]

[0021] Among them, DWConv() represents a depth-separable convolution operation with a convolution kernel size of k*k and a step size of k; represents the full set of fusion features, C represents the number of channels, and H l and W l is the spatial resolution of the fused features; l is the layer index of the improved TransFormer decoder, represents the N C-dimensional query features of the lth layer, and its initialization X0 represents the fusion features of the input improved Transformer decoder; the query sequence f Q 、f K and f V is the linear transformation function; M l-1 (x,y) represents M l-1 The binary mask prediction value at the (x,y) position is expressed as

[0022] In one embodiment of the present invention, the improved Transformer decoder includes a lightweight mask attention module, a self-attention module and a multi-layer perceptron module which are connected in series in a forward propagation direction; a residual connection and normalization module is arranged after the lightweight mask attention module, the self-attention module and the multi-layer perceptron module.

[0023] In one embodiment of the present invention, the multi-scale feature fusion module GAMS includes:

[0024] Scaling unit, which scales multiple continuous feature maps of the input to the same size;

[0025] Input convolution unit, use 1×1 convolution to reduce the channel depth of the scaled continuous feature map to 1;

[0026] A channel splicing unit performs channel splicing on multiple continuous feature maps output by the input convolution unit;

[0027] Softmax unit, uses the Softmax function to activate the continuous feature map after channel splicing to obtain the weight map;

[0028] A weight graph division unit, which divides the weight graph into a plurality of sub-weight graphs;

[0029] The weighted aggregation unit uses the weighted map and the corresponding continuous feature map to perform feature splicing;

[0030] The output convolution unit aggregates the output of the weighted aggregation unit through a 1×1 convolution and outputs the fused features.

[0031] In one embodiment of the present invention, the fusion feature is expressed as:

[0032]

[0033] Among them, O represents the fusion feature, W i represents the weighted graph, S i represents a continuous feature map and ⊙ represents a dot product operation.

[0034] In one embodiment of the present invention, the step of acquiring a noisy image, inputting the image into a backbone network, learning noise distribution and noise level information, and outputting a denoised image includes:

[0035] The noise observation value of the noisy image is input into the noise estimation subnetwork of the convolutional denoising blind network, the noise distribution of the noise observation value is learned, and the estimated noise level map is output;

[0036] The noise observation value and the estimated noise level map of the noisy image are input into the non-blind denoising sub-network of the convolutional denoising blind network to learn the noise distribution and noise level information and output the denoised image.

[0037] In one embodiment of the present invention, the noise estimation sub-network is a five-layer fully convolutional network; and the non-blind denoising sub-network is a convolutional network based on a codec architecture.

[0038] In one embodiment of the present invention, the acquisition of pixel-by-pixel embedded image features includes:

[0039] Using a deep learning model to extract features from the denoised image to obtain a feature map, and using a deconvolution network to obtain an amplified feature map;

[0040] Classify each pixel in the enlarged feature map to determine the category to which the pixel belongs;

[0041] According to the category to which the pixel belongs, each pixel is predicted, the position and shape information of each pixel is obtained, and the pixel-by-pixel embedded image features are generated based on the input denoised image.

[0042] In one embodiment of the present invention, the step of inputting the enhanced features into a multilayer perceptron to obtain a visual target segmentation result of the noisy image includes:

[0043] The enhanced features are input into a multilayer perceptron to generate a class prediction value sequence and a mask prediction value sequence; a dot product operation is performed on the pixel-by-pixel embedded image features and the mask prediction value sequence to obtain a binary mask value; the class prediction value sequence and the binary mask value are bit-wise multiplied and then binarized to obtain a visual target segmentation result of the noisy image.

[0044] The embodiment of the present invention further provides a visual target segmentation device for a noisy image, comprising:

[0045] The preprocessing module is used to obtain the noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output the denoised image;

[0046] The mask prediction module is used to input the denoised image into the improved MaskFormer model: after the pixel-level decoder, multiple continuous feature maps, a pixel-by-pixel embedded image feature and a query sequence are obtained; the multiple continuous feature maps are input into the multi-scale feature fusion module GAMS, and a weight map is obtained after channel downsampling and channel splicing; the weight map is divided into multiple sub-weight maps, and multiple continuous feature maps are weighted respectively to obtain multiple weighted features, and convolution aggregation is performed to output fused features; the fused features and the query sequence are input into the improved TransFormer decoder, the query sequence is masked, and a binary mask prediction value is obtained; the keys and values ​​of the fused features are downsampled using a depth-separable convolution operation; the attention weights between the query and the downsampled keys are calculated using lightweight mask attention and standard cross attention respectively, and the downsampled values ​​are weighted summed according to the attention weights to obtain enhanced features;

[0047] The result acquisition module is used to input the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

[0048] The above technical solution of the present invention has the following advantages compared with the prior art:

[0049] The visual target segmentation method for noisy images described in the present invention constructs a multi-scale feature fusion module GAMS, and obtains a weight map by channel downsampling and channel splicing of multiple continuous feature maps; the weight map is divided into multiple sub-weight maps, and multiple continuous feature maps are weighted and convolutionally aggregated respectively to output fusion features; GAMS characterizes the semantic information of feature images at different scales, fuses and splices features of different scales, and applies a multi-scale strategy to strengthen feature dependence, thereby improving the feature discrimination ability of the model and improving the segmentation accuracy; the present invention adds a lightweight mask attention module on the basis of the original Transformer decoder, and calculates the attention weight between the query and the downsampled key by constraining the cross attention to the foreground area of ​​the predicted mask of each query, and performs weighted summation on the downsampled value according to the attention weight to extract local features, thereby improving the segmentation accuracy of small targets; and the lightweight mask attention is set before the self-attention mechanism in the Transformer decoder, thereby reducing the amount of network parameters, improving the calculation efficiency, further improving the feature extraction ability and generalization of the network, and achieving better visual target segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0051] Figure 1 It is a flowchart of the steps of the visual target segmentation method for noisy images provided by the present invention;

[0052] Figure 2 It is a structural diagram of the existing MaskFormer model;

[0053] Figure 3 is a model structure diagram of the improved MaskFormer model provided by the present invention;

[0054] Figure 4 It is a structural diagram of the improved Transformer decoder provided by the present invention;

[0055] Figure 5 It is a structural schematic diagram of the multi-scale feature fusion module GAMS provided by the present invention;

[0056] Figure 6 It is the CBDNet network structure diagram provided by the present invention;

[0057] Figure 7 It is a comparison chart of the segmentation effects of multiple models provided by the present invention on the ADE20K dataset;

[0058] Figure 8 This is a comparison chart of the segmentation effects of the models provided by the present invention on the COCO-Stuff-10k dataset. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0060] Reference Figure 1 As shown, the flowchart of the method for visual target segmentation of a noisy image provided by the present invention includes the following specific steps:

[0061] S1: Obtain a noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output a denoised image;

[0062] S2: Input the denoised image into the improved MaskFormer model to obtain enhanced features, which includes:

[0063] After the pixel-level decoder, multiple continuous feature maps, a pixel-by-pixel embedded image feature and query sequence are obtained;

[0064] Input multiple continuous feature maps into the multi-scale feature fusion module GAMS, and obtain a weight map after channel downsampling and channel splicing; divide the weight map into multiple sub-weight maps, weight the multiple continuous feature maps respectively, obtain multiple weighted features, perform convolution aggregation, and output fusion features;

[0065] The fused features and the query sequence are input into the improved TransFormer decoder, the query sequence is masked, and a binary mask prediction value is obtained; the keys and values ​​of the fused features are downsampled using a depthwise separable convolution operation; the attention weights between the query and the downsampled keys are calculated using a lightweight mask attention and a standard cross attention respectively, and the weights are calculated based on the weights.

[0066] Perform weighted summation on the downsampled values ​​according to the attention weights to obtain enhanced features;

[0067] S3: Input the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

[0068] The embodiment of the present invention proposes a segmentation network based on an improved MaskFormer, replaces the Transformer decoding attention mechanism with a lightweight mask attention mechanism, constructs a Transformer decoder, constrains the foreground area of ​​each predicted mask by cross-attention, reduces the interference of the contaminated area, enhances the extraction of local features, and thus accurately divides the foreground area and the background area; adopts a multi-scale feature fusion module GAMS based on a gating mechanism to characterize the semantic information of different scales of the image, and improves the feature discrimination ability of the model; exchanges the order of self-attention and lightweight mask attention, reduces the amount of network calculation, and improves the model training efficiency.

[0069] Reference Figure 2 The figure shows the model structure diagram of the existing MaskFormer model; the existing MaskFormer model is an end-to-end image segmentation model based on Transformer, which can seamlessly convert the classification mode from pixel classification to mask classification. The model permutes the input image sequence into a set sequence, and continues the set prediction mechanism in DETR. Each set pair consists of a category prediction and a mask embedding vector. The role of the mask embedding vector is to perform a dot multiplication with each pixel to obtain a binary prediction mask. Finally, the model is supervised and trained through the same pixel binary mask loss function and a single classification loss function, achieving the purpose of simplifying semantic segmentation and panoramic segmentation, showing good segmentation effects.

[0070] MaskFormer is mainly composed of three parts: Transformer module, pixel level module and segmentation module. From the feature processing flow, MaskFormer is similar to DETR. First, a backbone module is used to extract image features. Then use the pixel decoder to extract each pixel embedding ε pixel , and upsample it. At the same time, the features are processed by the Transformer decoder to produce N sequences Q, which are then fed into a multi-layer perceptron (MLP) to independently produce N class predictions and N corresponding mask embeddings ε mask . Then the pixels are embedded into ε pixel With mask embedding ε mask Dot product operation is performed and N binary masks are predicted through sigmoid activation function. Finally, matrix multiplication operation is performed on binary mask and class prediction to get the final result. The following is a detailed introduction to MaskFormer module:

[0071] Backbone module, since MaskFormer inherits the backbone network of DETR and is compatible with various CNN models, the standard ResNet and SwinTransformer backbone networks are used for segmentation, respectively, and both achieve good segmentation results. Pixel-level decoder, mainly used to extract image features and generate pixel-by-pixel embeddings of binary mask predictions, providing a basis for subsequent segmentation. In CNN, most pixel decoders are added with feature pyramid ASPP modules and PSP modules in order to better aggregate the context information of the image. Transformer generates category predictions by focusing on the global features of the image, reducing the requirements for context aggregation. MaskFormer's pixel-level decoder is a lightweight model based on FPN. In the decoder stage, low-level image features are upsampled by 2 times through FPN and fused with deep features. The specific process is to first match the channel size of the feature image through 1×1 convolution and group normalization; secondly, use 3×3 convolution to fuse the spliced ​​features through group normalization and ReLU activation function; finally, use 1×1 convolution to obtain the pixel embedding ε pixel , and change the channel size of all feature maps to 256. Transformer module, MaskFormer adopts a Transformer decoder design similar to DETR, from image features and N learnable position embeddings to calculate its output and encode the information of the MaskFormer prediction part. In addition, this module uses 6 Transformer decoder layers, sets 100 position embeddings by default, and applies the same loss function after each decoder. Experimental results show that this decoder has better classification effect than ordinary CNN or Transformer. The segmentation module uses a multi-layer perceptron with 2 hidden layers and a total of 256 channels to process the feature sequence Q output by the Transformer decoder. The category prediction branch then applies a linear classifier to each segment of embedding Q and performs softmax activation to finally generate a category probability prediction. In addition, a no-category prediction is set to indicate that this segment of embedding does not belong to any region. On the mask prediction branch, the multi-layer perceptron will convert the embedding Q into N mask embeddings ε mask Finally, the i-th mask embedding ε is calculated by the pixel-level module mask and pixel-wise embedding ε pixel The dot product of , yields the corresponding binary mask prediction.

[0072] In summary, MaskFormer removes the prediction of the target box and the Transformer encoder based on DETR, and instead adds a pixel decoder and directly outputs the mask classification, saving a certain amount of time and space overhead. However, this method still has problems such as low training efficiency, excessive memory usage, and insufficient segmentation of small objects.

[0073] Therefore, in view of the shortcomings of the existing MaskFormer model, the present invention proposes a noise-corrupted image segmentation method based on an improved MaskFormer, and adopts the same meta-architecture as the existing MaskFormer model, namely the backbone, pixel decoder and Transformer decoder. The idea of ​​the model improvement is: first, the attention mechanism in the Transformer decoder is replaced with a lightweight mask attention to divide the foreground area and the background area. Secondly, in order to improve the feature discrimination ability of the model, a gated multi-scale feature fusion module GAMS is adopted to automatically utilize semantic information of different scales. Finally, the position order of self-attention and lightweight mask attention in the Transformer decoder is switched to reduce the amount of network parameters to improve computational efficiency. Reference Figure 3 As shown, it is a model structure diagram of the improved MaskFormer model provided by the present invention, which specifically includes a pixel decoder, a multi-scale feature fusion module GAMS, an improved Transformer decoder and a classification module.

[0074] In the embodiment of the present invention, the pixel decoder feeds the continuous feature maps of the first three layers to the GAMS module, guides the Transformer decoder by outputting features and query sequences, generates prediction values ​​of categories and masks in parallel, and applies a multi-scale strategy to strengthen global context dependency, thereby improving model accuracy.

[0075] Reference Figure 4 As shown, it is a structural diagram of an improved Transformer decoder; the improved Transformer decoder includes a lightweight mask attention module, a self-attention module and a multi-layer perceptron module which are sequentially connected in series along the forward propagation direction; a residual connection and normalization module are arranged after the lightweight mask attention module, the self-attention module and the multi-layer perceptron module.

[0076] In this embodiment, the key component of improving the Transformer decoder includes a lightweight mask attention operator, which extracts local features by constraining cross attention to the foreground area of ​​the predicted mask of each query, rather than focusing on the entire feature map, thereby ignoring the interference of background pixels. And using lightweight mask attention, the problem of slow convergence of the Transformer model is solved, while further reducing computational overhead.

[0077] The existing Transformer-based models converge slowly because the criss-cross attention layer takes a long time to learn the regional information of local objects. The expression of standard criss-cross attention is as follows:

[0078]

[0079] Where l is the layer index, represents the N C-dimensional query features of the lth layer, and so on, X0 represents the input query features of the Transformer decoder, are all transformed image features, where H l and W l is the spatial resolution of the image features, f Q 、f K and f V is a linear transformation function.

[0080] In order to further reduce the computational overhead, the present invention uses a depth-separable convolution operation to perform K before the original standard cross attention. l 、V l Downsampling is performed to obtain enhanced features. The specific steps include:

[0081] Key K for fusion features l Downsampling is performed, expressed as:

[0082] The value V of the fusion feature l Downsampling is performed, expressed as:

[0083] Compute query Q and downsampled key K using lightweight masked attention l The attention weight between them is used to calculate the downsampled value V according to the attention weight. l Perform weighted summation, expressed as:

[0084]

[0085] Calculate the attention weights between the query and the downsampled key according to the standard cross attention, and perform weighted summation of the downsampled values ​​according to the attention weights to obtain the enhanced feature X" l , expressed as:

[0086]

[0087] Among them, DWConv() represents a depth-separable convolution operation with a convolution kernel size of k*k and a step size of k; represents the full set of fusion features, C represents the number of channels, and H l and W l is the spatial resolution of the fused features; l is the layer index of the improved TransFormer decoder, represents the N C-dimensional query features of the lth layer, and its initialization X0 represents the fusion features of the input improved Transformer decoder; the query sequence f Q 、f K and f V is the linear transformation function; M l-1 (x,y) represents M l-1 The binary mask prediction value at the (x,y) position is expressed as M l-1 (x,y) is also adjusted to be consistent with K through depthwise separable convolution. l Same resolution. M0 is the binary mask prediction obtained from X0, i.e., the prediction value before the query sequence is input to the Transformer decoder.

[0088] Specifically, in order to optimize the Transformer decoder design, the improved Transformer decoder in this embodiment swaps the order of self-attention and cross-attention. This can optimize computational efficiency, because the query features of the first self-attention layer do not yet depend on the image features, so applying self-attention will not produce features containing too much spatial information, while cross-attention can make full use of the original image features, that is, lightweight mask attention is used as the first self-attention layer to focus on the original image features.

[0089] Feature fusion is crucial for semantic segmentation tasks. When the model fuses shallow and deep high- and low-resolution features, it can combine location information with semantic information. However, there are certain differences between semantic information of different scales. If these multi-scale feature information are processed only by simple splicing or connection operations, the segmentation performance of the model will be reduced. Therefore, an embodiment of the present invention provides a multi-scale feature fusion module (GAMS) to automatically utilize these differences to improve the feature discrimination ability of the model. Figure 5 As shown, it is a schematic diagram of the structure of GAMS; the multi-scale feature fusion module GAMS includes:

[0090] Scaling unit, which scales multiple continuous feature maps of the input to the same size;

[0091] Input convolution unit, use 1×1 convolution to reduce the channel depth of the scaled continuous feature map to 1;

[0092] A channel splicing unit performs channel splicing on multiple continuous feature maps output by the input convolution unit;

[0093] Softmax unit, uses the Softmax function to activate the continuous feature map after channel splicing to obtain the weight map;

[0094] A weight graph division unit, which divides the weight graph into a plurality of sub-weight graphs;

[0095] The weighted aggregation unit uses the weighted map and the corresponding continuous feature map to perform feature splicing;

[0096] The output convolution unit aggregates the output of the weighted aggregation unit through a 1×1 convolution and outputs the fused features.

[0097] In GAMS, we first transform the feature maps S of the first three layers of the pixel decoder into i (i=1,2,3) are rescaled to the same size, and then a 1×1 convolution is applied to convert S i The channel depth is reduced to 1, and then the mapped feature S iChannel concatenation is performed, and the weight map W is obtained through the softmax activation function. Among them, W is further divided into three weight maps W1, W2, and W3, which are used to adjust the input feature S i After weighted operation, these features are aggregated after 1×1 convolution to obtain the output fusion feature O, which is expressed as:

[0098] Among them, O represents the fusion feature, W i represents the weighted graph, S i represents a continuous feature map and ⊙ represents a dot product operation.

[0099] In this embodiment, the gated multi-scale feature fusion module (GAMS) uses the difference between high and low resolutions to improve the feature discrimination ability of the model, better identify targets with a small pixel footprint, and improve the segmentation accuracy of small targets.

[0100] In order to solve the problem of preprocessing noise-corrupted images, this embodiment selects a convolutional blind denoising network (CBDNet), which is characterized by being able to better handle situations containing real noise-clean image pairs. This model not only uses signal-correlated noise and the in-camera signal processing pipeline to synthesize real noisy images, but also adds noisy and noise-free clean images in the real world to train the denoising model CBDNet. In addition, in order to further provide an interactive strategy to conveniently correct the denoising results, a noise estimation subnetwork with asymmetric learning is embedded in CBDNet to suppress underestimation of the noise level. The network structure of CBDNet is as follows: Figure 6 As shown in the figure, it mainly includes two parts: one is the convolutional neural network CNNE (Convolutional Neural Network Estimation) which can estimate the noise of the image, and the other is a commonly used non-blind denoising neural network CNND (Convolutional Neural Network Denoising), which can perform denoising operations on noisy images.

[0101] Reference Figure 6 As shown, it is a schematic diagram of the network structure of the backbone network used in the present invention; in this embodiment, the backbone network uses a convolutional denoising blind network; the noise observation value of the noise image is input into the noise estimation subnetwork of the convolutional denoising blind network, the noise distribution of the noise observation value is learned, and the estimated noise level map is output; the noise observation value of the noise image and the estimated noise level map are input into the non-blind denoising subnetwork of the convolutional denoising blind network, the noise distribution and noise level information are learned, and the denoised image is output.

[0102] From the perspective of the model execution process, the CNNE subnet first takes the noise observation value y as input, and then learns the noise distribution in the input image through the fully convolutional network to generate an estimated noise level map σ^(y)=FE(y;WE), where WE represents the network parameters. For the feature dimension, the output noise level map of the subnet has the same size as the input y. In the next stage, the non-blind denoising subnet CNND takes both y and σ^(y) as input, and obtains the final denoising result x^=FD(y,σ^(y);WD) by learning the noise distribution and noise level information in the input image.

[0103] CNNE uses a common five-layer fully convolutional network without pooling and batch normalization operations. In each convolution (Conv) layer, the number of feature channels is set to 32 and the filter size is 3 × 3. ReLU nonlinearity is deployed after each Conv layer.

[0104] CNND is based on an encoder-decoder architecture that captures feature map information through operations such as symmetric skip connections, strided convolutions, and transposed convolutions. All convolution kernels in the network are 3×3 in size, and ReLU nonlinearity is applied after each Conv layer except the last layer. In addition, it has been empirically found that batch normalization has little help in denoising real photos, partly because the real noise distribution is fundamentally different from the Gaussian noise distribution.

[0105] In order to exploit the asymmetric sensitivity in blind denoising, CBDNet proposes an asymmetric noise loss estimation to avoid underestimation errors on the noise level map. Considering the estimated noise level σ^(y i ) and the ground truth σ(y i ), when σ^(y i )<σ(y i ), more penalty should be imposed on its MSE. Therefore, the asymmetric loss L on the noise estimation subnetwork asymm , is defined as the expression:

[0106]

[0107] Use the total variation regularizer to guide σ^(y i )'s smoothness L TV , expressed as:

[0108]

[0109] in and Represents the gradient operator along the horizontal and vertical directions. For the input image x and the output x^ of non-blind denoising, the reconstruction loss expression L is defined rec :

[0110]

[0111] Combining the above three equations and their respective weighted coefficients, the total loss expression L can be obtained:

[0112] L=L rec +λ asymm L asymm +λ TV L TV ;

[0113] λ asymm and λ TV They represent the trade-off parameters of asymmetric loss and TV regularizer respectively. CBDNet considers both heteroscedastic Gaussian noise and the real noise model of the camera internal processing process, which can improve the generalization ability of the model to real noise.

[0114] In this embodiment, the acquisition of the pixel-by-pixel embedded image features includes: using a deep learning model to extract features from the denoised image to obtain a feature map, and using a deconvolution network to obtain an enlarged feature map; classifying each pixel in the enlarged feature map to determine the category to which the pixel belongs; predicting each pixel according to the category to which the pixel belongs, obtaining the position and shape information of each pixel, and generating a pixel-by-pixel embedded image feature based on the input denoised image.

[0115] Specifically, after obtaining the enhanced features, the visual target segmentation result of the noisy image is obtained, which specifically includes: inputting the enhanced features into a multilayer perceptron to generate a category prediction value sequence and a mask prediction value sequence; performing a dot product operation on the pixel-by-pixel embedded image features and the mask prediction value sequence to obtain a binary mask value; multiplying the category prediction value sequence and the binary mask value bit by bit and then performing a binarization process to obtain the visual target segmentation result of the noisy image.

[0116] The visual target segmentation method for noisy images described in the present invention constructs a multi-scale feature fusion module GAMS, and obtains a weight map by channel downsampling and channel splicing of multiple continuous feature maps; the weight map is divided into multiple sub-weight maps, and multiple continuous feature maps are weighted and convolutionally aggregated respectively to output fusion features; GAMS characterizes the semantic information of feature images at different scales, fuses and splices features of different scales, and applies a multi-scale strategy to strengthen feature dependence, thereby improving the feature discrimination ability of the model and improving the segmentation accuracy; the present invention adds a lightweight mask attention module on the basis of the original Transformer decoder, and calculates the attention weight between the query and the downsampled key by constraining the cross attention to the foreground area of ​​the predicted mask of each query, and performs weighted summation on the downsampled value according to the attention weight to extract local features, thereby improving the segmentation accuracy of small targets; and the lightweight mask attention is set before the self-attention mechanism in the Transformer decoder, thereby reducing the amount of network parameters, improving the calculation efficiency, further improving the feature extraction ability and generalization of the network, and achieving better visual target segmentation effect.

[0117] Based on the above embodiment, in this embodiment, the server operating system and version number are 64-bit Ubuntu18.04.5, model training is performed based on the deep learning framework PyTorch, and the program is run on a server with 3 NVIDIATitanV Volta GPUs with a video memory size of 12GB. The software environment is Python3.7+CUDA10.1+CUDNN7.6.0+Paddleseg2.4.0. This embodiment selects two types of data sets commonly used in the field of image segmentation, one is the ADE20K data set, and the other is the COCO-Stuff-10k data set. The ADE20K data set can be applied to a variety of segmentation tasks, where the training set contains 20210 images, the validation set and the test set are 2000 images and 3352 images respectively, with a total of 150 categories; due to the problem of category imbalance, the ADE20K image data has always been a difficult point in the segmentation task. The COCO-Stuff-10k dataset is taken from the COCO dataset and is a challenging scene parsing dataset; it contains about 10,000 images with dense pixel labels, of which the training set and test set contain 9,000 images and 1,000 images respectively, with a total of 171 categories.

[0118] In order to verify the segmentation performance of the visual target segmentation method for noisy images provided by the present invention, in this embodiment, Pixel Acc and MIoU are used as evaluation indicators. This embodiment uses the symbols used by Long et al. to summarize the main indicators of image segmentation. Let n ijis the number of pixels of category i predicted as category j, where i,j∈(1,...,C), and C is the number of different semantic classes. Let t i =∑ j n ij is the total number of pixels in the i-th category.

[0119] Pixel Accuracy is the most commonly used indicator for image segmentation, which indicates the percentage of correctly classified pixels in an image. Its formula is:

[0120] Mean Intersection over Union (MIoU) means that the model calculates the ratio of the predicted result of each category to the true value, and then sums and calculates the average value. It is the most important evaluation indicator for image segmentation. Its formula is:

[0121]

[0122] The evaluation indicators of this embodiment also use single-scale MIoU (ssMIoU) and multi-scale MIoU (msMIoU); ssMIoU is called single-scale MIoU, which refers to the average intersection-over-union ratio at a single scale, and msMIoU is called multi-scale MIoU, which refers to the average intersection-over-union ratio at multiple scales. This embodiment verifies the effectiveness of the segmentation model provided by the present invention from single scale to multi-scale.

[0123] Specifically, the two types of data sets are firstly processed with random noise, and then the noise is modeled using CBDNet to achieve the denoising effect, and finally the clear image is fed as input into the model provided by the invention.

[0124] For the ADE20K dataset, the initial learning rate is set to 0.005, the image is cropped to 608×608, the batch size is 16, and the training is performed for 240 epochs. The ADE20K dataset is used for verification, and multiple segmentation models are compared from four different angles, including PSPNet and MaskFormer, which are common models in the field of noise-damaged image segmentation. The comparison results are shown in Table 1:

[0125] Table 1 ADE20K comparison experiment

[0126]

[0127]

[0128] As can be seen from Table 1, the Pixel Accuracy and MIoU index values ​​of the PSPNet model on the ADE20K dataset are 78.69% and 41.76% respectively. Compared with PSPNet, the improved MaskFormer model proposed in the present invention has increased these indicators by 3.06% and 8.09% respectively, and has also improved compared with MaskFormer, both of which are the highest indicators. In addition, the present invention also conducted an ablation experiment on the backbone network of MaskFormer, using ResNet50 and ResNet101 respectively. According to the results, it can be seen that the use of ResNet101 has a significant improvement in the MIoU index. In summary, compared with other network models, the improved MaskFormer model provided by the present invention still has a certain improvement in segmentation performance while reducing the computational cost and model prediction time.

[0129] Reference Figure 7 As shown, it is a comparison chart of the segmentation effects of multiple models on the ADE20K dataset. The first column is the ADE20K original image, the second column is the label truth map of the image, and the third to fifth columns are the segmentation result maps of each method. By comparison, it can be seen that the PSPNet model does not handle the detailed information of the image well, and the segmentation effect for the image boundary needs to be improved; MaskFormer has been greatly improved compared to PSPNet, but there are still some small edge errors, such as the unclear segmentation of the red road in the last picture; the fifth column is the segmentation result map of the improved MaskFormer model provided by the present invention. It can be seen that the segmentation effect on the ADE20K dataset is better, and the details and edge parts of the image are properly handled, reflecting the superiority of the model of the present invention.

[0130] In addition to the experiments on the ADE20K dataset, this paper also uses the COCO-Stuff-10k scene segmentation dataset to verify the proposed method. For COCO-Stuff-10k, the initial learning rate is set to 0.0001, the weight decay is 0.0001, the image is cropped to 512×512, the batch size is 16, and 500 epochs are trained. The comparison of segmentation results under the COCO-Stuff-10k dataset is shown in Table 2:

[0131] Table 2 COCO-Stuff-10k comparative experiment

[0132]

[0133]

[0134] As shown in Table 2, this embodiment uses PSPNet, MaskFormer and the improved MaskFormer model provided by the present invention for comparative experiments, and uses ssMIoU and msMIoU as segmentation evaluation indicators of the data set. The ssMIoU and msMIoU values ​​of the PSPNet model on the COCO-Stuff-10k data set are 37.14% and 38.37%, respectively. The MaskFormer model has improved indicators compared to the PSPNet. Similar to the ADE20K data set, the backbone networks of MaskFormer in this embodiment use ResNet101 and SwinL, respectively, among which SwinL shows better segmentation effects. Based on this, the model of this embodiment also uses SwinL as the backbone extraction network to obtain the best segmentation performance.

[0135] Reference Figure 8 As shown, it is a comparison chart of the segmentation effects of each model on the COCO-Stuff-10k dataset. The first column is the original image of COCO-Stuff-10k, the second column is the true value map of the image label, and the third to fifth columns are the segmentation result maps of PSPNet, MaskFormer and the method provided by the present invention. By comparison, it can be seen that the PSPNet model has a poor effect on the segmentation of the image and cannot accurately capture the edge and contour information of the image; the MaskFormer model can pay attention to the edge information of the image, thereby greatly improving the segmentation effect based on the PSPNet, but there is still a problem of poor processing of some details of the image. For example, in the fourth image, the segmentation effect of MaskFormer for the background fence is not accurate enough; compared with the other two models, the method provided by the present invention performs best in both evaluation indicators and segmentation effects, reflecting the effectiveness of the model, and the improved MaskFormer model has better segmentation performance.

[0136] Based on the above embodiment, in this embodiment, a visual target segmentation device for a noise image is also provided, which specifically includes:

[0137] The preprocessing module 100 is used to obtain a noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output a denoised image;

[0138] The mask prediction module 200 is used to input the denoised image into the improved MaskFormer model: after the pixel-level decoder, multiple continuous feature maps, a pixel-by-pixel embedded image feature and a query sequence are obtained; the multiple continuous feature maps are input into the multi-scale feature fusion module GAMS, and a weight map is obtained after channel downsampling and channel splicing; the weight map is divided into multiple sub-weight maps, and multiple continuous feature maps are weighted respectively to obtain multiple weighted features, and convolution aggregation is performed to output fused features; the fused features and the query sequence are input into the improved TransFormer decoder, the query sequence is masked, and a binary mask prediction value is obtained; the keys and values ​​of the fused features are downsampled using a depth-separable convolution operation; the attention weights between the query and the downsampled keys are calculated using lightweight mask attention and standard cross attention respectively, and the downsampled values ​​are weighted summed according to the attention weights to obtain enhanced features;

[0139] The result acquisition module 300 is used to input the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

[0140] The visual target segmentation device for noisy images of this embodiment is used to implement the aforementioned visual target segmentation method for noisy images. Therefore, the specific implementation method of the visual target segmentation device for noisy images can be seen in the embodiment part of the visual target segmentation method for noisy images in the previous text. For example, the preprocessing module 100, the mask prediction module 200, and the result acquisition module 300 are respectively used to implement steps S1, S2 and S3 in the aforementioned visual target segmentation method for noisy images. Therefore, its specific implementation method can refer to the description of the corresponding various parts of the embodiment, which will not be repeated here.

[0141] The visual target segmentation method for noisy images described in the present invention constructs a multi-scale feature fusion module GAMS, and obtains a weight map by channel downsampling and channel splicing of multiple continuous feature maps; the weight map is divided into multiple sub-weight maps, and multiple continuous feature maps are weighted and convolutionally aggregated respectively to output fusion features; GAMS characterizes the semantic information of feature images at different scales, fuses and splices features of different scales, and applies a multi-scale strategy to strengthen feature dependence, thereby improving the feature discrimination ability of the model and improving the segmentation accuracy; the present invention adds a lightweight mask attention module on the basis of the original Transformer decoder, and calculates the attention weight between the query and the downsampled key by constraining the cross attention to the foreground area of ​​the predicted mask of each query, and performs weighted summation on the downsampled value according to the attention weight to extract local features, thereby improving the segmentation accuracy of small targets; and the lightweight mask attention is set before the self-attention mechanism in the Transformer decoder, thereby reducing the amount of network parameters, improving the calculation efficiency, further improving the feature extraction ability and generalization of the network, and achieving better visual target segmentation effect.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0146] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A visual target segmentation method for noisy images, characterized in that: include: Obtain a noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output a denoised image; The denoised image is input into the improved MaskFormer model to obtain enhanced features, which include: After the pixel-level decoder, multiple continuous feature maps, a pixel-by-pixel embedded image feature and query sequence are obtained; Input multiple continuous feature maps into the multi-scale feature fusion module GAMS, and obtain a weight map after channel downsampling and channel splicing; divide the weight map into multiple sub-weight maps, weight the multiple continuous feature maps respectively, obtain multiple weighted features, perform convolution aggregation, and output fusion features; The fused features and the query sequence are input into the improved TransFormer decoder, the query sequence is masked, and a binary mask prediction value is obtained; the keys and values ​​of the fused features are downsampled using a depthwise separable convolution operation; the attention weights between the query and the downsampled keys are calculated using lightweight mask attention and standard cross attention respectively, and the downsampled values ​​are weighted summed according to the attention weights to obtain enhanced features; The enhanced features are input into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

2. The visual target segmentation method for noisy images according to claim 1, characterized in that: The method uses a depth-separable convolution operation to downsample the keys and values ​​of the fused features; uses lightweight mask attention and standard cross attention in turn to calculate the attention weights between the query and the downsampled keys, and performs weighted summation on the downsampled values ​​according to the attention weights to obtain enhanced features, including: Key K for fusion features l Downsampling is performed, expressed as: The value V of the fusion feature l Downsampling is performed, expressed as: Compute query Q and downsampled key K using lightweight masked attention l The attention weight between them is used to calculate the downsampled value V according to the attention weight. l Perform weighted summation, expressed as: Calculate the attention weights between the query and the downsampled key according to the standard cross attention, and perform weighted summation of the downsampled values ​​according to the attention weights to obtain the enhanced feature X" l , expressed as: Among them, DWConv() represents a depth-separable convolution operation with a convolution kernel size of k*k and a step size of k; represents the full set of fusion features, C represents the number of channels, and H l and W l is the spatial resolution of the fused features; l is the layer index of the improved TransFormer decoder, represents the N C-dimensional query features of the lth layer, and its initialization X0 represents the fusion features of the input improved Transformer decoder; the query sequence f Q 、f K and f V is the linear transformation function; M l-1 (x,y) represents M l-1 The binary mask prediction value at the (x,y) position is expressed as 3. The visual target segmentation method for noisy images according to claim 1, characterized in that: The improved Transformer decoder includes a lightweight mask attention module, a self-attention module and a multi-layer perceptron module which are connected in series in a forward propagation direction; a residual connection and normalization module is arranged after the lightweight mask attention module, the self-attention module and the multi-layer perceptron module.

4. The visual target segmentation method for noisy images according to claim 1, characterized in that: The multi-scale feature fusion module GAMS includes: Scaling unit, which scales multiple continuous feature maps of the input to the same size; Input convolution unit, use 1×1 convolution to reduce the channel depth of the scaled continuous feature map to 1; A channel splicing unit performs channel splicing on multiple continuous feature maps output by the input convolution unit; Softmax unit, uses the Softmax function to activate the continuous feature map after channel splicing to obtain the weight map; A weight graph division unit, which divides the weight graph into a plurality of sub-weight graphs; The weighted aggregation unit performs feature concatenation after weighting the sub-weight map and the corresponding continuous feature map; The output convolution unit aggregates the output of the weighted aggregation unit through a 1×1 convolution and outputs the fused features.

5. The method for visual target segmentation of a noisy image according to claim 4, characterized in that: The fusion feature is expressed as: Among them, O represents the fusion feature, W i represents the weighted graph, S i represents a continuous feature map and ⊙ represents a dot product operation.

6. The visual target segmentation method for noisy images according to claim 1, characterized in that: The method of acquiring a noisy image, inputting the image into a backbone network, learning noise distribution and noise level information, and outputting a denoised image includes: The noise observation value of the noisy image is input into the noise estimation subnetwork of the convolutional denoising blind network, the noise distribution of the noise observation value is learned, and the estimated noise level map is output; The noise observation value and the estimated noise level map of the noisy image are input into the non-blind denoising sub-network of the convolutional denoising blind network to learn the noise distribution and noise level information and output the denoised image.

7. The method for visual target segmentation of a noisy image according to claim 6, characterized in that: The noise estimation sub-network is a five-layer fully convolutional network; the non-blind denoising sub-network is a convolutional network based on a codec architecture.

8. The visual target segmentation method for noisy images according to claim 1, characterized in that: The acquisition of pixel-by-pixel embedded image features includes: Using a deep learning model to extract features from the denoised image to obtain a feature map, and using a deconvolution network to obtain an amplified feature map; Classify each pixel in the enlarged feature map to determine the category to which the pixel belongs; According to the category to which the pixel belongs, each pixel is predicted, the position and shape information of each pixel is obtained, and the pixel-by-pixel embedded image features are generated based on the input denoised image.

9. The visual target segmentation method for noisy images according to claim 1, characterized in that: The step of inputting the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image includes: The enhanced features are input into a multilayer perceptron to generate a class prediction value sequence and a mask prediction value sequence; a dot product operation is performed on the pixel-by-pixel embedded image features and the mask prediction value sequence to obtain a binary mask value; the class prediction value sequence and the binary mask value are bit-wise multiplied and then binarized to obtain a visual target segmentation result of the noisy image.

10. A visual target segmentation device for a noisy image, characterized in that: include: The preprocessing module is used to obtain the noisy image, input it into the backbone network, learn the noise distribution and noise level information, and output the denoised image; A mask prediction module is used to input the denoised image into an improved MaskFormer model: after a pixel-level decoder, a plurality of continuous feature maps, a pixel-by-pixel embedded image feature and a query sequence are obtained; Input multiple continuous feature maps into the multi-scale feature fusion module GAMS, and obtain a weight map after channel downsampling and channel splicing; divide the weight map into multiple sub-weight maps, weight the multiple continuous feature maps respectively, obtain multiple weighted features, and perform convolution aggregation to output fusion features; input the fusion features and the query sequence into the improved TransFormer decoder, perform mask processing on the query sequence, and obtain a binary mask prediction value; The keys and values ​​of the fused features are downsampled using a depth-wise separable convolution operation. The attention weights between the query and the downsampled keys are calculated using lightweight mask attention and standard cross attention, and the downsampled values ​​are weighted summed according to the attention weights to obtain enhanced features. The result acquisition module is used to input the enhanced features into a multi-layer perceptron to obtain a visual target segmentation result of the noisy image.

Images