A noisy image segmentation method with phased integration of denoising modules

Abstract

The present invention provides a noisy image segmentation method that incorporates a denoising module in stages. First, the noisy image is input into the backbone network, and after convolution operations, feature maps of four stages are extracted; secondly, the feature maps extracted in the fourth stage are subjected to a dual attention mechanism to obtain a preliminary semantic segmentation result. On this basis, the feature distinctions of different stages of the backbone network are utilized, and multi-stage semantic features are iteratively fused to form a mode in which denoising helps segmentation and segmentation helps denoising; finally, the three semantic segmentation results are combined to form a final segmentation result, and the parameters are further optimized by hybrid cross entropy loss. The present invention utilizes collaborative denoising and segmentation to improve the semantic segmentation accuracy of noisy images, and solves the problem that the denoising link in the existing semantic segmentation method for noisy images loses semantic information, thereby affecting the accuracy of subsequent target category division and the segmentation completeness of the target contour.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a noisy image segmentation method integrating a denoising module in stages. Background Art

[0002] The goal of semantic segmentation is to determine the category of each pixel (such as belonging to the background, person or car, etc.), so as to convert some original images into masks with highlighted regions of interest. At present, many advanced segmentation methods have been applied in various fields, such as autonomous driving, scene parsing, object detection and human-computer interaction. Recently, deep neural networks have achieved remarkable success in semantic segmentation tasks. For example, PSANet, DenseAPP and DANet. However, the premise of the success of these networks depends on high-quality training data sets, that is, clean and noise-free images. However, in practical applications, due to the shooting environment, focus failure and machine shaking, the images collected by the image acquisition device often have different degrees of noise information, such as Gaussian noise, short noise and thermal noise. These noise information are uncontrollable. Even the most sophisticated image acquisition equipment cannot control the environment when shooting real images. Noise information often covers some small texture structures of the image, thereby reducing the ability of the semantic segmentation model. When the current mainstream semantic segmentation model is trained with a noisy data set, the segmentation accuracy is significantly reduced. In order to reduce the interference of noise on semantic information, the most direct approach is to complete the image denoising task before performing the semantic segmentation task. A one-step denoising and one-step segmentation cascade method is adopted. Although this method can remove noise, it also causes the loss of some small texture semantic information, and still cannot achieve the result of using clean images to train the segmentation model. When using a noisy dataset to train DANet, the noise information seriously interferes with the image texture structure, not only affecting the target boundary part, making it impossible to accurately obtain context information, but also causing the target area to be incorrectly divided, and the semantic segmentation accuracy is significantly reduced. Adding a denoising module to DANet to form a simple cascade architecture can weaken the damage of noise information to the target texture structure, but it inevitably causes the loss of some texture information, resulting in incorrect positioning in the subsequent one-step semantic segmentation and incomplete segmentation of the target area. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a noisy image segmentation method that incorporates a denoising module in stages, and utilizes collaborative denoising and segmentation to improve the semantic segmentation accuracy of noisy images, thereby solving the problem that the denoising link in the existing semantic segmentation method for noisy images loses semantic information, thereby affecting the accuracy of subsequent target category division and the completeness of target contour segmentation.

[0004] To achieve the above object, the present invention adopts the following technical solution: a noisy image segmentation method integrating a denoising module in stages, comprising the following steps:

[0005] Step S1: On the clean PASCAL VOC 2012 dataset {y (1) ,y (2) ,...,y (m)} is randomly superimposed with Gaussian noise with a mean of 0 and a standard deviation of [0,30] to obtain the noise image training set {x (1) ,x (2) ,...,x (m)};

[0006] Step S2: Transform the noise image x (i) Input into the backbone network ResNet50, and pass through Stage1, Stage2, Stage3 and Stage4 in sequence to extract features of each stage;

[0007] Step S3: Input the feature map f4 generated by the backbone network Stage4 into the dual attention module DAM, refine the features, and output the preliminary segmentation result z1;

[0008] Step S4: Input the feature map f3 generated by the backbone network Stage3 and the preliminary segmentation result z1 into the segmentation and denoising block SDBSC based on stage-by-stage collaboration; first, through the linear transformation formula, the stage-by-stage features of the backbone network, the segmentation results and the multi-scale features of the denoising task are combined to generate a new feature map, and then through the segmentation module SSM in SDBSC, a new segmentation result z2 is generated;

[0009] Step S5: Input the feature map f2 generated by the backbone network Stage 2 and the preliminary segmentation result z2 into the segmentation denoising block SDBSC based on stage-by-stage collaboration, repeat step S4 to generate a new segmentation result z3, and pass it through the clean image y (i) Calculate the mean square error loss L for the denoised image d , L d It can be expressed as:

[0010]

[0011] where y i represents the Ground Truth of pixel i, represents the probability estimate of pixel i, and n represents the number of pixels;

[0012] Step S6: Finally, the segmentation results z1z2z3 of each stage are superimposed to generate a semantic segmentation result with multi-stage feature fusion.

[0013] Step S7: By segmenting the label z Calculate the mixed cross entropy loss L S , which can be expressed as Where Ls1 represents the cross entropy loss:

[0014]

[0015] Among them, p represents the number of pixels in a picture. represents the Ground Truth category of pixel i, represents the probability estimate of pixel i, and Ls2 represents the mIOU loss:

[0016]

[0017] Among them, X represents the predicted pixel set, and Y represents the GT pixel set.

[0018] In a preferred embodiment: according to the method for segmenting noisy images with a denoising module in stages as claimed in claim 1, it is characterized in that: in said S3, it also includes the following steps:

[0019] Step S31: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C ×HW , feature map B HW×C ; Perform matrix multiplication on B and A, and calculate the spatial attention feature map M through the softmax layer HW×HW , which can be expressed as:

[0020]

[0021] Among them, M ji represents the connection between the i-th position and the j-th position in the feature map, H and W represent the length and width of the feature map f4 respectively, and B i represents the i-th position of matrix B, A j Represents the jth position of matrix A. The feature map f4 C×H×W Reshape to C C×HW Multiply it by M and reshape it into a feature map P with the same size as the original feature map C×H×W , multiply it by a scale parameter λ, initialize λ to 0, and continuously assign more weights through learning, which can be expressed as:

[0022]

[0023] Among them, P j represents the jth position of feature map P, C i represents the i-th position of matrix C;

[0024] Step S32: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C ×HW , feature map B HW×C ; Perform matrix multiplication on A and B, and calculate the channel attention feature map N through the softmax layer C×C , which can be expressed as:

[0025]

[0026] Among them, N ji Represents the connection between the i-th position and the j-th channel in the feature map, express The feature map f4 C×H×W Reshape to C C×HW Multiply it with the matrix N and reshape it into a feature map Q with the same size as the original feature map C ×H×W , multiply it by a scale parameter μ, initialize μ to 0, and express it as:

[0027]

[0028] Where Q j Represents the j-th position of the feature map Q.

[0029] In a preferred embodiment: in said S4, the following steps are also included:

[0030] Step S41: Input image R 3×H×W , perform multiple convolution operations with a kernel of 3 and a padding of 1 and use a global average pooling downsampling method to obtain the feature map R 4C×H / 4×W / 4 , obtain the low-level features of the noisy image;

[0031] Step S42: The low-level features of the noise image are subjected to dilated convolution with different expansion factors (rate3, rate6, rate9) to fuse multi-scale information and obtain a new feature map R 4C×H / 4×W / 4 , and then through the linear transformation formula:

[0032]

[0033] where α(·) and is a linear transformation function, and x represents the feature map R after the dilated convolution. 4C×H / 4×W / 4 , stage represents the feature map output by the backbone network, S out Represents the output of the previous stage of semantic segmentation Output is the feature map after fusion;

[0034] Step S43: In the decoding stage of the denoising link, up-sampling is performed using a deconvolution method, and the linear conversion formula in step S42 is applied in the last stage of the decoding stage.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) By introducing the idea of ​​multi-scale and using dilated convolution, the problem of small texture structures in the image being covered by noise is better solved. In addition, by transforming the staged segmentation into a denoising method, the semantic information of each stage, the low-level features in the denoising link, and the segmentation results are effectively combined using linear transformation, so that semantic segmentation promotes image denoising;

[0037] 2) Through a method that utilizes high-level semantic features and low-level semantic features extracted at each stage in the backbone network to enhance the semantic information of the target contour, the noisy image can be denoised and segmented in stages, helping the denoising and segmentation tasks from different levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The figure is a flow chart of a noisy image segmentation method incorporating a denoising module in stages in a preferred embodiment of the present invention.

[0039] Figure 2 This is a comparison chart of the preferred embodiment of the present invention and other noisy image semantic segmentation algorithms. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0043] like Figure 1 As shown, a noisy image segmentation method of the present invention that integrates a denoising module in stages is implemented according to the following steps:

[0044] Step S1: On the clean PASCAL VOC 2012 dataset {y (1) ,y (2) ,...,y (m)} is randomly superimposed with Gaussian noise with a mean of 0 and a standard deviation of [0,30] to obtain the noise image training set {x (1) ,x (2) ,...,x (m)};

[0045] Step S2: Transform the noise image x (i) Input into the backbone network ResNet50, and pass through Stage1, Stage2, Stage3 and Stage4 in sequence to extract features of each stage;

[0046] Step S3: Input the feature map f4 generated by the backbone network Stage 4 into the dual attention module (DAM), refine the features, and output the preliminary segmentation result z1;

[0047] Step S4: Input the feature map f3 generated by the backbone network Stage 3 and the preliminary segmentation result z1 into the segmentation and denoising block based on stage-by-stage collaboration (SDBSC). First, through the linear transformation formula, the stage-by-stage features of the backbone network, the segmentation results and the multi-scale features of the denoising task are combined to generate a new feature map, and then through the segmentation module SSM in the SDBSC, a new segmentation result z2 is generated;

[0048] Step S5: Input the feature map f2 generated by the backbone network Stage 2 and the preliminary segmentation result z2 into the segmentation denoising block based on stage-by-stage collaboration (SDBSC), repeat step S4, generate a new segmentation result z3, and pass the clean image y (i) Calculate the mean square error loss L for the denoised image d , L d It can be expressed as:

[0049]

[0050] where y i represents the Ground Truth of pixel i, represents the probability estimate of pixel i, and n represents the number of pixels;

[0051] Step S6: Finally, the segmentation results z1z2z3 of each stage are superimposed to generate a semantic segmentation result with multi-stage feature fusion.

[0052] Step S7: By segmenting the label z Calculate the mixed cross entropy loss L S , which can be expressed as Where Ls1 represents the cross entropy loss:

[0053]

[0054] Among them, p represents the number of pixels in a picture. represents the Ground Truth category of pixel i, represents the probability estimate of pixel i, and Ls2 represents the mIOU loss:

[0055]

[0056] Among them, X represents the predicted pixel set, and Y represents the GT pixel set.

[0057] In step S3, the following steps are also included:

[0058] Step S31: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C ×HW , feature map B HW×C Perform matrix multiplication on B and A, and calculate the spatial attention feature map M through the softmax layer. HW×HW , which can be expressed as:

[0059]

[0060] Among them, M ji represents the connection between the i-th position and the j-th position in the feature map, H and W represent the length and width of the feature map f4 respectively, and B i represents the i-th position of matrix B, A j Represents the jth position of matrix A. The feature map f4 C×H×W Reshape to C C×HW Multiply it by M and reshape it into a feature map P with the same size as the original feature map C×H×W , multiply it by a scale parameter λ, initialize λ to 0, and continuously assign more weights through learning, which can be expressed as:

[0061]

[0062] Among them, P j represents the jth position of feature map P, C i represents the i-th position of matrix C;

[0063] Step S32: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C×HW , feature map B HW×C . Perform matrix multiplication on A and B, and calculate the channel attention feature map N through the softmax layer. C×C , which can be expressed as:

[0064]

[0065] Among them, N ji Represents the connection between the i-th position and the j-th channel in the feature map, express The feature map f4 C×H×W Reshape to C C×HW Multiply it with the matrix N and reshape it into a feature map Q with the same size as the original feature map C ×H×W , multiply it by a scale parameter μ, initialize μ to 0, and express it as:

[0066]

[0067] Where Q j Represents the j-th position of the feature map Q.

[0068] In the step S4, the following steps are also included:

[0069] Step S41: Input image R 3×H×W , perform multiple convolution operations with a kernel of 3 and a padding of 1 and use a global average pooling downsampling method to obtain the feature map R 4C×H / 4×W / 4 , obtain the low-level features of the noisy image;

[0070] Step S42: The low-level features of the noise image are subjected to dilated convolution with different expansion factors (rate3, rate6, rate9) to fuse multi-scale information and obtain a new feature map R 4C×H / 4×W / 4 , and then through the linear transformation formula:

[0071]

[0072] where α(·) and is a linear transformation function, and x represents the feature map R after the dilated convolution. 4C×H / 4×W / 4 , stage represents the feature map output by the backbone network, S out Represents the output of the previous stage of semantic segmentation Output is the feature map after fusion;

[0073] Step S43: In the decoding stage of the denoising link, up-sampling is performed using a deconvolution method, and the linear conversion formula in step S42 is applied in the last stage of the decoding stage.

[0074] The following is the specific implementation process of the present invention.

[0075] The specific steps of applying the algorithm proposed in the present invention to semantic segmentation of noisy images are as follows:

[0076] 1. In the clean PASCAL VOC 2012 dataset {y (1) ,y (2) ,...,y (m)} is randomly superimposed with Gaussian noise with a mean of 0 and a standard deviation of [0,30] to obtain the noise image training set {x (1) ,x (2) ,...,x (m)};

[0077] 2. The noise image x (i) Input into the backbone network ResNet50, and pass through Stage1, Stage2, Stage3 and Stage4 in sequence to extract features of each stage;

[0078] 3. The feature map f4 generated by the backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C×HW , feature map B HW×C Perform matrix multiplication on B and A, and calculate the spatial attention feature map M through the softmax layer. HW×HW ;

[0079] 4. The feature map f4 generated by the dry network Stage 4 C×H×W By reshaping, we can get the feature maps A C×HW , feature map B HW×C . Perform matrix multiplication on A and B, and calculate the channel attention feature map N through the softmax layer. C×C ;

[0080] 5. The spatial attention feature map M HW×HW and channel attention feature map N C×C Respectively with A C×HW Perform matrix multiplication, then reshape and add to obtain the preliminary segmentation result z1;

[0081] 6. Input the feature map f3 generated by the backbone network Stage 3 and the preliminary segmentation result z1 into the segmentation denoising block based on stage-by-stage collaboration (SDBSC);

[0082] 7. For the input image R 3×H×W , perform multiple convolution operations with a kernel of 3 and a padding of 1 and use a global average pooling downsampling method to obtain the feature map R 4C×H / 4×W / 4, obtain the low-level features of the noisy image;

[0083] 8. The low-level features of the noise image are subjected to dilated convolution with different expansion factors (rate3, rate6, rate9) to fuse multi-scale information and obtain a new feature map R 4C×H / 4×W / 4 , through the linear transformation formula: Get the fused feature map;

[0084] 9. In the decoding stage of the denoising link, the upsampling adopts the deconvolution method, and the linear transformation formula is applied in the last link of the decoding stage to obtain the image after preliminary denoising.

[0085] 10. Pass the denoised image through the backbone network ResNet50, and then repeat steps 3)-5) to obtain the preliminary segmentation result z2.

[0086] 11. Input the feature map f2 generated by the backbone network Stage 2 and the preliminary segmentation result z2 into the stage-by-stage collaboration-based segmentation denoising block (SDBSC), repeat steps 7)-10) to obtain a new segmentation result z3 and a denoised image.

[0087] 12. Superimpose the staged segmentation results z1z2z3 to generate a semantic segmentation result with multi-stage feature fusion

[0088] 13. Calculate the mean square error loss for the denoised image

[0089] 14. By splitting the label z Calculate the mixed cross entropy loss: in

[0090] Figure 2 : is a qualitative comparison result diagram of the algorithm in this embodiment and other methods on the PASCAL VOC 2012 dataset with a Gaussian noise standard deviation of 20. Figure 2 As can be seen from the three columns (c), (d), and (e), after adding the denoising module, the semantic segmentation quality is still unsatisfactory. The target range is misidentified, and the target boundary is imprecise and not smooth. Figure 2 (f) shows the result of DMS. The problem of misidentification has been improved to a certain extent, but the boundary area is not well identified. Figure 2 (g) shows the results of the algorithm in this embodiment. It can be clearly seen that both multi-category and small targets can be well segmented. Moreover, it can be seen from the images in the 4th and 5th rows that the algorithm in this embodiment has achieved a significant improvement in the target boundary.

Claims

1. A noisy image segmentation method incorporating a denoising module in stages, characterized in that: The following steps are involved: Step S1: On the clean PASCAL VOC 2012 dataset {y (1) ,y (2) ,...,y (m) } is randomly superimposed with Gaussian noise with a mean of 0 and a standard deviation of [0,30] to obtain the noise image training set {x (1) ,x (2) ,...,x (m) }; Step S2: Transform the noise image x (i) Input into the backbone network ResNet50, and pass through Stage1, Stage2, Stage3 and Stage4 in sequence to extract features of each stage; Step S3: Input the feature map f4 generated by the backbone network Stage4 into the dual attention module DAM, refine the features, and output the preliminary segmentation result z1; Step S4: Input the feature map f3 generated by the backbone network Stage3 and the preliminary segmentation result z1 into the segmentation and denoising block SDBSC based on stage-by-stage collaboration; first, through the linear transformation formula, the stage-by-stage features of the backbone network, the segmentation results and the multi-scale features of the denoising task are combined to generate a new feature map, and then through the segmentation module SSM in SDBSC, a new segmentation result z2 is generated; Step S5: Input the feature map f2 generated by the backbone network Stage 2 and the preliminary segmentation result z2 into the segmentation denoising block SDBSC based on stage-by-stage collaboration, repeat step S4 to generate a new segmentation result z3, and pass it through the clean image y (i) Calculate the mean square error loss L for the denoised image d , L d It is expressed as: where y i represents the Ground Truth of pixel i, represents the probability estimate of pixel i, and n represents the number of pixels; Step S6: Finally, the segmentation results z1z2z3 of each stage are superimposed to generate a semantic segmentation result with multi-stage feature fusion. Step S7: By segmenting the label z Calculate the mixed cross entropy loss L S , expressed as Where L S1 Denotes the cross entropy loss: Among them, p represents the number of pixels in a picture. represents the Ground Truth category of pixel i, represents the probability estimate of pixel i, L S2 Represents mIOU loss: Among them, X represents the predicted pixel set, and Y represents the ground truth pixel set; In step S3, the following steps are also included: Step S31: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C×HW , feature map B HW×C ; Perform matrix multiplication on B and A, and calculate the spatial attention feature map M through the softmax layer HW×HW , expressed as: Among them, M ji represents the connection between the i-th position and the j-th position in the feature map, H and W represent the length and width of the feature map f4 respectively, and B i represents the i-th position of matrix B, A j Represents the jth position of matrix A; the feature map f4 C×H×W Reshape to C C×HW Multiply it by M and reshape it into a feature map P with the same size as the original feature map C×H×W , multiply it by a scale parameter λ, initialize λ to 0, and continuously assign more weights through learning, which can be expressed as: Among them, P j represents the jth position of the feature map P, C i represents the i-th position of matrix C; Step S32: Feature map f4 generated by backbone network Stage 4 C×H×W By reshaping, we can get the feature maps A C×HW , feature map B HW×C ; Perform matrix multiplication on A and B, and calculate the channel attention feature map N through the softmax layer C×C , expressed as: Among them, N ji Represents the connection between the i-th position and the j-th channel in the feature map, express The feature map f4 C×H×W Reshape to C C×HW Multiply it with the matrix N and reshape it into a feature map Q with the same size as the original feature map C×H×W , multiply it by a scale parameter μ, initialize μ to 0, and express it as: Where Q j Represents the j-th position of the feature map Q.

2. The noisy image segmentation method according to claim 1, characterized in that: In said S4, the following steps are also included: Step S41: Input image R 3×H×W , perform multiple convolution operations with a kernel of 3 and a padding of 1 and use a global average pooling downsampling method to obtain the feature map R 4C×H / 4×W / 4 , obtain the low-level features of the noisy image; Step S42: The low-level features of the noise image are subjected to dilated convolution with different expansion factors (rate3, rate6, rate9) to fuse multi-scale information and obtain a new feature map R 4C×H / 4×W / 4 , and then through the linear transformation formula: where α(·) and is a linear transformation function, and x represents the feature map R after the dilated convolution. 4C×H / 4×W / 4 , stage represents the feature map output by the backbone network, S out Represents the output of the previous stage of semantic segmentation Output is the feature map after fusion; Step S43: In the decoding stage of the denoising link, up-sampling is performed using a deconvolution method, and the linear conversion formula in step S42 is applied in the last stage of the decoding stage.

Images