A method and device for denoising sonar images based on a generative adversarial network combined with contrastive learning

Abstract

A kind of fusion contrast learning's generation confrontation network sonar image denoising method and device, its method includes: constructing generation network, input noisy image, through fusion attention mechanism module and contrast learning module, carry out image denoising processing;Construct discriminant network, input noise-free image without noise and the denoising image generated by generation network, carry out image discrimination;S3, train the confrontation network consisting of generation network and discriminant network, and save the final parameter of training end;S4, input noise image to the generation network that has been trained, obtain denoising image.The fusion attention mechanism module and contrast learning module of the present application can improve the understanding and processing capacity of generation network for input image, enhance the capture and retention capacity of network for image details, accurately restore the main features and detail information of noisy image.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing and underwater detection, and relates to a generative adversarial network sonar image denoising method and device fusing contrast learning. BACKGROUND

[0002] Sonar is a technology that detects and images through the propagation of sound waves in water and the principle of echo. It is widely used in marine exploration and underwater navigation. Because light waves attenuate quickly in water, sound waves have stronger penetration and longer transmission distance, becoming the preferred means of underwater imaging. Sonar generates images of seabed topography or underwater objects by emitting sound waves and receiving echo signals reflected from the seabed or underwater objects. However, due to factors such as water flow, waves, and the movement of sonar equipment itself, echo signals are often affected by noise and distortion, resulting in distortion and noise in the imaging results, which poses a challenge to subsequent data analysis and processing.

[0003] The noise and distortion present in sonar images seriously affect the quality of the images and the accuracy of subsequent processing. Therefore, image denoising has become a key step in sonar data processing. The goal of denoising technology is to remove as much noise as possible from the image while preserving the details of the target object. Traditional image denoising methods such as Gaussian filtering and median filtering can eliminate noise to some extent, but they often weaken important information in the image. In recent years, deep learning-based denoising methods have received widespread attention. These methods can more effectively separate noise and effective signals by learning a large amount of sample data, thereby improving the quality of the image and the processing effect. However, in sonar image analysis, the collection of sonar image data is relatively complex and is limited by environmental conditions, and the collection and sharing of sonar image data may contain sensitive information. Therefore, the difficulty of data collection and the limited number of labeled samples make small sample learning a problem that needs to be solved urgently.

[0004] Contrast learning, as a self-supervised learning method, can effectively improve the performance of the model on small sample data by constructing sample pairs and learning the similarities and differences between samples. In addition, generative adversarial networks have shown excellent performance in image generation and denoising. It consists of a generator and a discriminator, and through the adversarial training of the two, the generator can gradually generate realistic images, and the discriminator can continuously improve its ability to distinguish between real and fake images. This two-way optimization mechanism enables the network to play an important role in the denoising task and provides a new solution for sonar image denoising. SUMMARY

[0005] The application aims to reduce the influence of insufficient sample quantity caused by difficult sonar image acquisition, so as to improve the effect of neural network image denoising.

[0006] The application can autonomously learn optimal parameter configuration and model strategy on a given small sample sonar data set, improve the effect of sonar image denoising through contrast learning, attention mechanism and game confrontation.

[0007] To solve the above technical problems, the first aspect of the application relates to a generation adversarial network sonar image denoising method fusing contrast learning, which specifically comprises the following steps:

[0008] S1, constructing a generation network, inputting a noisy image, and performing image denoising processing through a fusion attention mechanism module and a contrast learning module;

[0009] S2, constructing a discrimination network, inputting a noise-free image without noise and a denoising image generated by the generation network, and performing image discrimination;

[0010] S3, training an adversarial network composed of the generation network and the discrimination network, and saving the final parameters at the end of training;

[0011] S4, inputting a noise image into the trained generation network to obtain a denoising image.

[0012] Step S1 further comprises the following steps:

[0013] S11, designing a feature extraction network composed of three convolutional layers, extracting image features, and converting the input original noisy image P into a multi-layer feature map, denoted as , wherein i represents the i-th layer feature;

[0014] S12, designing a fusion attention mechanism network, taking , that is, the output feature map F of the last layer of the feature extraction network as the network input, and the feature map F last ∈P H×W×C is processed through a channel attention mechanism module to obtain a feature map F', and then processed through a spatial attention mechanism module to obtain a feature map F'', through the fusion attention mechanism network, a feature map F'' of mixed channel and spatial two-dimensional information is obtained, and the calculation formula is as follows:

[0015]

[0016] , wherein M c and M srespectively represent the channel attention mechanism module and the spatial attention mechanism module.The channel attention mechanism module processes the input feature map F to obtain F', and the spatial attention mechanism module processes the input feature map F' to obtain F":

[0017] M c (F)=ξ(C(C(AvgP(F)))+C(C(MaxP(F)))) (2)

[0018] M s (F')=ξ(C(Cat(AvgP(F')+MaxP(F')))) (3)

[0019] Wherein, ξ represents a Sigmoid function, AvgP represents average pooling, MaxP represents maximum pooling, C represents a convolution layer, and Cat represents a concatenation function;

[0020] S13, a contrast learning network is designed, the network is composed of a contrast learning module, a generator G encoder encoded feature layer, and generates contrast learning features through two MLP networks H, calculates cross-entropy loss, and constrains the generation network through the contrast learning network.The contrast learning module designed in step S13 is independently selected to perform cross-entropy operation on different feature maps, and the cross-entropy calculation formula is as follows:

[0021]

[0022] The contrast learning features are represented as:

[0023]

[0024] Wherein represents the output feature of the i-th layer, t i represents the i-th layer feature, and i represents the layer number;

[0025] S14, an image restoration network is designed, the network is composed of multiple sets of MLP networks, each set of MLP network is composed of two deconvolution layers and one convolution layer, and the feature map F" is restored into a denoising image P' generated by the generation network through the image restoration network.

[0026] The second aspect of the application relates to a fusion contrast learning-based generative adversarial network sonar image denoising device, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the fusion contrast learning-based generative adversarial network sonar image denoising method.

[0027] The third aspect of the present application relates to a computer readable storage medium, having stored thereon a program which, when executed by a processor, implements a sonar image denoising method of a generative adversarial network based on fusion contrast learning.

[0028] The working principle of the present application is:

[0029] In the training stage, considering the scarcity caused by the difficulty of sonar image acquisition, the existing denoising model does not have good effect on sonar image denoising, the generator has feature extraction capability for the feature of the target domain noise-free image through the generative adversarial network, so that the generated denoised image has the feature of the noise-free image, and at the same time, through the contrast learning module, the content distribution of the generated denoised image is constrained, so that the denoised image is consistent with the input noisy image in the content distribution, and finally the noise-free image with the same main content as the noisy image is obtained.

[0030] The innovation of the present application is:

[0031] The present application proposes a method of applying a generative adversarial network to the denoising of forward-looking sonar images, and proposes a network model training method combining a fusion attention mechanism and contrast learning for small sample image denoising.

[0032] The present application has the following beneficial effects:

[0033] 1. Solve the problem of few sonar image samples, the present application proposes a sonar image denoising method of a generative adversarial network based on fusion contrast learning, which can be used for denoising network training without using non-paired data sets, i.e. without using data pairs composed of noisy images and noise-free images, simplifying the data preparation process and improving the practicability and universality of the algorithm.

[0034] 2. Solve the problem of poor neural network denoising effect of sonar images caused by the particularity of sonar images, the present application proposes a sonar image denoising method of a generative adversarial network based on fusion contrast learning, which can effectively preserve the content and detail information of the image while removing noise, and improve the denoising effect. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of the method of the present application.

[0036] Figure 2 is a residual network structure diagram of the present application.

[0037] Figure 3 is a contrast learning module flowchart of the present application.

[0038] Figure 4 is a device structure diagram of the present application. DETAILED DESCRIPTION

[0039] In order to better understand the above technical solution, the technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0040] Example 1

[0041] like Figure 1 As shown in the figure, a sonar image denoising method based on generative adversarial network integrating contrastive learning is proposed. The implementation process specifically includes the following steps:

[0042] S1. Build a generative network, input a noisy image, and perform image denoising after fusing the attention mechanism module and the contrastive learning module.

[0043] S11. Design a feature extraction network. The network consists of three convolutional layers to form an encoding module. c represents the convolution kernel size, s represents the convolution step size, p represents padding, and k represents the number of convolution kernels. Conv1: c7s1k64, Conv2: c3s2p1k128, and Conv3: c3s2p1k256 are used as feature extraction networks to transform the original noisy image P into a multi-layer feature. Represents, where i represents the i-th layer feature;

[0044] S12. Design a fusion attention mechanism network, which consists of nine cascaded residual networks. That is, the last layer of the feature extraction network outputs the feature map F as the network input. After passing through two cascaded convolutional layers Conv1: c3s1k64 and Conv2: c3s1k64, instance normalization and ReLU activation function, it first passes through the channel attention mechanism module and then passes through the spatial attention mechanism module to obtain a feature map F that mixes information in both spatial and channel dimensions. Its structure is as follows Figure 2 As shown;

[0045] Feature map F last ∈P H×W×C The feature map F′ is obtained by processing the channel attention mechanism module, and the feature map F″ is obtained by processing the spatial attention mechanism module. By fusing the attention mechanism network, the feature map F″ of the mixed channel and spatial dimensions is obtained. The calculation formula is as follows:

[0046]

[0047] Among them, M c and M s They represent the channel attention mechanism module and the spatial attention mechanism module respectively. The channel attention mechanism module processes the input feature map F to obtain F′, and the spatial attention mechanism module processes the input feature map F′ to obtain F″:

[0048] M c(F) = ξ(C(C(AvgP(F)) + C(C(MaxP(F)))) (2)

[0049] M s (F') = ξ(C(Cat(AvgP(F') + MaxP(F')))) (3)

[0050] wherein ξ represents a Sigmoid function, AvgP represents average pooling, MaxP represents maximum pooling, C represents a convolution layer, and Cat represents a concatenation function;

[0051] S13, a contrast learning network is designed, the network is composed of a contrast learning module, a generator G encoder encoded feature layer, and generates a contrast learning feature through two MLP networks H, calculates a cross-entropy loss, and constrains the generation network through the contrast learning network. The contrast learning feature is represented as:

[0052]

[0053] wherein represents the output feature of the i-th layer, t i represents the i-th layer feature, and i represents the layer number;

[0054] S14, an image restoration network is designed, the network is composed of multiple groups of MLP networks, each group of MLP network is composed of two deconvolution layers DeConv1: c3s0.5p1k128, DeConv2: c3s0.5p1k64 and a convolution layer Conv: c7s1k3, and the feature map F" is restored into the denoising image P' generated by the generation network through the image restoration network.

[0055] S2, a discrimination network is constructed, a noise-free image without noise and a denoising image generated by the generation network are input, and image discrimination is performed. The discrimination network is composed of five groups of convolution layers, wherein Conv1: c4s2p1k64, Conv2: c4s2p1k128, Conv3: c4s2p1k256, Conv4: c4s1p1k512, and Conv: c4s1p1k1, each convolution layer contains batch normalization and Leaky ReLU activation function, and the discrimination network finally outputs a value between 0-1, representing the probability that the input image is a noise-free image, which is used to measure the denoising effect.

[0056] S3, the adversarial network composed of the generation network and the discrimination network is trained, and the final parameters at the end of training are saved. During the training of the generation adversarial network, the overall training loss is composed of the sum of the adversarial loss and the contrast learning loss.

[0057] Adversarial loss is used to ensure that the denoised image G(X) generated by the generator G is visually similar to the true noise-free image Y, making the denoised image as similar as possible to the noise-free image, so that the generator G can generate a sufficiently realistic denoised image and the discriminator D cannot distinguish between the denoised image and the noise-free image, thereby achieving the denoising effect. The adversarial loss formula is as follows:

[0058]

[0059] Contrastive learning loss is used to constrain the generative network and solve the distortion problem that may occur in the generative network during the image generation process, so that the distribution of the generated image G(X) is consistent with the distribution of the noisy image X. Contrastive learning loss is composed of cross entropy loss;

[0060]

[0061] Where t represents the number of patches in each layer of the feature map t∈{1,2,..,S I}, where S i Indicates that there are S in the i-th layer i spatial location, Indicates that the dimension of the feature vector of the sth patch of the i-th layer is C i , Indicates that in the i-th layer, the features of s are removed from all patchesS;

[0062] The cross entropy loss is calculated as shown in formula (9)

[0063]

[0064] like Figure 3 As shown, the positive example, output, and N negative examples are mapped to K-dimensional vectors t,t + ,t - , normalize the vector to a unit sphere to prevent the space from collapsing or expanding, and calculate the cross entropy loss l, where τ is the scale hyperparameter, often called the temperature coefficient, which represents the probability of the positive sample being selected.

[0065] S4. Input the noisy image into the trained generative network to obtain the denoised image.

[0066] The fused attention mechanism of the present invention can further extract features, improve the generator's understanding and processing capabilities of the input image, and at the same time enhance the network's ability to capture and retain image details; the generative adversarial network can support small-sample sonar image denoising, solving the problem that traditional sonar image denoising requires paired data sets; the contrastive learning module can suppress possible distortion and erroneous information in the generated image, constrain the generated image, and accurately restore the main features and detail information of the noisy image.

[0067] Embodiment 2

[0068] As Figure 4 The embodiment relates to a fusion contrast learning generative adversarial network sonar image denoising device, including a memory and one or more processors, the memory stores executable codes, and the one or more processors execute the executable codes to implement the fusion contrast learning generative adversarial network sonar image denoising method of embodiment 1.

[0069] Embodiment 3

[0070] The embodiment relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the fusion contrast learning generative adversarial network sonar image denoising method of embodiment 1.

[0071] Finally, it should be noted that the above embodiments of the present application are merely examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Those skilled in the art can make some changes and modifications to the technical content described without departing from the technical scope of the present application. Any modification and transformation made without departing from the spirit and principle of the present application should belong to the scope of the technical scheme of the present application.

Claims

1. A method for denoising sonar images using a generative adversarial network with fusion contrastive learning, the method comprising: The method comprises the following steps: S1, constructing a generation network, inputting a noisy image, and performing image denoising processing through a fusion attention mechanism module and a contrast learning module; S2, constructing a discrimination network, inputting a noise-free image and a denoised image generated by the generation network, and performing image discrimination; S3, training an adversarial network composed of the generation network and the discrimination network, and saving the final parameters at the end of training; during the training of the adversarial network, the overall training loss is composed of the sum of the adversarial loss and the contrast learning loss; adversarial loss, for ensuring the generator generated denoised image is visually similar to the real, noise-free image lets the generator generate denoised images that are as close as possible to the noise-free images lets the discriminator be unable to distinguish between the denoised images and the noise-free images, thereby achieving the effect of denoising The contrast learning loss is used to constrain the generation network, solves the distortion problem that may occur in the generation image process of the generation network, and makes the distribution of the generated denoising image consistent with the distribution of the noisy image . ​ S4, inputting a noisy image into the trained generation network to obtain a denoised image; In step S1, the following steps are further included: S11, a design feature extraction network is designed, which is composed of three convolutional layers to form an encoding module, extracts image features, and converts the input original noisy image into a multi-layer feature map, denoted as , wherein represents the layer feature; S12, design a fusion attention mechanism network, and The output feature map of the last layer of the feature extraction network As a network input, the feature map After processing by the channel attention mechanism module, the feature map , and after processing by the spatial attention mechanism module, the feature map Through the fusion attention mechanism network, the feature map with mixed channel and spatial information is obtained ; S13、 Design a contrast learning network, which is composed of a contrast learning module, and select a generator The encoded feature layer is encoded by 2 Network Generate contrast learning features, calculate cross-entropy loss, and constrain the generation network through the contrast learning network; S14, design an image restoration network, the network is composed of multiple groups Network composition, each group The network is composed of two deconvolution layers and one convolution layer. Through the image restoration network, the feature map Restored into a denoised image generated by the generation network .

2. The method of claim 1, wherein the method is a fusion contrastive learning generative adversarial network sonar image denoising method. The fusion attention mechanism network designed in step S12 is composed of nine cascaded residual networks. The residual network is composed of two cascaded convolution layers, which are first subjected to a channel attention mechanism module and then a spatial attention mechanism module after instance normalization and ReLU activation function, to obtain the final feature map.

3. The method of claim 1, wherein the method is a fusion contrastive learning generative adversarial network sonar image denoising method. The discrimination network in step S2 is composed of five groups of convolution layers, each of which contains batch normalization and LeakyReLU activation function. The discrimination network finally outputs a value between 0 and 1, representing the probability that the input image is a noise-free image, which is used to measure the denoising effect.

4. A device for denoising sonar images using a fusion of contrastive learning and a generative adversarial network, characterized in that, The device comprises a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the method of any one of claims 1-3.

5. A computer readable storage medium, characterized in that, A program is stored thereon, which is executed by a processor to implement the method of any one of claims 1-3.

Images