A sonar image denoising method and system based on semantic guidance
By adopting a semantic guidance-based denoising method in sonar image processing, using RFA and GSE modules, the problems of spatial correlation and information loss of speckle noise in sonar images are solved, and high-quality speckle removal effect is achieved.
Patent Information
- Application Number
- CN202410365866.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-28
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2044-03-28
AI Technical Summary
The spatial correlation of speckle noise in sonar images conflicts with the basic assumptions of blind-spot networks, resulting in the learning of the identity mapping of noise images, and the blind-spot structure causes a large number of pixels to fail to participate in the prediction of clean signals, resulting in information loss and reducing the quality of despot removal.
Using a semantic guidance-based sonar image denoising method, by constructing a despeculation noise model including the first convolutional layer, the RFA module, the GSE module and the output convolutional module, the diamond-shaped masking convolutional layer, the receptive field enhancement module and the global semantic enhancement module are used to overcome noise spatial correlation and reduce information loss.
It realizes the high-quality sonar image despot removal effect, and has the dual advantages of effective noise removal and high-quality texture recovery, and overcomes the problems of spatial correlation and information loss of speckle noise.
Smart Images
Figure CN118261818B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and more particularly to a sonar image denoising method and system based on semantic guidance. Background Art
[0002] Currently, nanoimaging technology is achieved by sonar systems that visualize underwater environments by emitting sound waves and coherently demodulating the reflected waves. Due to the inherent characteristics of its coherent acquisition process, sonar images are inevitably damaged by speckle noise. This makes sonar images difficult to interpret, which has a negative impact on downstream visual tasks such as image segmentation, detection, and recognition. Self-supervised learning methods can learn to remove speckle directly from noisy images without the need for clean samples. This is based on blind-spot networks (BSNs). By taking the pixels around the noisy pixels as input and setting the noisy pixels themselves as learning targets, the neural network has the ability to efficiently remove speckle.
[0003] However, although the sonar image despeckling method based on the blind spot network avoids the difficulty of obtaining real sonar images without speckle noise, it always faces two problems in achieving the goal of high-quality despeckling:
[0004] First, the inherent spatial correlation of sonar image speckle noise conflicts with the basic assumption of the blind spot network - the blind spot network assumes that noise is independent pixel by pixel. Therefore, training the blind spot network directly on sonar images may learn a noise image identity mapping instead of an effective denoiser.
[0005] Second, the inherent blind spot structure of the blind spot network is achieved by superimposing masked pixels on the receptive field of the pixels to be denoised. This prevents the neural network from utilizing the noise information of the denoised pixels themselves, but also causes a large number of pixels to fail to participate in the prediction of clean signals, resulting in a significant loss of available information and reducing the quality of clean texture signal recovery during the despeckle process.
[0006] Therefore, how to simultaneously overcome the spatial correlation of speckle noise and reduce information loss to achieve high-quality sonar image despeckle effect is an urgent problem that technicians in this field need to solve. Summary of the invention
[0007] In view of this, the present invention provides a sonar image denoising method and system based on semantic guidance, which simultaneously overcomes the problems of spatial correlation of speckle noise and reduces information loss, and achieves high-quality sonar image de-speckling effect.
[0008] In order to achieve the above object, the present invention adopts the following technical solution:
[0009] A sonar image denoising method based on semantic guidance, comprising:
[0010] Construct a speckle noise removal model including the first convolution layer, RFA module, GSE module and output convolution module;
[0011] Acquire a plurality of sonar images, and input the sonar images into the first convolutional layer to obtain a first extracted feature map;
[0012] Inputting the first extracted feature map into the RFA module to obtain an enhanced feature map;
[0013] Inputting the enhanced feature map and the sonar image into the GSE module respectively to obtain a semantic fusion feature map;
[0014] Inputting the semantic fusion feature map into the output convolution module to obtain a denoised nanoimage;
[0015] The speckle noise removal model is trained based on the denoised sonar image and the sonar image loss function to obtain a trained speckle noise removal model;
[0016] The sonar image to be processed is input into the trained speckle noise removal model to obtain the corresponding denoised sonar image.
[0017] Preferably, the RFA module includes a first branch network and a second branch network in parallel;
[0018] Inputting the first extracted feature map into the first branch network and the second branch network respectively, and obtaining a first local feature map and a second local feature map correspondingly;
[0019] The first local feature map and the second local feature map are fused to obtain the enhanced feature map.
[0020] Preferably, the first branch network comprises: a D-Conv convolution layer, a first PD operation layer and a first F-Conv convolution layer;
[0021] The first extracted feature map is input into the D-Conv convolution layer to obtain an aggregated feature map;
[0022] The aggregated feature map is input to the first PD operation layer for spatial rearrangement to obtain a first rearranged feature map;
[0023] The first rearranged feature map is input into the first F-Conv convolutional layer to obtain the first local feature map.
[0024] Preferably, the second branch network comprises: a Conv convolution layer, a second PD operation layer and a P-Conv convolution layer;
[0025] The first extracted feature map is input into the Conv convolution layer to obtain a second extracted feature map;
[0026] The second extracted feature map is input into the second PD operation layer for spatial rearrangement to obtain a second rearranged feature map;
[0027] The second rearranged feature map is input into the P-Conv convolution layer to obtain the second local feature map.
[0028] Preferably, the weights in the D-Conv convolution layer whose Manhattan distance from the center weight of the convolution kernel is no more than 4 are all set to 0 and cannot be learned;
[0029] The weights are specifically set as:
[0030] W D (i, j) = 0, where d ij =|i-6|+|j-6|≤4, 1≤i, j≤11
[0031] Among them, W D (i, j) represents the weight of the D-Conv convolution kernel at the i-th row and j-th column position, d ij W represents the Manhattan distance between the weight at the i-th row and j-th column of the D-Conv convolution kernel and the center weight at the 6th row and 6th column of the convolution kernel center. D (i, j) = 0 means that the weight at the i-th row and j-th column of the D-Conv convolution kernel is set to 0 and cannot be learned.
[0032] Preferably, the F-Conv convolutional layer weights are specifically set as:
[0033] W F (a, b)=0, where (a, b)∈{(1, 2), (2, 1) (2, 3) (3, 2)}
[0034] Among them, W F (a, b) represents the weight of the F-Conv convolution kernel at the ath row and bth column, W F (a, b) = 0 means that the weight at the a-th row and b-th column of the F-Conv convolution kernel is set to 0 and cannot be learned.
[0035] Preferably, the P-Conv convolution layer weights are specifically set as:
[0036] W P (c,d)=0,where(c,d)∈{(1,1),(1,3),(2,2),(3,1),(3,3)}
[0037] Among them, W P(c, d) represents the weight of the P-Conv convolution kernel at the cth row and dth column position, W P (c, d) = 0 means that the weight at the c-th row and d-th column of the P-Conv convolution kernel is set to 0 and cannot be learned.
[0038] Preferably, the GSE module includes: a semantic encoder, a semantic injection module and a PD -1 Operational layer;
[0039] The sonar image is input into the semantic encoder for feature extraction to obtain global semantic features;
[0040] The global semantic feature and the enhanced feature map are respectively input into the semantic injection module to obtain a semantic enhanced feature map;
[0041] The semantically enhanced feature map is input into the PD -1 The operation layer obtains the fusion feature.
[0042] Preferably, the semantic injection module comprises a plurality of sequentially connected semantic injection networks;
[0043] The multiple semantic injection networks have the same structure, and all include: a semantic injection layer, a second F-Conv convolution layer, a ReLU activation layer, and a second convolution layer;
[0044] The enhanced feature map and the global semantic feature are respectively input into the semantic injection layer to obtain a first enhanced feature;
[0045] The first enhanced feature is sequentially processed by the second F-Conv convolution layer, the ReLU activation layer and the second convolution layer, and then fused with the enhanced feature map to obtain a first output feature map;
[0046] The first output feature map and the global semantic feature are respectively input into the next semantic injection network for processing to obtain a second output feature map, and so on, through processing through all the semantic injection networks, finally obtaining the semantic enhancement feature.
[0047] A sonar image denoising system based on semantic guidance, comprising: a model building module, a feature extraction module, a feature processing module, a model training module and a result output module;
[0048] The model building module is used to build a speckle noise removal model including a first convolution layer, an RFA module, a GSE module and an output convolution module;
[0049] The feature extraction module is used to obtain a plurality of sonar images, and input the sonar images into the first convolution layer to obtain a first extracted feature map; and input the first extracted feature map into the RFA module to obtain an enhanced feature map;
[0050] The feature processing module is used to input the enhanced feature map and the sonar image into the GSE module respectively to obtain a semantic fusion feature map; input the semantic fusion feature map into the output convolution module to obtain a denoised sonar image;
[0051] The model training module is used to train the speckle noise removal model based on the joint loss function of the denoised sonar image and the sonar image to obtain a trained speckle noise removal model;
[0052] The result output module is used to input the sonar image to be processed into the trained speckle noise removal model to obtain the corresponding denoised sonar image.
[0053] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a sonar image denoising method and system based on semantic guidance, which has the following beneficial effects:
[0054] 1. Diamond-shaped Masked Convolution Layer (D-Conv), the weights within the range of Manhattan distance no more than 4 from the center weight of the convolution kernel are set to 0 and cannot be learned. When aggregating the information of pixels in the area around the pixel to be denoised, avoid using pixels with strong noise correlation with the noise of the pixel to be denoised, and effectively use pixels in the area of the pixel to be denoised with weak or no correlation with its noise, so as to overcome the spatial correlation of noise and enhance the utilization of local high-value information.
[0055] 2. Through the Receptive Field Augmentation (RFA) module, an enhanced receptive field is introduced for the pixels to be denoised, which not only avoids the use of pixels with strong spatial correlation with the noise of the pixels to be denoised, but also can use more pixels with weak spatial correlation with the noise of the pixels to be denoised. Therefore, the RFA module can overcome the spatial correlation of speckle noise while enhancing the utilization of the field pixel information within the receptive field.
[0056] 3. Through the Global Semantic Enhancement (GSE) module, the global semantic information of the entire image can be extracted to enhance the blind spot network's ability to perceive and utilize global information, thereby effectively alleviating the inherent information loss of the blind spot network while meeting the basic assumptions of the blind spot network.
[0057] 4. By combining the RFA module and the GSE module, the spatial correlation of speckle noise and the problem of reducing information loss are overcome at the same time. It can have the dual advantages of effective noise removal and high-quality texture restoration, and achieve high-quality sonar image despeckling effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0059] Figure 1 A flow chart of a sonar image denoising method based on semantic guidance provided by the present invention.
[0060] Figure 2 This is a schematic diagram of the structure of the speckle noise removal model provided by the present invention.
[0061] Figure 3 This is a schematic diagram of the RFA module structure provided by the present invention.
[0062] Figure 4 The PD operation layer and the PD provided by the present invention -1 Schematic diagram of feature map transformation at the operation layer.
[0063] Figure 5 This is a schematic diagram of the semantic injection network structure provided by the present invention.
[0064] Figure 6 A schematic diagram of the structure of a sonar image denoising system based on semantic guidance provided by the present invention. DETAILED DESCRIPTION
[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] Example 1
[0067] like Figure 1 As shown, an embodiment of the present invention discloses a sonar image denoising method based on semantic guidance, comprising:
[0068] Construct a speckle noise removal model including the first convolution layer, RFA module, GSE module and output convolution module;
[0069] Acquire multiple sonar images, and input the sonar images into a first convolutional layer to obtain a first extracted feature map;
[0070] Inputting the first extracted feature map into the RFA module to obtain an enhanced feature map;
[0071] The enhanced feature map and sonar image are input into the GSE module respectively to obtain the semantic fusion feature map;
[0072] Input the semantic fusion feature map into the output convolution module to obtain the denoised nanoimage;
[0073] The speckle removal noise model is trained based on the denoised sonar image and sonar image loss function to obtain a trained speckle removal noise model;
[0074] The sonar image to be processed is input into the trained speckle noise removal model to obtain the corresponding denoised sonar image.
[0075] Example 2
[0076] Speckle noise is a multiplicative and spatially correlated noise that appears as granular light and dark textures in sonar images, which makes sonar images difficult to interpret and thus has a negative impact on downstream vision tasks such as image segmentation, detection, and recognition. Therefore, it is of great significance to provide a reliable sonar image despeckle method.
[0077] Based on the problem of how to simultaneously overcome the spatial correlation of speckle noise and reduce information loss, an embodiment of the present invention discloses a sonar image denoising method based on semantic guidance, comprising:
[0078] like Figure 2 As shown in the figure, a speckle noise removal model including the first convolution layer, the RFA module, the GSE module and the output convolution module is constructed.
[0079] A plurality of sonar images are acquired, and the sonar images are input into a first convolutional layer to obtain a first extracted feature map.
[0080] Preferably, the first convolution layer of this embodiment adopts a 1x1 convolution layer, the number of input channels is 1, and the number of output channels is 24. Through the first convolution layer, the sonar image with a channel number of 1 can be dimensionally upgraded to become a first extracted feature map with a channel number of 24. The output side of the first convolution layer is connected to a ReLU activation layer.
[0081] Preferably, the first extracted feature map F0 is: F0∈R h×w×24 , where h, w and 24 represent the height, width and number of channels of the first extracted feature map respectively.
[0082] The first extracted feature map is input into the RFA module to obtain an enhanced feature map.
[0083] Preferably, Figure 3 As shown, the RFA module includes a first branch network and a second branch network in parallel;
[0084] Inputting the first extracted feature map into the first branch network and the second branch network respectively, and obtaining a first local feature map and a second local feature map correspondingly;
[0085] The first local feature map is fused with the second local feature map to obtain an enhanced feature map.
[0086] Preferably, the first local feature map and the second local feature map are subjected to an element-wise addition operation to obtain an enhanced feature map F1: F1∈R h×w×128 , where h, w and 128 represent the height, width and number of channels of the enhanced feature map respectively.
[0087] Preferably, the first branch network includes: a D-Conv convolution layer, a first PD operation layer and a first F-Conv convolution layer;
[0088] The first extracted feature map is input into the D-Conv convolution layer to obtain an aggregated feature map;
[0089] The aggregated feature map is input into the first PD operation layer for spatial rearrangement to obtain a first rearranged feature map;
[0090] The first rearranged feature map is input into the first F-Conv convolutional layer to obtain a first local feature map.
[0091] Preferably, the second branch network includes: a Conv convolution layer, a second PD operation layer and a P-Conv convolution layer;
[0092] The first extracted feature map is input into the Conv convolution layer to obtain the second extracted feature map;
[0093] The second extracted feature map is input into the second PD operation layer for spatial rearrangement to obtain a second rearranged feature map;
[0094] The second rearranged feature map is input into the P-Conv convolution layer to obtain the second local feature map.
[0095] Preferably, the D-Conv convolution layer of this embodiment adopts a diamond-shaped masked convolution (D-Conv) convolution layer with a convolution kernel size of 11×11.
[0096] Preferably, considering the spatial correlation characteristics of sonar image noise, that is, given a pixel x to be denoised, the noise n applied to it x The noise applied to the pixel y∈R(x) within a certain range R(x) is correlated, and the noise is related to the Manhattan distance d between pixel x and pixel y. xy =|x i -y i |+|x j +y j | gradually decreases with the increase of i and i Represents the row coordinates of pixel x and pixel y in the entire image, x j and j Represents the column coordinates of pixel x and pixel y in the entire image, respectively. xy When ≥ 4, the noise spatial correlation between pixel x and pixel y has been reduced to a level that can be ignored. In order to better utilize the domain information of the pixel to be denoised to restore its clean signal while not being affected by the pixels in its domain that have noise correlation with it, the present invention specifically designs the convolution kernel and its weights of D-Conv.
[0097] Preferably, Figure 3 As shown in the figure, the weights of the black covered areas in the D-Conv convolutional layer are set to 0 and cannot be learned; the weights of the white parts are learnable and will be automatically adjusted as the network is optimized.
[0098] Preferably, the weights in the D-Conv convolution layer whose Manhattan distance from the center weight of the convolution kernel is less than 4 are all set to 0 and cannot be learned;
[0099] The weights are specifically set as:
[0100] W D (i, j) = 0, where d ij =|i-6|+|j-6|≤4,1≤i,j≤11
[0101] Among them, W D (i, j) represents the weight of the D-Conv convolution kernel at the i-th row and j-th column position, d ij W represents the Manhattan distance between the weight at the i-th row and j-th column of the D-Conv convolution kernel and the center weight at the 6th row and 6th column of the convolution kernel center. D (i, j) = 0 means that the weight at the i-th row and j-th column of the D-Conv convolution kernel is set to 0 and cannot be learned.
[0102] Preferably, by setting the weights of the D-Conv convolution layer, it is possible to ensure that when the D-Conv convolution layer aggregates the information of pixels in the area surrounding the pixel to be denoised, it avoids using pixels with strong noise correlation with the noise of the pixel to be denoised, and effectively utilizes pixels in the area of the pixel to be denoised that have weak or no correlation with its noise. Accordingly, the feature vector at a specific position on the feature map extracted by the D-Conv convolution layer aggregates the pixel information of the area except for the pixels in the area with strong noise correlation in the current position area, thereby overcoming the spatial correlation of noise and strengthening the utilization of local high-value information.
[0103] Preferably, the first F-Conv convolutional layer has the same structure as the second F-Conv convolutional layer, and the weights are specifically set as:
[0104] W F (a, b) = 0, where (a, b) ∈ {(1, 2), (2, 1), (2, 3), (3, 2)}
[0105] Among them, W F (a, b) represents the weight of the F-Conv convolution kernel at the ath row and bth column, W F (a, b) = 0 means that the weight at the a-th row and b-th column of the F-Conv convolution kernel is set to 0 and cannot be learned.
[0106] Preferably, in this embodiment, the convolution kernel size of the first F-Conv convolution layer is 3×3, such as Figure 3 As shown in the figure, the weights of the black covered areas in the F-Conv convolution kernel are reset to 0 and cannot be learned; the first F-Conv convolution layer adopts the form of dilated convolution during use, and the dilated scaling ratio is set to 2.
[0107] Preferably, the weights of the P-Conv convolutional layer are specifically set to:
[0108] W P (c, d) = 0, where (c, d)∈{(1, 1), (1, 3), (2, 2), (3, 1), (3, 3)}
[0109] Among them, W P (c, d) represents the weight of the P-Conv convolution kernel at the cth row and dth column position, W P (c, d) = 0 means that the weight at the c-th row and d-th column of the P-Conv convolution kernel is set to 0 and cannot be learned.
[0110] Preferably, the convolution kernel size of the P-Conv convolution layer in this embodiment is 3×3, such as Figure 3As shown in the figure, the weights of the black covered areas in the P-Conv convolution kernel are reset to 0 and cannot be learned; the P-Conv convolution layer adopts the form of dilated convolution during use, and the dilated scaling ratio is set to 2.
[0111] Preferably, by setting the first F-Conv convolution layer and the P-Conv convolution layer, it can be ensured that the RFA module can use two branches to enhance the receptive field of the pixels to be denoised while also ensuring the blind spot structure requirements of the network, thereby avoiding the noise information of the pixels to be denoised from participating in the prediction process of their clean signals, thereby avoiding learning the identity mapping.
[0112] Preferably, by using the first F-Conv convolution layer and the P-Conv convolution layer to process the extracted 2-branch feature maps respectively and perform element-by-element addition, the final enhanced feature map can not only contain all the information aggregated by the 2 branches of the RFA module, which is equivalent to obtaining a feature map with a larger range, but also avoid the feature at the (i, j) position of the feature map from being affected by the field pixels in the input image that have noise correlation with the pixels at the (i, j) position.
[0113] Preferably, Figure 4 As shown, the first PD operation layer and the second PD operation layer have the same structure and data processing method, both of which spatially rearrange the extracted feature maps to avoid interference from noise spatial correlation, thereby having the functions of overcoming noise spatial correlation and retaining the basic structure of the blind spot network.
[0114] Preferably, the pixel-shuffle downsampling (PD) operation layer has an adjustable hyperparameter, denoted as a step factor. In this embodiment, the step factor is set to 5.
[0115] The enhanced feature map and sonar image are input into the GSE module respectively to obtain the semantic fusion feature map.
[0116] Preferably, the GSE module includes: a semantic encoder, a semantic injection module and a PD -1 Operational layer;
[0117] The sonar image is input into the semantic encoder for feature extraction to obtain global semantic features;
[0118] The global semantic features and enhanced feature maps are respectively input into the semantic injection module to obtain the semantic enhanced feature map;
[0119] Semantic enhanced feature map input to PD -1 Operation layer, obtain the semantic fusion feature map.
[0120] Preferably, Figure 4 As shown, the inverse pixels are randomly shuffled and downsampled PD -1The operation layer implements the inverse operation of the PD operation layer, PD -1 The operation layer has an adjustable hyperparameter, denoted as the step factor, and the step factor in this embodiment is set to 5.
[0121] Preferably, the semantic encoder extracts the semantic representation of high-value information about the potential clean image from the sonar image, and the extracted global semantic feature z ∈ R 1×1×k is highly compressed and compact, avoiding the introduction of global semantic information from leaking too much noise information at blind spots; the semantic encoder is denoted as: f SE : R h×w×1 →R 1×1×k , the dimension of the global semantic feature is much smaller than the total dimension of the input sonar image Y ∈ R h×w×1 , that is, k << hw; the global semantic feature extraction process is: z = f SE (Y).
[0122] Preferably, the image feature extraction network pre-trained on large-scale image datasets such as ImageNet is used as the network of the semantic encoder of the present invention, including at least: ResNet, DenseNet, VGG-Net or Vision-Transformer.
[0123] Preferably, the semantic encoder in this embodiment: pre-trains on ImageNet for image classification tasks by selecting the ResNet18 network structure. After the pre-training is completed, the last linear classification layer of ResNet18 is removed and the remaining image feature extraction part of the network is selected as the semantic encoder structure of the present invention, and the network weights pre-trained on the ImageNet image classification task are loaded.
[0124] Preferably, the semantic encoder in this embodiment includes, connected in sequence: an input layer, a convolutional layer, a residual block, and a global average pooling layer;
[0125] Input layer: Receives an RGB image of size 160x160.
[0126] Convolutional layer: There are a total of 4 convolutional layers, and each convolutional layer uses a 3x3 convolutional kernel and a ReLU activation function to extract local features of the image.
[0127] Residual block: There are a total of 8 residual blocks, and each residual block is composed of two convolutional layers and a skip connection, which is used to solve the problems of gradient disappearance and gradient explosion in deep convolutional neural networks.
[0128] Global average pooling layer: Performs global average pooling on the feature map and converts the feature map into a one-dimensional vector.
[0129] Preferably, the semantic injection module includes a plurality of sequentially connected semantic injection networks;
[0130] like Figure 5 As shown, multiple semantic injection networks have the same structure, including: a semantic injection layer, a second F-Conv convolution layer, a ReLU activation layer, and a second convolution layer;
[0131] The enhanced feature map and the global semantic feature are respectively input into the semantic injection layer to obtain the first enhanced feature;
[0132] The first enhanced feature is processed by the second F-Conv convolution layer, the ReLU activation layer and the second convolution layer in sequence and then fused with the enhanced feature map to obtain the first output feature map;
[0133] The first output feature map and the global semantic feature are respectively input into the semantic injection layer in the next semantic injection network to obtain the second enhanced feature;
[0134] The second enhanced feature is processed by the second F-Conv convolution layer, the ReLU activation layer and the second convolution layer in sequence and then fused with the first output feature map to obtain a second output feature map;
[0135] The second output feature map and the global semantic feature are respectively input into the next semantic injection network for processing to obtain the third output feature map, and so on. After being processed by all the semantic injection networks, the final output feature map is used as the semantic enhancement feature.
[0136] Preferably, in this embodiment, the second F-Conv convolution layer adopts a dilated convolution, the convolution kernel size is 3x3, the diffusion parameter S of the dilated convolution is equal to 2, and the second convolution layer adopts a 1x1 convolution layer.
[0137] Preferably, the mth output feature map output by the mth semantic injection network is: F m ∈R h×w×c , the global semantic feature extracted by the semantic encoder is z, and the semantic injection layer in the m+1th semantic injection network is transformed into a semantic affine function f SA : R 1×1×k →R 1×1×c Get a calibrated semantic feature z′=f whose channel dimension matches Fm SA (z); and then through the semantic injection function f SI : R h×w×c ×R 1×1×c →R h×w×c Coupling F m and Z′, realizing the semantic information from Z′ to F m transfer to achieve F m Semantic information enhancement;
[0138] The semantic injection function achieves coupling F by channel-by-channel multiplication, channel-by-channel addition, channel-by-channel multiplication and addition, and cascading along the channel dimension.m And Z', this embodiment adopts channel-by-channel multiplication, the m+1th enhanced feature: in Indicates channel-by-channel multiplication operation;
[0139] The m+1th enhanced feature is further aggregated through the F-Conv convolution layer with a convolution kernel size of 3x3, the ReLU activation layer, and a 1x1 second convolution layer to obtain the m+1th output feature map:
[0140] F m+1 =f 1×1卷积 (f F-Conv (F m+1 ′))+F m
[0141] Among them, f F-Conv (·) represents the convolution operation performed by the second F-Conv convolutional layer, f 1×1卷积 (·) denotes the convolution operation performed by the second 1x1 convolutional layer.
[0142] The semantically fused feature map is input into the output convolution module to obtain the denoised nanoimage.
[0143] Preferably, the output convolution module includes a plurality of convolutional layers with decreasing numbers of input and output channels connected in series, and a ReLU activation layer is arranged between adjacent convolutional layers. The present embodiment adopts three 1x1 convolutional layers, and the number of input channels / the number of output channels correspond to 128 / 64, 64 / 32, and 32 / 1, respectively, thereby realizing a gradual reduction in the dimension of the feature map, thereby converting the learned high-dimensional feature map into a denoising result map with a final channel number of 1.
[0144] The speckle removal noise model is trained based on the denoised sonar image and sonar image loss function to obtain a trained speckle removal noise model.
[0145] Preferably, this embodiment adopts a self-supervised learning training paradigm to train the speckle noise removal model until the loss function converges, and the optimizer adopts the Adam optimizer, and the sonar image loss function is specifically:
[0146]
[0147] Where Y represents the input sonar image, The final output denoised nanoimage is shown.
[0148] Preferably, the semantic encoder of the GSE module first loads the weights of the original complete network after pre-training on the ImageNet dataset, and further fine-tunes during the optimization process of the entire network structure proposed in the present invention, thereby promoting the semantic encoder to better adapt to the needs of the sonar image despeckling task during the joint optimization process.
[0149] The sonar image to be processed is input into the trained speckle noise removal model to obtain the corresponding denoised sonar image.
[0150] Preferably, the image despeckle task collected by ARIS Explorer 3000 forward-looking sonar is taken as an example to specifically illustrate the method of the present invention:
[0151] Construct a dataset. Collect no less than 1000 forward-looking sonar images and divide them into training set and test set in a ratio of 8:2. The images of the test set are further patched with a coverage ratio of 20 pixels. The patch size after cropping is 160x160 to meet the needs of the proposed sonar image despeckle framework.
[0152] Constructing a semantically guided self-supervised despeckle model for sonar images:
[0153] The step factors of the first PD operation layer and the second PD operation layer are both set to 5, and the number of semantic injection blocks used is set to 9;
[0154] The semantic encoder of the GSE module uses the ResNet18 pre-trained on the ImageNet large-scale image dataset as the basic structure except for the linear classification layer;
[0155] The output feature map of the GSE module needs to pass through three layers of 1x1 convolutional layers with input channels of {128, 64, 32} and output channels of {64, 32, 1} to obtain the final despeckling result map, where a ReLU activation layer is set between adjacent 1x1 convolutional layers.
[0156] The cropped patches are used as training samples in batches and sent to the constructed self-supervised speckle removal model. The output result graph is used to perform L1 loss function calculation, and the learnable parameters of the constructed self-supervised speckle removal space are optimized (the network parameters of the semantic encoder part of the GSE module are fine-tuned) until the loss function converges.
[0157] By inputting the image of the test set into the trained self-supervised speckle removal noise model, the corresponding speckle removal image result can be obtained.
[0158] Example 3
[0159] like Figure 6 As shown, a sonar image denoising system based on semantic guidance includes: a model building module, a feature extraction module, a feature processing module, a model training module and a result output module;
[0160] A model building module, used to build a speckle noise removal model including a first convolution layer, an RFA module, a GSE module and an output convolution module;
[0161] A feature extraction module is used to obtain a plurality of sonar images, and input the sonar images into a first convolutional layer to obtain a first extracted feature map; and input the first extracted feature map into an RFA module to obtain an enhanced feature map;
[0162] The feature processing module is used to input the enhanced feature map and the sonar image into the GSE module respectively to obtain a semantic fusion feature map; the semantic fusion feature map is input into the output convolution module to obtain a denoised sonar image;
[0163] A model training module is used to train a speckle noise removal model based on a denoised sonar image and a sonar image loss function to obtain a trained speckle noise removal model;
[0164] The result output module is used to input the sonar image to be processed into the trained speckle noise removal model to obtain the corresponding denoised sonar image.
[0165] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a sonar image denoising method and system based on semantic guidance, which has the following beneficial effects:
[0166] 1. Diamond-shaped Masked Convolution Layer (D-Conv), the weights within the range of Manhattan distance no more than 4 from the center weight of the convolution kernel are set to 0 and cannot be learned. When aggregating the information of pixels in the area around the pixel to be denoised, avoid using pixels with strong noise correlation with the noise of the pixel to be denoised, and effectively use pixels in the area of the pixel to be denoised with weak or no correlation with its noise, so as to overcome the spatial correlation of noise and enhance the utilization of local high-value information.
[0167] 2. Through the Receptive Field Augmentation (RFA) module, an enhanced receptive field is introduced for the pixels to be denoised, which not only avoids the use of pixels with strong spatial correlation with the noise of the pixels to be denoised, but also can use more pixels with weak spatial correlation with the noise of the pixels to be denoised. Therefore, the RFA module can overcome the spatial correlation of speckle noise while enhancing the utilization of the field pixel information within the receptive field.
[0168] 3. Through the Global Semantic Enhancement (GSE) module, the global semantic information of the entire image can be extracted to enhance the blind spot network's ability to perceive and utilize global information, thereby effectively alleviating the inherent information loss of the blind spot network while meeting the basic assumptions of the blind spot network.
[0169] 4. By combining the RFA module and the GSE module, the spatial correlation of speckle noise and the problem of reducing information loss are overcome at the same time. It can have the dual advantages of effective noise removal and high-quality texture restoration, and achieve high-quality sonar image despeckling effect.
[0170] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0171] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A sonar image denoising method based on semantic guidance, characterized in that: include: Construct a speckle noise removal model including the first convolution layer, the receptive field enhancement RFA module, the global semantic enhancement GSE module and the output convolution module; Acquire a plurality of sonar images, and input the sonar images into the first convolutional layer to obtain a first extracted feature map; Inputting the first extracted feature map into the receptive field enhancement RFA module to obtain an enhanced feature map; Inputting the enhanced feature map and the sonar image into the global semantic enhancement GSE module respectively to obtain a semantic fusion feature map; Inputting the semantic fusion feature map into the output convolution module to obtain a denoised image; The speckle noise removal model is trained based on the denoised sonar image and the sonar image loss function to obtain a trained speckle noise removal model; The sonar image to be processed is input into the trained speckle noise removal model to obtain a corresponding speckle noise removal sonar image.
2. The sonar image denoising method based on semantic guidance according to claim 1, characterized in that: The RFA module includes a first branch network and a second branch network in parallel; Inputting the first extracted feature map into the first branch network and the second branch network respectively, and obtaining a first local feature map and a second local feature map correspondingly; The first local feature map and the second local feature map are fused to obtain the enhanced feature map.
3. The sonar image denoising method based on semantic guidance according to claim 2, characterized in that: The first branch network includes: a D-Conv convolution layer, a first PD operation layer and a first F-Conv convolution layer; The first extracted feature map is input into the D-Conv convolution layer to obtain an aggregated feature map; The aggregated feature map is input to the first PD operation layer for spatial rearrangement to obtain a first rearranged feature map; The first rearranged feature map is input into the first F-Conv convolutional layer to obtain the first local feature map.
4. The sonar image denoising method based on semantic guidance according to claim 2, characterized in that: The second branch network includes: a Conv convolution layer, a second PD operation layer and a P-Conv convolution layer; The first extracted feature map is input into the Conv convolution layer to obtain a second extracted feature map; The second extracted feature map is input into the second PD operation layer for spatial rearrangement to obtain a second rearranged feature map; The second rearranged feature map is input into the P-Conv convolution layer to obtain the second local feature map.
5. The sonar image denoising method based on semantic guidance according to claim 3, characterized in that: The weights in the D-Conv convolution layer whose Manhattan distance from the center weight of the convolution kernel is no more than 4 are all set to 0 and cannot be learned; The weights are specifically set as: W D (i,j)=0,where d ij =|i-6|+|j-6|≤4,1≤i,j≤11 Among them, W D (i,) represents the weight of the D-Conv convolution kernel at the i-th row and j-th column position, d ij W represents the Manhattan distance between the weight at the i-th row and j-th column of the D-Conv convolution kernel and the center weight at the 6th row and 6th column of the convolution kernel center. D (i, j) = 0 means that the weight at the i-th row and j-th column of the D-Conv convolution kernel is set to 0 and cannot be learned.
6. The sonar image denoising method based on semantic guidance according to claim 3, characterized in that: The F-Conv convolution layer weights are specifically set to: W F (a,b)=0,where(a,b)∈{(1,2),(2,1),(2,3),(3,2)} Among them, W F (a, b) represents the weight of the F-Conv convolution kernel at the ath row and bth column, W F (a, b) = 0 means that the weight at the a-th row and b-th column of the F-Conv convolution kernel is set to 0 and cannot be learned.
7. The sonar image denoising method based on semantic guidance according to claim 4, characterized in that: The P-Conv convolution layer weights are specifically set to: W P (c,d)=0,where(c,d)∈{(1,1),(1,3),(2,2),(3,1),(3,3)} Among them, W P (c, d) represents the weight of the P-Conv convolution kernel at the cth row and dth column position, W P (c, d) = 0 means that the weight at the c-th row and d-th column of the P-Conv convolution kernel is set to 0 and cannot be learned.
8. The sonar image denoising method based on semantic guidance according to claim 1, characterized in that: The GSE module includes: a semantic encoder, a semantic injection module and a PD -1 Operational layer; The sonar image is input into the semantic encoder for feature extraction to obtain global semantic features; The global semantic feature and the enhanced feature map are respectively input into the semantic injection module to obtain a semantic enhanced feature map; The semantically enhanced feature map is input into the PD -1 The operation layer obtains the semantic fusion feature map.
9. The sonar image denoising method based on semantic guidance according to claim 8, characterized in that: The semantic injection module includes a plurality of semantic injection networks connected in sequence; The multiple semantic injection networks have the same structure, and all include: a semantic injection layer, a second F-Conv convolution layer, a ReLU activation layer, and a second convolution layer; The enhanced feature map and the global semantic feature are respectively input into the semantic injection layer to obtain a first enhanced feature; The first enhanced feature is sequentially processed by the second F-Conv convolution layer, the ReLU activation layer and the second convolution layer, and then fused with the enhanced feature map to obtain a first output feature map; The first output feature map and the global semantic feature are respectively input into the next semantic injection network for processing to obtain a second output feature map, and so on, through processing through all the semantic injection networks, finally obtaining the semantic enhancement feature.
10. A sonar image denoising system based on semantic guidance, characterized in that: include: Model building module, feature extraction module, feature processing module, model training module and result output module; The model building module is used to build a speckle noise removal model including a first convolution layer, a receptive field enhancement RFA module, a global semantic enhancement GSE module and an output convolution module; The feature extraction module is used to obtain a plurality of sonar images, and input the sonar images into the first convolution layer to obtain a first extracted feature map; and input the first extracted feature map into the receptive field enhancement RFA module to obtain an enhanced feature map; The feature processing module is used to input the enhanced feature map and the sonar image into the global semantic enhancement GSE module respectively to obtain a semantic fusion feature map; input the semantic fusion feature map into the output convolution module to obtain a denoised sonar image; The model training module is used to train the speckle noise removal model based on the denoised sonar image and sonar image loss function to obtain a trained speckle noise removal model; The result output module is used to input the sonar image to be processed into the trained speckle noise removal model to obtain the corresponding speckle noise removal sonar image.