Stained sea surface image restoration method based on space alignment generative adversarial network, electronic equipment and readable storage medium

Abstract

The invention belongs to the technical field of computer vision and image processing, and particularly discloses a stained sea surface image restoration method based on a spatial alignment generative adversarial network, electronic equipment and a readable storage medium, and the method comprises the steps: S1, inputting a sea surface image which comprises a to-be-restored stained sea surface image and a reference image; s2, multi-level feature maps of the stained sea surface image and the reference image are extracted through an encoder; s3, constructing a space alignment module based on deformable convolution, and performing space alignment on the feature map of the reference image and the feature map of the stained sea surface image; s4, constructing a space complementary attention module, and fusing the fouling feature map and the space alignment feature map; s5, generating a repaired image through a decoder; and S6, generating a network repair result by discriminating network optimization.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and specifically relates to a method for repairing a defaced sea surface image based on a spatial alignment generative adversarial network, an electronic device, and a readable storage medium. Background Art

[0002] During the long-term use of unmanned surface vessels, the lenses of environmental perception cameras are easily blocked by dust, mud and other stains, resulting in local information loss in the captured images, seriously affecting the subsequent visual algorithm's detection accuracy of sea surface obstacles, and posing a potential hidden danger to the navigation safety of the unmanned surface vessel; therefore, it is necessary to use real-time image restoration technology to automatically fill in the damaged areas to restore the true visual content of the image and lay the foundation for the implementation of subsequent visual tasks.

[0003] Existing image restoration methods mainly include traditional methods and deep learning-based methods. Among them, traditional methods are mainly based on the assumptions of image structural similarity and texture feature consistency, and fill the damaged areas by diffusing the boundaries of the damaged areas or extracting texture patches from known areas of the image; however, this type of method can only process some images with simple textures and small damaged areas; for large-area damaged images, the restoration results often have problems such as blurred content and missing semantics.

[0004] Compared with traditional methods, deep learning-based image restoration methods can link the underlying features of the image with high-level semantic information to infer the content of the missing area of the image, so its restoration results are often closer to the judgment of the human eye. However, existing deep learning methods still use the known content of the damaged image to infer and restore the content of the missing area. In many cases, its restoration results are inconsistent with the actual scene, making it unusable for subsequent tasks such as obstacle perception and recognition. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method for repairing a defaced sea surface image based on a spatial alignment generative adversarial network, an electronic device, and a readable storage medium.

[0006] Based on the above objectives, the present invention is achieved through the following technical solutions:

[0007] A first aspect of the present invention provides a method for restoring a defaced ocean surface image based on a spatially aligned generative adversarial network, comprising the following steps:

[0008] S1. Input a sea surface image, which includes a damaged sea surface image to be repaired and a reference image.

[0009] S2. Extract multi-level feature maps of the polluted sea surface image and the reference image through the encoder.

[0010] S3. Construct a spatial alignment module based on deformable convolution to spatially align the feature map of the reference image with the feature map of the polluted sea surface image.

[0011] S4. Construct a spatial complementary attention module to fuse the defacement feature map and the spatial alignment feature map.

[0012] S5. Generate the restored image through the decoder.

[0013] S6. Use the discriminant network to optimize the restoration results of the generated network.

[0014] According to the above-mentioned method for repairing a polluted sea surface image based on a spatial alignment generative adversarial network, preferably, in step S1, the polluted sea surface image is a polluted sea surface image I from a main perspective. Src ; The reference image is a reference image I of an adjacent viewing angle Ref The defaced sea surface image and the reference image are collected simultaneously using a multi-eye vision system to ensure time synchronization.

[0015] According to the above-mentioned method for repairing defaced sea surface images based on spatially aligned generative adversarial networks, preferably, in step S2, the encoder is composed of a stack of multiple convolutional layers and conventional residual modules, and the output of each convolutional layer is nonlinearized by a LeakyReLU function; wherein the conventional residual module connects the input and the output of the nonlinear layer through an identity mapping, which is used to avoid the degradation problem of deep neural networks and enable the network to converge to the local minimum faster.

[0016] The encoder extracts multi-level feature maps in three stages, wherein the L1 stage extracts shallow features through a 3×3 convolution layer with a step size of 1 and four consecutive conventional residual modules; the L2 stage extracts mid-level features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the L3 stage extracts deep features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the encoder extracts features from the input image by sharing weights, and finally obtains multi-level feature maps similar to pyramid structures for the defaced sea surface image and the reference image. and

[0017] According to the above-mentioned method for restoring a defaced sea surface image based on a spatial alignment generative adversarial network, preferably, in step S3, the step of the spatial alignment module spatially aligning the feature map of the reference image with the feature map of the defaced sea surface image comprises:

[0018] S31, stitching along the channel direction: Assume that the contaminated sea surface image I Src With the reference view image I RefThe feature maps at the nth stage of the encoder are and Will and Splicing along the channel direction is used to combine the feature map of the reference view Feature map of the polluted sea surface image Maintain spatial alignment.

[0019] S32, through a 1×1 convolution layer to reduce its number of channels to C, and then use the PReLU activation function for nonlinear processing to obtain the intermediate feature map

[0020] S33, then use the deformable convolution layer of the spatial alignment module to perform spatial transformation on the feature map of the reference image to generate a feature map after the reference image and the polluted sea surface image are aligned

[0021] According to the above-mentioned method for repairing a defaced sea surface image based on a spatial alignment generative adversarial network, preferably, in step S3, a 3×3 deformable convolution layer is used to extract the defaced sea surface image from the intermediate feature map. We learn a parameter tensor T with a size of 27×W×H and a 27-dimensional channel number. n ; Since the 3×3 deformable convolution kernel has 9 sampling positions, and each sampling position needs to learn an offset vector Δp k =[Δx,Δy] T With a modulation weight Δm k Therefore, when the spatial resolution of the input and output feature maps of the deformable convolution remains unchanged, it is necessary to pre-learn a parameter tensor T with 27-dimensional channels. n ; Among them, the first 18 dimensions are the offsets of the 9 sampling positions of the convolution kernel in the horizontal and vertical directions {Δp k} k＝1:9 , and the last 9 dimensions are the modulation weights {Δm k} k＝1:9 ; Finally, using the parameter tensor T of the deformable convolution, the offset vector Δp k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference view feature map Feature map of polluted sea surface image Align spatially to generate aligned feature maps for:

[0022]

[0023] Among them, each sampling position is denoted as p k∈R, the sampling grid R={(-1,-1),(-1,0),...,(0,1),(1,1)}, and each position on the output feature map is denoted as p0.

[0024] Offset vector Δp k The deformable convolution has the ability to adaptively adjust the convolution kernel sampling position; but considering the offset vector Δp learned through the convolution operation k It may be a fraction, so formula (2) also needs to be implemented through bilinear interpolation:

[0025]

[0026] Among them, p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the y-axis; G(·,·) represents the bilinear interpolation kernel. Since it is two-dimensional, it can be decomposed into two one-dimensional kernels:

[0027]

[0028] Among them, g(a,b)=max(0,1-|ab|); Since only a small part of q is non-zero in formula (3), the corresponding result can be solved quickly; finally, the corresponding spatial alignment feature maps are generated at different stages of the encoder.

[0029] According to the above-mentioned method for repairing the defaced sea surface image based on the spatial alignment generative adversarial network, preferably, in theory, at the nth stage of the encoder, the feature map after spatial alignment Feature map of the polluted sea surface image There is a complementary characteristic when repairing images; therefore, based on this characteristic, a spatial complementary attention module is constructed to improve the network's repair accuracy for damaged areas; the specific steps of S4 include:

[0030] S41. Feature map after spatial alignment and feature maps of polluted sea surface images Splice along the channel direction.

[0031] S42, sequentially pass through the 1×1 convolution layer and the PReLU activation function layer to obtain the intermediate feature map

[0032] S43, the intermediate feature map Send it to another 1×1 convolution layer and Sigmoid function layer to get the spatial complementary attention coefficient tensor

[0033] S44, the attention coefficient matrix of different channels is respectively compared with the feature map of the polluted sea surface image and spatially aligned feature maps Perform element-wise multiplication and then add element-by-element to obtain the spatial complementary attention feature map

[0034] According to the above-mentioned method for repairing the defaced sea surface image based on the spatial alignment generative adversarial network, preferably, let C and M be the number of channels and the number of feature vectors of the feature map, and M = H × W; wherein H and W represent the height and width of the feature map, respectively; the defaced sea surface image feature map of the nth stage of the generative network is generated. Aligning feature maps with space Rewrite as and Then the spatial complementary attention feature map The eigenvector at any position on Defined as:

[0035]

[0036] in, represents the element at the mth position and the i-th dimension channel on the attention coefficient tensor of the nth stage of the encoder, and satisfies α n,m for:

[0037]

[0038] in, and They are the feature maps of polluted sea surface images and spatially aligned feature maps The spatial position index on is the eigenvector of m; U(·,·) represents the channel splicing operation;

[0039] and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers; δ is the PReLU activation function.

[0040] According to the above-mentioned method for restoring a defaced sea surface image based on a spatially aligned generative adversarial network, preferably, in step S5, the decoder includes two semantic flow alignment upsampling modules, which learn the semantic flow between feature maps of adjacent levels in the network in a manner similar to constructing optical flow between adjacent frames, so as to eliminate alignment errors when fusing features of different levels; the step of generating a restored image by the decoder includes:

[0041] S51, given two adjacent layers of spatial complementary attention feature maps and Two 1×1 convolutional layers are used to ensure that the number of channels of high-level and shallow-level feature maps remains consistent.

[0042] S52, through a bilinear interpolation upsampling layer to the high-level feature map of spatial complementary attention The resolution of the shallow feature map is adjusted to consistent.

[0043] S53, the upsampled feature map and the shallow feature map Splice along the channel direction and predict a semantic flow field through a 3×3 convolution layer

[0044] S54. High-level feature map of spatial complementary attention based on semantic flow field Warp is performed to align the space with the shallow feature map, and the repaired image is generated through a 3×3 convolution layer and a PReLU activation function layer.

[0045] The decoder uses a spatial grid to establish a coordinate mapping relationship between cross-level features. The spatial grid is used to describe the spatial position of image pixels and to establish a coordinate mapping relationship between cross-level features. Through deformation, distortion and interpolation operations guided by semantic flow, spatial alignment of feature maps of different resolutions is achieved, ultimately improving the accuracy of image restoration. For the spatial grid Ω n Each pixel p in n , first according to the spatial correspondence (p n +Δ n (p n )) / 2 maps it to the high-level n+1, and then performs bilinear interpolation according to the mapped coordinates to obtain a high-resolution feature map. The formula is as follows:

[0046]

[0047] Among them, w is the weight of bilinear interpolation; N(p n+1 ) represents the spatial complementary attention feature map p n+1 The neighborhood position of p n Represents the spatial grid Ω n Each pixel in Δ n (p n ) represents the offset of the pixel point in the spatial correspondence; p n+1 Indicates the pixel mapped to the high-level n+1; Represents pixel p n The spatial complementary attention feature map after mapping to the high level is abbreviated as

[0048] When spatial complementary attention is applied to low-resolution feature maps After semantic stream upsampling and high-resolution feature map After the features of Element-by-element addition is performed to fuse context information at different levels, and a feature map with rich semantic information and detail information is obtained by sequentially passing through a 3×3 convolutional layer and a PReLU activation function layer.

[0049] According to the above-mentioned method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network, preferably, in step S6, the discriminant network is used to distinguish between the repaired image generated by the generative network and the true image, thereby optimizing the repair result of the generative network through adversarial training; specifically, the discriminant network is composed of 5 convolutional layers, the first 4 convolutional layers are used to extract features from the input image, and the last convolutional layer is used to output a one-dimensional result; wherein, except for the last convolutional layer, the outputs of the remaining convolutional layers are connected to a batch normalization layer and a LeakyReLU activation layer by default; the input of the discriminant network is the repaired image I generated by the generative network Pred Or the true value image I GT , the output is a scalar value, which represents the probability that the input image is the true value image.

[0050] The loss function is used to train the spatial alignment generative adversarial network. The loss function includes the feature alignment loss function L align , reconstruction loss function L rec And the adversarial loss function L adv ,Right now:

[0051]

[0052] Where L represents the loss function of the method; λ1, λ2 and λ3 are the weights of each loss function, λ1 is set to 0.5, λ2 is set to 0.99, and λ3 is set to 0.01; L align Used to strengthen the learning of spatial alignment feature maps; L rec Calculate the network generated image through L1 loss and the true value image I GT The distance between them is used to generate the output of the network; L adv It is used to train the generative network G and the discriminative network D against each other. The specific definition is:

[0053]

[0054] Among them, L1(·,·) is the L1 loss function; G represents the generation network, D represents the discrimination network, and I Srcrepresents the polluted sea surface image, I GT Represents the true value image.

[0055] A second aspect of the present invention provides an electronic device comprising a memory and a processor, wherein a computer program is stored on the memory, and when the processor executes the computer program, any step of the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in the first aspect is implemented.

[0056] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer processor, the computer program implements any step of the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in the first aspect.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention, on the one hand, introduces reference image information from adjacent viewpoints into different stages of a generative adversarial network to ensure that the restored content is consistent with the real scene. It also utilizes a spatially complementary attention module to strengthen the network's learning of the semantic structure of damaged areas, enabling the multi-view system to restore the visual content of defective areas with high quality when the lens is contaminated. This enhances the robustness of the restoration and accurately repairs damaged areas in the image. Furthermore, the invention uses a semantic stream alignment upsampling module to fuse feature maps from different stages, thereby improving the network's ability to represent multi-scale objects in the image. This invention can learn missing visual content from reference images, ensuring that the restoration results are highly consistent with the visual content of the real scene.

[0059] 2. The present invention assists and constrains the repair process of damaged images by introducing image information from adjacent perspectives into a generative adversarial network to ensure that the visual content of the defective area is restored truthfully and accurately. Taking into account the problem that it is impossible to simultaneously obtain a damaged sea surface image and the corresponding true image during actual sea surface image acquisition, the present invention jointly trains the proposed image repair network with artificially synthesized damaged sea surface images and their true images. This can not only accurately repair the damaged areas in the image, but also ensure that the repaired content is consistent with the real scene by introducing image information from adjacent perspectives, with high real-time performance and accuracy.

[0060] 3. The present invention fully utilizes the characteristic of deformable convolution to change the theoretical receptive field through the spatial alignment module. The convolution kernel can adaptively adjust the sampling position through the learnable offset vector, which can ensure the spatial alignment accuracy of the reference perspective image and the main perspective defaced sea surface image. In this way, the image information of the reference perspective is introduced into the restoration network to assist the network in repairing the defaced area, which can improve the quality of image restoration and better restore the visual content of the defective area, thereby retaining more texture details and making the image quality clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0062] Figure 2 Schematic diagram of the structure of a conventional residual module in Example 1 of the present invention;

[0063] Figure 3 Schematic diagram of the structure of the spatial alignment module in Example 1 of the present invention;

[0064] Figure 4 Schematic diagram of the structure of the spatial complementary attention module in Example 1 of the present invention;

[0065] Figure 5 Schematic diagram of the structure of the semantic stream alignment upsampling module in Example 1 of the present invention;

[0066] Figure 6 Schematic diagram of a comparative example of restoration using different models on a synthetic damaged sea surface image in Example 1 of the present invention;

[0067] Figure 7 Schematic diagram of the visualization result of the feature map in the image restoration model in Example 1 of the present invention;

[0068] Figure 8 Schematic diagram of a restoration example of the present invention on a real defaced sea surface image in Example 1 of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below through specific examples, but the scope of the present invention is not limited thereto.

[0070] Example 1

[0071] A method for repairing damaged sea surface images based on spatial alignment generative adversarial networks, the overall framework of which is as follows Figure 1 As shown, the following steps are included:

[0072] S1. Input a sea surface image, which includes a damaged sea surface image to be repaired and a reference image.

[0073] The polluted sea surface image is a polluted sea surface image I from a main perspective.Src ; The reference image is a reference image I of an adjacent viewing angle Ref The defaced sea surface image and the reference image are collected simultaneously using a multi-eye vision system to ensure time synchronization.

[0074] S2. Extract multi-level feature maps of the polluted sea surface image and the reference image through the encoder.

[0075] The encoder is composed of a stack of multiple convolutional layers and conventional residual modules, and the output of each convolutional layer is nonlinearized by the LeakyReLU function; wherein, the conventional residual module connects the input and the output of the nonlinear layer through an identity mapping to avoid the degradation problem of the deep neural network and enable the network to converge to the local minimum faster. Its specific structure is as follows Figure 2 shown.

[0076] The encoder extracts multi-level feature maps in three stages, wherein the L1 stage extracts shallow features through a 3×3 convolution layer with a step size of 1 and four consecutive conventional residual modules; the L2 stage extracts mid-level features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the L3 stage extracts deep features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the encoder extracts features from the input image by sharing weights, and finally obtains multi-level feature maps similar to pyramid structures for the defaced sea surface image and the reference image. and

[0077] S3. Construct a spatial alignment module based on deformable convolution to spatially align the feature map of the reference image with the feature map of the polluted sea surface image.

[0078] The structure of the spatial alignment module is as follows Figure 3 As shown, the step of spatially aligning the feature map of the reference image with the feature map of the polluted sea surface image by the spatial alignment module includes:

[0079] S31, stitching along the channel direction: Assume that the contaminated sea surface image I Src With the reference view image I Ref The feature maps at the nth stage of the encoder are and Will and Splicing along the channel direction is used to combine the feature map of the reference view Feature map of the polluted sea surface image Maintain spatial alignment.

[0080] S32, through a 1×1 convolution layer to reduce its number of channels to C, and then use the PReLU activation function for nonlinear processing to obtain the intermediate feature map

[0081] S33, then use the deformable convolution layer of the spatial alignment module to perform spatial transformation on the feature map of the reference image to generate a feature map after the reference image and the polluted sea surface image are aligned

[0082] A 3×3 deformable convolution layer is used to extract the intermediate feature map We learn a parameter tensor T with a size of 27×W×H and a 27-dimensional channel number. n ; Since the 3×3 deformable convolution kernel has 9 sampling positions, and each sampling position needs to learn an offset vector Δp k =[Δx,Δy] T With a modulation weight Δm k Therefore, when the spatial resolution of the input and output feature maps of the deformable convolution remains unchanged, it is necessary to pre-learn a parameter tensor T with 27-dimensional channels. n ; Among them, the first 18 dimensions are the offsets of the 9 sampling positions of the convolution kernel in the horizontal and vertical directions {Δp k} k＝1:9 , and the last 9 dimensions are the modulation weights {Δm k} k＝1:9 ; Finally, using the parameter tensor T of the deformable convolution, the offset vector Δp k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference view feature map Feature map of polluted sea surface image Align spatially to generate aligned feature maps for:

[0083]

[0084] Among them, each sampling position is denoted as p k ∈R, the sampling grid R={(-1,-1),(-1,0),...,(0,1),(1,1)}, and each position on the output feature map is denoted as p0.

[0085] Offset vector Δp k The deformable convolution has the ability to adaptively adjust the convolution kernel sampling position; but considering the offset vector Δp learned through the convolution operation k It may be a fraction, so formula (2) also needs to be implemented through bilinear interpolation:

[0086]

[0087] Among them, p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the y-axis; G(·,·) represents the bilinear interpolation kernel. Since it is two-dimensional, it can be decomposed into two one-dimensional kernels:

[0088]

[0089] Among them, g(a,b)=max(0,1-|ab|); Since only a small part of q is non-zero in formula (3), the corresponding result can be solved quickly; finally, the corresponding spatial alignment feature maps are generated at different stages of the encoder.

[0090] S4. Construct a spatial complementary attention module to fuse the defacement feature map and the spatial alignment feature map.

[0091] The structure of the spatial complementary attention module is as follows Figure 4 As shown; theoretically, at the nth stage of the encoder, the feature map after spatial alignment Feature map of the polluted sea surface image There is a complementary characteristic when repairing images; therefore, based on this characteristic, a spatial complementary attention module is constructed to improve the network's repair accuracy for damaged areas; the specific steps of S4 include:

[0092] S41. Feature map after spatial alignment and feature maps of polluted sea surface images Splice along the channel direction.

[0093] S42, sequentially pass through the 1×1 convolution layer and the PReLU activation function layer to obtain the intermediate feature map

[0094] S43, the intermediate feature map Send it to another 1×1 convolution layer and Sigmoid function layer to get the spatial complementary attention coefficient tensor

[0095] S44, the attention coefficient matrix of different channels is respectively compared with the feature map of the polluted sea surface image and spatially aligned feature maps Perform element-wise multiplication and then add element-by-element to obtain the spatial complementary attention feature map

[0096] Let C and M be the number of channels and feature vectors of the feature map, and M = H × W; where H and W represent the height and width of the feature map, respectively; the feature map of the polluted sea surface image at the nth stage of the network is generated Aligning feature maps with space Rewrite as and Then the spatial complementary attention feature map The eigenvector at any position on Defined as:

[0097]

[0098] in, represents the element at the mth position and the i-th dimension channel on the attention coefficient tensor of the nth stage of the encoder, and satisfies α n,m for:

[0099]

[0100] in, and They are the feature maps of polluted sea surface images and spatially aligned feature maps The spatial position index on is the eigenvector of m; ∪(·,·) represents the channel splicing operation;

[0101] and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers; δ is the PReLU activation function.

[0102] S5. Generate the restored image through the decoder.

[0103] The decoder includes two semantic stream alignment upsampling modules, such as Figure 5 As shown in FIG, this module learns the semantic flow between feature maps of adjacent levels in the network in a manner similar to constructing optical flow between adjacent frames, so as to eliminate the alignment error when fusing features of different levels; the steps of generating the restored image by the decoder include:

[0104] S51, given two adjacent layers of spatial complementary attention feature maps and Two 1×1 convolutional layers are used to ensure that the number of channels of high-level and shallow-level feature maps remains consistent.

[0105] S52, through a bilinear interpolation upsampling layer to the high-level feature map of spatial complementary attention The resolution of the shallow feature map is adjusted to consistent.

[0106] S53, the upsampled feature map and the shallow feature map Splice along the channel direction and predict a semantic flow field through a 3×3 convolution layer

[0107] S54. High-level feature map of spatial complementary attention based on semantic flow field Warp is performed to align the space with the shallow feature map, and the repaired image is generated through a 3×3 convolution layer and a PReLU activation function layer.

[0108] The decoder uses a spatial grid to establish a coordinate mapping relationship between cross-level features. n Each pixel p in n , first according to the spatial correspondence (p n +Δ n (p n )) / 2 maps it to the high-level n+1, and then performs bilinear interpolation according to the mapped coordinates to obtain a high-resolution feature map. The formula is as follows:

[0109]

[0110] Among them, w is the weight of bilinear interpolation; N(p n+1 ) represents the spatial complementary attention feature map p n+1 The neighborhood position of p n Represents the spatial grid Ω n Each pixel in Δ n (p n ) represents the offset of the pixel point in the spatial correspondence; p n+1 Indicates the pixel mapped to the high-level n+1; Represents pixel p n The spatial complementary attention feature map after mapping to the high level is abbreviated as

[0111] When spatial complementary attention is applied to low-resolution feature maps After semantic stream upsampling and high-resolution feature map After the features of Element-by-element addition is performed to fuse context information at different levels, and a feature map with rich semantic information and detail information is obtained by sequentially passing through a 3×3 convolutional layer and a PReLU activation function layer.

[0112] S6. Use the discriminant network to optimize the generated network's repair results.

[0113] The discriminant network is used to distinguish the inpainted image generated by the generative network from the true image, thereby optimizing the inpainting result of the generative network through adversarial training. Specifically, the discriminant network consists of 5 convolutional layers, the first 4 convolutional layers are used to extract features from the input image, and the last convolutional layer is used to output a one-dimensional result. Except for the last convolutional layer, the outputs of the remaining convolutional layers are connected to a batch normalization layer and a LeakyReLU activation layer by default. The input of the discriminant network is the inpainted image I generated by the generative network. Pred Or the true value image I GT , the output is a scalar value, which represents the probability that the input image is the true value image.

[0114] The loss function is used to train the spatial alignment generative adversarial network. The loss function includes the feature alignment loss function L align , reconstruction loss function L rec And the adversarial loss function L adv ,Right now:

[0115] L=λ1L align +λ2L rec +λ3L adv (8)

[0116] Where L represents the loss function of the method; λ1, λ2 and λ3 are the weights of each loss function, λ1 is set to 0.5, λ2 is set to 0.99, and λ3 is set to 0.01; L align Used to strengthen the learning of spatial alignment feature maps; L rec Calculate the network generated image through L1 loss and the true value image I GT The distance between them is used to generate the output of the network; L adv It is used to train the generative network G and the discriminative network D against each other. The specific definition is:

[0117]

[0118] Among them, L1(·,·) is the L1 loss function; G represents the generation network, D represents the discrimination network, and I Src represents the polluted sea surface image, I GT Represents the true value image.

[0119] This example is based on the Ubuntu 16.04 operating system, using the Python programming language and the PyTorch deep learning framework. The experimental hardware environment includes two NVIDIA GTX1080Ti GPUs and an Intel Core i7-6700K CPU. The specific implementation steps are as follows:

[0120] (1) Experimental data preparation;

[0121] In this embodiment, a binocular sea scene dataset containing 5000 pairs of binocular images with a resolution of 512×384 is collected by a homemade unmanned surface vessel; 2500 pairs are randomly selected as the training set, 1000 pairs as the validation set, and 1500 pairs as the test set, covering typical near-shore and port scenes under different lighting conditions; when training the image restoration network, a pair of synchronized binocular images is randomly selected, and a random irregular gray mask is applied to a random position of the left-view image. This mask, together with the right-view reference image, is used as the input of the image restoration network; the single-input image restoration network is trained using the left-view defective image and its corresponding true value image.

[0122] (2) Model building and training;

[0123] ① Adjust the input image resolution to 256×256;

[0124] ②Use batch training method to randomly divide the training set into several batches, each batch size is 4.

[0125] ③. Adaptive Moment Estimation (Adam) optimizer with a momentum factor of 0.5 is used to update the network parameters so that the loss function converges quickly to the local optimal solution.

[0126] ④. Using the formula L = λ1L align +λ2L rec +λ3L adv The network prediction results were evaluated using the loss function (λ1, λ2, and λ3 are 0.5, 0.99, and 0.01, respectively), and the initial learning rate was set to 2×10 -4 , and the initial learning rate remains unchanged for the first 100 training epochs, and the Poly learning strategy with an exponent of 2 is used to update the learning rate for the next 100 epochs until the training stops.

[0127] (3) Selection of performance evaluation indicators;

[0128] This example uses the quantitative performance evaluation indicators commonly used in image restoration:

[0129] ① Mean Absolute Error (MAE): The expected absolute difference between the pixel values of the predicted image and the true image, reflecting the average deviation between data points. The smaller the value, the more similar the restored image is to the true image. It can be expressed as:

[0130]

[0131] Where M is the number of image pixels, and i represents the i-th pixel position of the image.

[0132] ② Mean Squared Error (MSE): The expected value of the square of the pixel difference between the restored image and the true image, reflecting the degree of data dispersion. The smaller the value, the better the image restoration quality. It is expressed as:

[0133]

[0134] ③. Peak Signal to Noise Ratio (PSNR): This represents the ratio of the maximum possible signal power to the destructive noise power that affects accuracy. It is used to evaluate image quality. The larger the value, the closer the restored image is visually to the true image. It is expressed in dB as follows:

[0135]

[0136] Among them, MAX I is the maximum possible pixel value of the image, and n is the number of bits of the binary value of the image pixel.

[0137] ④ Structural Similarity Index Measure (SSIM): measures the similarity between two images. The larger the value, the more similar the restored image is to the true image, and the better the image restoration quality is. It can be expressed as:

[0138]

[0139] in, for The average value, μ IGT For I GT The average value of for The variance, σ IGT For I GT The variance of For I GT and I GT The covariance of C1=[K1(2 n -1)] and C2=[K2(2 n -1)] is a constant used to maintain stability, where n is the number of bits of the image pixel binary value, and the values of K1 and K2 are 0.01 and 0.03 respectively.

[0140] ⑤. Average operation time: evaluate the model execution efficiency and time complexity.

[0141] (4) Selection of control model;

[0142] Representative image restoration networks such as DFNet, PConv, RN, and STN-GAN are selected as controls; among them, STN-GAN and the model of this embodiment need to input the damaged sea surface image and the adjacent perspective reference image at the same time, while the other networks only need to input the damaged sea surface image; all networks are trained and tested on the synthetic damaged images, and the learning strategy in step (2) is used to optimize the model parameters to evaluate the restoration performance of this model.

[0143] (5) Comparison and analysis of experimental results;

[0144] ① Quantitative Evaluation: Experiments were conducted on a synthetic damaged image dataset. The results are shown in Table 1. This embodiment achieves the best performance in terms of MAE, MSE, PSNR, and SSIM. The PSNR is 5.893 dB higher than the second-place STN-GAN, demonstrating its effectiveness in the task of repairing defaced sea surface images.

[0145] Table 1: Quantitative evaluation results of different models in the synthetic damaged sea surface image restoration experiment;

[0146]

[0147] ② Qualitative evaluation: The restoration results of different models on the synthetic damaged sea surface images are as follows: Figure 6 As shown in the figure, there are artifacts and over-smoothing problems in the connection between the defective area repaired by DFNet, PConv, and RN models and the non-defective area, and the visual content is inconsistent with the actual scene; STN-GAN and this embodiment introduce reference images, and the image quality is better, but the global spatial transformation network used by STN-GAN has alignment errors, which leads to spatial dislocation and visual blur in the repaired area. The spatial alignment module of this embodiment makes full use of the characteristic of deformable convolution to change the theoretical receptive field, and enables the convolution kernel to adaptively adjust the sampling position through the learnable offset vector, thereby ensuring the spatial alignment accuracy of the reference perspective image and the main perspective defaced sea surface image. In this way, the image information of the reference perspective is introduced into the repair network to assist the network in repairing the defaced area, thereby improving the quality of image repair, better restoring the visual content of the defective area, retaining more texture details and making the picture quality clearer.

[0148] ③. Feature map visualization: The feature map visualization results in the image restoration model of this embodiment are as follows Figure 7 As shown, the feature map after spatial alignment Relative to the original reference image I Ref The spatial structure is obviously offset, and feature alignment is achieved with the area to be repaired from the main perspective. The repair model of this embodiment can learn the missing visual content from the reference image, ensuring that the repair result is highly consistent with the visual content of the real scene.

[0149] ④. Computational time evaluation: The average computational time of different models in repairing synthetic damaged sea surface images is shown in Table 2. Among them, DFNet has the fastest inference speed (6ms) and STN-GAN has the slowest (45ms). The method in this embodiment has a longer computational time than DFNet due to processing two input images, but the actual inference speed reaches 34 frames per second, which is faster than the frame rate of the unmanned surface vessel perception camera (25 frames per second), and therefore does not slow down the operating efficiency of the visual system.

[0150] Table 2 shows the average operation time of different models in the synthetic damaged sea surface image restoration experiment;

[0151]

[0152] ⑤ Qualitative evaluation of ground truth images: Considering that it is impossible to obtain the real defaced sea surface images and their corresponding ground truth images required for quantitative evaluation in the real world, we only qualitatively evaluate the restoration performance on the real defaced sea surface images. The results are as follows: Figure 8 As shown in the comparison between the second row and the last row in the figure, the damaged area repaired by this embodiment maintains a high degree of consistency with the corresponding reference perspective image in terms of visual content and texture details, ensuring that subsequent visual tasks are smoothly carried out on the repaired image.

[0153] Example 2

[0154] An electronic device includes a memory and a processor, wherein a computer program is stored on the memory, and when the processor executes the computer program, any step of the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in Example 1 is implemented.

[0155] Furthermore, the method and process for restoring defaced ocean surface images based on a spatially aligned generative adversarial network described in Example 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program borne on a computer-readable medium, the computer program containing program code for executing the method. In such an embodiment, the computer program can be downloaded and installed from a network and / or installed from removable media. When executed by a processor, the computer program performs the aforementioned functions defined in the method of this application.

[0156] Example 3

[0157] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of a method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in Example 1.

[0158] The computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0159] The computer program code for performing the operations of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as python, C++, and also conventional procedural programming languages or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0160] The computer-readable storage medium of this embodiment can be accelerated by hardware such as a GPU, and the parallel computing advantages of the GPU can be used to accelerate the processing of any step in the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in Example 1.

[0161] In summary, the present invention effectively overcomes the deficiencies in the prior art and has a high industrial application value. The above embodiments serve to illustrate the substantial content of the present invention, but are not intended to limit the scope of protection of the present invention. Those skilled in the art will appreciate that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of protection of the technical solutions of the present invention.

[0162] The above embodiments are specific implementation methods of the present invention, but the implementation methods of the present invention are not limited to the above embodiments. Any other combination, change, modification, substitution, and simplification that does not exceed the design concept of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network, characterized in that: The following steps are involved: S1. Inputting a sea surface image, wherein the sea surface image includes a damaged sea surface image to be repaired and a reference image; S2, extracting multi-level feature maps of the polluted sea surface image and the reference image through an encoder; S3. Construct a spatial alignment module based on deformable convolution to spatially align the feature map of the reference image with the feature map of the polluted sea surface image; S4, build a spatial complementary attention module to fuse the defacement feature map and the spatial alignment feature map; S5, generating a restored image through a decoder; S6. Use the discriminant network to optimize the generated network's repair results.

2. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 1 is characterized in that: In step S2, the encoder extracts multi-level feature maps in three stages, wherein the L1 stage extracts shallow features through a 3×3 convolution layer with a step size of 1 and four consecutive conventional residual modules; the L2 stage extracts middle features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the L3 stage extracts deep features through a 3×3 convolution layer with a step size of 2 and a 3×3 convolution layer with a step size of 1; the encoder extracts features from the input image by sharing weights, and finally obtains multi-level feature maps of the defaced sea surface image and the reference image respectively. and 3. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 1, characterized in that: In step S3, the steps of performing spatial alignment by the spatial alignment module include: S31, stitching along the channel direction: Assume that the contaminated sea surface image I Src With the reference view image I Ref The feature maps at the nth stage of the encoder are and Will and Splicing along the channel direction; S32, through a 1×1 convolution layer to reduce its number of channels to C, and then use the PReLU activation function for nonlinear processing to obtain the intermediate feature map S33, then use the deformable convolution layer of the spatial alignment module to perform spatial transformation on the feature map of the reference image to generate the aligned feature map 4. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 3 is characterized in that: In step S3, a 3×3 deformable convolution layer is used to extract the intermediate feature map We learn a parameter tensor T with a size of 27×W×H and a 27-dimensional channel number. n ; Among them, the first 18 dimensions are the offsets of the 9 sampling positions of the convolution kernel in the horizontal and vertical directions {ΔP k } k＝1:9 , and the last 9 dimensions are the modulation weights {Δm k } k＝1:9 ; Finally, using the parameter tensor T of the deformable convolution, the offset vector ΔP k and modulation weight Δm k Adaptively adjust the sampling position of the convolution kernel to ensure the reference view feature map Feature map of polluted sea surface image Align spatially to generate aligned feature maps for: Among them, each sampling position is recorded as P k ∈R, sampling grid R = {(-1,-1), (-1,0), ..., (0,1), (1,1)}, each position on the output feature map is denoted as P0; This is achieved through bilinear interpolation: Among them, p represents the feature map Any of the above sampling positions may be a fraction, and p=p0+p k +Δp k ;q represents the feature map Any integer spatial position on the graph; G(·,·) represents the bilinear interpolation kernel, which is decomposed into two one-dimensional kernels: G(q,p)=g(q x ,p x )·g(q y ,p y ) (3) Among them, g(a,b)=max(0,1-|ab|); Finally, the corresponding spatial alignment feature maps are generated at different stages of the encoder 5. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 1, characterized in that: The specific steps of S4 include: S41. Feature map after spatial alignment and feature maps of polluted sea surface images Splicing along the channel direction; S42, sequentially pass through the 1×1 convolution layer and the PReLU activation function layer to obtain the intermediate feature map S43, the intermediate feature map Send it to another 1×1 convolution layer and Sigmoid function layer to get the spatial complementary attention coefficient tensor S44, the attention coefficient matrix of different channels is respectively compared with the feature map of the polluted sea surface image and spatially aligned feature maps Perform element-wise multiplication and then add element-by-element to obtain the spatial complementary attention feature map 6. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 5, characterized in that: Let C and M be the number of channels and feature vectors of the feature map, respectively, and M = H × W; Among them, H and W represent the height and width of the feature map respectively; the feature map of the polluted sea surface image at the nth stage of the network is generated Aligning feature maps with space Rewrite as and Then the spatial complementary attention feature map The eigenvector at any position on Defined as: in, represents the element at the mth position and the i-th dimension channel on the attention coefficient tensor of the nth stage of the encoder, and satisfies α n,m for: in, and They are the feature maps of polluted sea surface images and spatially aligned feature maps The spatial position index on is the eigenvector of m; ∪(·,·) represents the channel splicing operation; and and are the linear transformation parameters and bias parameters of the two 1×1 convolutional layers; δ is the PReLU activation function.

7. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 1, characterized in that: In step S5, the decoder includes two semantic stream alignment upsampling modules for eliminating alignment errors when fusing features at different levels. The decoder generates a restored image by: S51, given two adjacent layers of spatial complementary attention feature maps and Use two 1×1 convolutional layers to ensure that the number of channels of high-level and shallow feature maps is consistent; S52, through a bilinear interpolation upsampling layer to the high-level feature map of spatial complementary attention The resolution of the shallow feature map is adjusted to consistent; S53, the upsampled feature map and the shallow feature map Splice along the channel direction and predict a semantic flow field through a 3×3 convolution layer S54. High-level feature map of spatial complementary attention based on semantic flow field Perform deformation and distortion to align the space with the shallow feature map, and generate the repaired image through a 3×3 convolution layer and a PReLU activation function layer; The decoder uses a spatial grid to establish a coordinate mapping relationship between cross-level features. n Each pixel p in n , first according to the spatial correspondence (p n +Δ n (p n )) / 2 maps it to the high-level n+1, and then performs bilinear interpolation according to the mapped coordinates to obtain a high-resolution feature map. The formula is as follows: Among them, w is the weight of bilinear interpolation; N(p n+1 ) represents the spatial complementary attention feature map p n+1 The neighborhood position of Δ n (p n ) represents pixel p n The offset in the spatial correspondence; p n+1 Indicates the pixel mapped to the high-level n+1; Represents pixel p n The spatial complementary attention feature map after mapping to the high level is abbreviated as When spatial complementary attention is applied to low-resolution feature maps After semantic stream upsampling and high-resolution feature map After the features of Element-by-element addition is performed to fuse context information at different levels, and a feature map with rich semantic information and detail information is obtained by sequentially passing through a 3×3 convolutional layer and a PReLU activation function layer.

8. The method for restoring a damaged sea surface image based on a spatially aligned generative adversarial network according to claim 1, characterized in that: In step S6, the discriminant network is used to distinguish the restored image generated by the generative network from the true image, thereby optimizing the restoration result of the generative network through adversarial training; the input of the discriminant network is the restored image I generated by the generative network. Pred Or the true value image I GT , the output is a scalar value, indicating the probability that the input image is the true value image; The loss function is used to train the spatial alignment generative adversarial network. The loss function includes the feature alignment loss function L align , reconstruction loss function L rec And the adversarial loss function L adv ,Right now: L=λ1L align +λ2L rec +λ3L adv (8) Where L represents the loss function of the method; λ1, λ2 and λ3 are the weights of each loss function, λ1 is set to 0.5, λ2 is set to 0.99, and λ3 is set to 0.01; L align Used to strengthen the learning of spatial alignment feature maps; L rec Calculate the network generated image through L1 loss and the true value image I GT The distance between them is used to generate the output of the network; L adv It is used to train the generative network G and the discriminative network D against each other. The specific definition is: Among them, L1(·,·) is the L1 loss function; G represents the generation network, D represents the discrimination network, and I Src represents the polluted sea surface image, I GT Represents the true value image.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, it implements any step in the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer processor, it implements any step in the method for repairing a defaced sea surface image based on a spatially aligned generative adversarial network as described in any one of claims 1 to 8.

Images