Remote sensing image restoration method and system based on heterogenous image
By using the structural information of heterologous images and pre-trained images to repair the network, the problem of missing a large number of areas in remote sensing images is solved, and the accuracy and authenticity of the repair results are improved.
Patent Information
- Application Number
- CN202411536817.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, when solving remote sensing images with a large number of missing areas, it is difficult to ensure the accuracy and authenticity of repair results.
Using a remote sensing image repair method based on heterologous images, the structural information of heterologous images is used as prior knowledge, and image repair is performed through pre-trained image repair networks, including edge detectors, structure generators and image generators.
Improve the accuracy and authenticity of image repair results to ensure the image quality of the repaired area.
Smart Images
Figure CN119477759A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing technology, and in particular to a remote sensing image restoration method based on heterogeneous images. Background Art
[0002] Due to sensor defects or atmospheric conditions, the acquired remote sensing data contains a large amount of missing information (such as dead pixels, light pollution, and cloud pollution). There are two main traditional image restoration technologies: 1. Diffusion-based methods: gradually diffuse the pixel information around the damaged area in the image and create new textures to fill the missing area. This method is suitable for images with small missing areas. It lacks high-level semantic understanding of the image and may not be able to successfully restore the meaningful texture structure of the missing area. 2. Patch-based methods: (1) Find the most relevant matching patch in the known area of the image and then copy the information to fill the missing area at the pixel level. In most cases, the missing content of the image is unstructured and relevant patches may not be found in the image. (2) Locate images similar to the target missing image in the existing image database and then select appropriate patches to fill in. This method requires a large amount of domain data acquisition and best match search and is only applicable to a limited range of scenarios. Deep learning methods can obtain more high-level semantic understanding of images and have better image restoration effects. They are increasingly widely used in image restoration. However, when solving images with large areas of missing content, the existing technology may have distorted and inaccurate image restoration results due to the lack of sufficient information acquisition. Summary of the invention
[0003] In view of the above technical problems, the purpose of the present invention is to provide a remote sensing image restoration method based on heterogeneous images, which uses the structural information of the heterogeneous images as prior knowledge to repair the missing areas of the image to be restored, thereby improving the accuracy and authenticity of the image restoration results.
[0004] The present invention provides a remote sensing image restoration method based on heterogeneous images, comprising:
[0005] S1. Acquire an image to be restored, a background mask of the image to be restored, and a heterogeneous image, wherein the heterogeneous image is an image of the same time and the same area as the image to be restored but from a different source;
[0006] S2, using a pre-trained image restoration network to obtain a target image based on the image to be restored, the background mask of the image to be restored, and the heterogeneous image; wherein the image restoration network includes an edge detector, a structure generator, and an image generator;
[0007] The edge detector is used to extract structural information from the image to be restored and the heterogeneous image respectively, to obtain a structural edge map of the image to be restored and a structural edge map of the heterogeneous image;
[0008] The structure generator is used to generate a structure prediction map of the image to be restored according to the structure edge maps of the image to be restored and the heterogeneous image;
[0009] The image generator is used to generate a target image according to the image to be repaired and the structure prediction map.
[0010] The specific steps of obtaining the target image using the pre-trained image restoration network in S2 based on the image to be restored, the background mask of the image to be restored and the heterogeneous image are as follows:
[0011] S21: after splicing the image to be restored and its background mask, the image and the heterogeneous image are respectively input into an edge detector to obtain a structural edge map of the image to be restored and the heterogeneous image;
[0012] S22: splicing the image to be restored and the heterogeneous image with their respective structure edge maps, and inputting the two spliced results into a structure generator to obtain a structure prediction map of the image to be restored;
[0013] S23 splices the image to be repaired with the background mask, and then inputs the image and the structure prediction map into an image generator to obtain a target image.
[0014] The structure generator includes an encoder and a decoder. The encoder includes a structure feature extraction module and a feature fusion module. The decoder includes a structure feature reconstruction module.
[0015] The structural feature extraction module is used to extract structural features of the image to be restored and the heterogeneous image after splicing with their respective structural edge maps;
[0016] The feature fusion module is used to fuse the two structural features extracted by the structural feature extraction module;
[0017] The structural feature reconstruction module is used to reconstruct the fused features to obtain a structural prediction graph.
[0018] The image generator adopts a U-Net network structure, including an encoder, an intermediate layer and a decoder, and a jump connection is used between the encoder and the decoder; the encoder includes an image feature extraction module, the decoder includes an image feature reconstruction module, and the intermediate layer includes multiple residual blocks;
[0019] Wherein, the image feature extraction module is used to extract the features of the structure prediction map and the missing image respectively;
[0020] The intermediate layer is used to learn the correlation between two features extracted in the image feature extraction module;
[0021] The image feature reconstruction module is used to generate an image of the missing area of the missing image according to the correlation between the obtained features to obtain a target image.
[0022] The specific process of the image restoration network pre-training includes:
[0023] Obtain a training sample set, wherein the training sample set includes multiple training sample groups, each training sample group includes a complete image A, a missing image A', a background mask M b , structural edge graph S A and heterologous image B;
[0024] The missing image A', the foreign image B and the background mask M b Input the image restoration network to obtain the structure prediction image A1 and the restoration image C;
[0025] The structure prediction graph A1 is combined with the missing image A' and the heterogeneous image B to obtain the first false data. A Merge the missing image A' and the heterogeneous image B to obtain the first true data, input the first false data and the first true data into the structure discriminator D1 respectively to obtain the first discriminant value and the second discriminant value; input the repaired image C and the complete image A into the image discriminator D2 respectively to obtain the third discriminant value and the fourth discriminant value;
[0026] Based on the structure prediction graph A1 and the restored image C, the comprehensive loss function of the image restoration network is calculated, and the parameters of the image restoration network are updated according to the comprehensive loss function to obtain a pre-trained image restoration network.
[0027] Furthermore, the structure discriminator D1 is constructed by analyzing the structure edge graph S A and the structural features of the structure prediction graph A1 are discriminated, and the authenticity of the structural features in the structure prediction graph A1 is judged according to the discrimination result;
[0028] The image discriminator D2 discriminates the image features of the complete image A and the restored image C, and determines the authenticity of the image features in the restored image C according to the discrimination result.
[0029] Furthermore, the comprehensive loss function includes a first loss function L1 and a second loss function L2.
[0030] The first loss function L1 includes the structure generator loss L G1 and structural loss L ST .
[0031] The structure generator loss L G1 Obtained by calculating the difference between the first discriminant value and the second discriminant value;
[0032] The structural loss L ST By calculating the structure prediction graph A1 and the structure edge graph S A The differences in structural features between them are obtained.
[0033] The second loss function L2 includes the reconstruction loss L C , image generator loss L G2 , Perceptual loss L perc and style loss L style .
[0034] The reconstruction loss L C The pixel difference between the restored image C and the complete image A is calculated.
[0035] The image generator loss L G2 Obtained by calculating the difference between the third discriminant value and the fourth discriminant value;
[0036] The perceptual loss L perc Obtained by calculating the difference between the feature map of the restored image C and the feature map of the complete image A;
[0037] The style loss L style It is obtained by calculating the difference between the feature map mapping matrix of the restored image C and the feature map mapping matrix of the complete image A.
[0038] On the other hand, the present invention also provides a remote sensing image restoration system based on heterogeneous images, which is used to implement the remote sensing image restoration method based on heterogeneous images as described above, comprising:
[0039] An image acquisition module is used to acquire the image to be restored and the heterogeneous image, wherein the heterogeneous image is an image of the same time and the same area as the image to be restored but from a different source;
[0040] The image restoration module is used to use the image restoration network to restore the image to be restored based on the heterogeneous image to obtain a target image.
[0041] The image restoration network includes: an edge detector, a structure generator and an image generator.
[0042] The edge detector is used to extract structural information from the image to be restored and the heterogeneous image, and obtain structural edge maps of the image to be restored and the heterogeneous image respectively;
[0043] The structure generator is used to generate a structure prediction map of the image to be restored according to the structure edge map;
[0044] The image generator is used to generate a complete target image according to the structure prediction map.
[0045] The beneficial effects produced by the present invention are:
[0046] 1. The present invention constructs an image restoration network, including an edge detector, a structure generator and an image generator. The network can use heterogeneous images to restore remote sensing images with a high degree of effective domain loss. Based on the structural information of the heterogeneous images as prior knowledge, the structure generator is used to predict and generate the structure of the missing area of the image to be restored, thereby guiding the generation of image texture, color, etc. in the subsequent image generator, ensuring the accuracy and authenticity of the image of the restored area in the target image.
[0047] 2. The present invention adopts a structure discriminator and an image discriminator to evaluate the two stages of structure prediction and image generation respectively, and constructs a first loss function and a second loss function to optimize the structure generator and the image generator respectively, and finally realizes the optimization of the image restoration network and improves the quality of the generated target image. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 A flow chart of a remote sensing image restoration method based on heterogeneous images provided by the present invention;
[0050] Figure 2 A schematic diagram of the process of generating a target image using the image restoration network provided by the present invention;
[0051] FIG3( a ) is a schematic diagram of the structure of a structure generator in an image restoration network provided by the present invention;
[0052] FIG3( b ) is a schematic diagram of the structure of an image generator in an image restoration network provided by the present invention;
[0053] Figure 4 A schematic diagram of the pre-training process of the image restoration network provided by the present invention. DETAILED DESCRIPTION
[0054] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0055] refer to Figure 1 The present invention provides a remote sensing image restoration method based on heterogeneous images, the method comprising:
[0056] S1. Acquire an image to be restored, a background mask of the image to be restored, and a heterogeneous image, wherein the heterogeneous image is an image obtained at the same time and in the same area as the image to be restored but from a different source.
[0057] S2. Using a pre-trained image restoration network, a target image is obtained based on the image to be restored, a background mask of the image to be restored, and a foreign image; wherein the image restoration network includes an edge detector, a structure generator, and an image generator.
[0058] The background mask in S1 is generated according to the image to be restored and is used to extract the known area of the image to be restored; the image to be restored is any one or more combinations of remote sensing images with shadows, remote sensing images covered by clouds and remote sensing images with dead pixels.
[0059] The S1 also includes preprocessing of all images, including denoising, contrast enhancement, color correction, registration, cropping, etc., to improve the accuracy of subsequent processing.
[0060] The edge detector is used to extract useful structural information from the image to be restored and the heterogeneous image respectively, and obtain structural edge maps of the image to be restored and the heterogeneous image respectively.
[0061] The structure generator is used to generate a structure prediction map of the image to be restored according to the structure edge maps of the image to be restored and the heterogeneous image.
[0062] The image generator is used to generate a target image according to the image to be repaired and the structure prediction map.
[0063] The edge detector may adopt any one of a Canny edge detection algorithm, a Laplacian edge detection algorithm, a Scharr edge detection algorithm, an edge preserving smoothing algorithm, and the like.
[0064] The structure generator G1 includes an encoder and a decoder. The encoder includes a structure feature extraction module E1 and a feature fusion module F1. The decoder includes a structure feature reconstruction module R1.
[0065] Furthermore, the structural feature extraction module E1 includes two branches, which are respectively used to extract the structural features of the image to be restored and the heterogeneous image after splicing with their respective structural edge maps. In this embodiment, each branch includes 4 convolution blocks. In the first three convolution blocks, each convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm), a ReLU activation layer and a pooling layer, and the last convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer.
[0066] The feature fusion module F1 is used to fuse the two structural features extracted in the structural feature extraction module. In this embodiment, the feature fusion module F1 includes p residual block (p≥5) structures, each residual block includes two convolutional layers (Conv), each convolutional layer is connected to a batch normalization layer (BatchNorm) and a ReLU activation function, and an expanded convolutional layer is used in the residual block.
[0067] The structural feature reconstruction module R1 is used to reconstruct the fused features to obtain a structural prediction graph. In this embodiment, the structural feature reconstruction module R1 includes 4 convolution blocks, the first 3 convolution blocks include a convolution layer (Conv), a batch normalization layer (BatchNorm), a ReLU activation layer and an upsampling layer, and the last convolution block includes a convolution layer (Conv) and a softmax function layer.
[0068] The image generator G2 adopts a U-Net network structure, including an encoder, an intermediate layer and a decoder, and a jump connection is used between the encoder and the decoder. The encoder includes an image feature extraction module E2, and the decoder includes an image feature reconstruction module R2.
[0069] Furthermore, the image feature extraction module E2 is used to extract the features of the structure prediction map and the image to be restored respectively. In this embodiment, the image feature extraction module E2 includes 5 convolution blocks, and each of the first 4 convolution blocks includes a convolution layer (Conv), a batch normalization layer (BatchNorm), a ReLU activation layer and a pooling layer, and the last convolution block includes a convolution layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer.
[0070] The intermediate layer is used to learn the correlation between two features extracted in the image feature extraction module. In this embodiment, the intermediate layer includes q ResBlock residual blocks (q≥8), each residual block includes two convolutional layers, each convolutional layer is connected to a batch normalization layer (BatchNorm) and a ReLU activation function, and the last convolutional layer is not connected to an activation function.
[0071] The image feature reconstruction module R2 is used to generate an image of the missing area of the missing image according to the correlation between the two features obtained to obtain the target image. In this embodiment, the image feature reconstruction module R2 includes 5 transposed convolution blocks and an output layer, each transposed convolution block includes an upsampling layer, a transposed convolution layer (Conv), a batch normalization layer (BatchNorm) and a ReLU activation layer; the output layer is a 1×1 convolution layer.
[0072] The specific steps of obtaining the target image using the pre-trained image restoration network in S2 based on the image to be restored, the background mask of the image to be restored and the heterogeneous image are as follows:
[0073] S21 splices the image to be restored and its background mask, and then inputs the image to be restored and the heterogeneous image into an edge detector respectively to obtain a structural edge map of the image to be restored and a structural edge map of the heterogeneous image.
[0074] S22: splicing the structure edge map of the image to be restored with the image to be restored, splicing the structure edge map of the heterogeneous image with the heterogeneous image, and inputting the spliced results into the structure generator to obtain a structure prediction map of the image to be restored;
[0075] The specific process of obtaining the structure prediction map of the image to be repaired is as follows:
[0076] S221 splices the structural edge map of the image to be restored with the image to be restored and inputs them into the structural feature extraction module E1 to obtain the structural features of the image to be restored; splices the structural edge map of the heterogeneous image with the heterogeneous image and inputs them into the structural feature extraction module E1 to obtain the structural features of the heterogeneous image.
[0077] S222 Input the structural features of the image to be restored and the structural features of the heterogeneous image into the feature fusion module F1 for feature fusion.
[0078] S223 inputs the fused features into the structural feature reconstruction module R1 for feature reconstruction to obtain a structural prediction map of the image to be repaired.
[0079] S23 splices the image to be repaired with the background mask, and then inputs the image generator with the structure prediction map, generates an image based on the structure information of the missing area corresponding to the structure prediction map and the image information of the known area in the image to be repaired, to obtain the target image.
[0080] Among them, the specific process of obtaining the target image in the image generator is as follows:
[0081] S231 splices the image to be restored with the background mask, and then inputs the image and the structure prediction map into an image feature extractor for feature extraction to obtain image features of the image to be restored and image features of the structure prediction map.
[0082] S232 inputs the image features of the image to be restored and the image features of the structure prediction map into the middle layer to learn the correlation between the two image features.
[0083] S233 inputs the image features of the image to be repaired, the image features of the structure prediction map, and the correlation between the two image features into the image feature reconstruction module, and generates the image features of the missing area in the image to be repaired with reference to the image features of the structure prediction map to obtain the target image.
[0084] The specific process of the image restoration network pre-training includes:
[0085] 1. Obtain a training sample set, which includes multiple training sample groups, each of which contains a complete image A, a missing image A', a background mask M b , structural edge graph S A and heterologous image B.
[0086] The heterogeneous image B is a remote sensing image that is taken at the same time and in the same area as the complete image A but has a different source.
[0087] The missing image A' and its background mask M b The specific generation method is:
[0088] Use a random number generator to randomly generate a background mask M of a certain shape and size on the complete image A b , apply the background mask to the remote sensing image A to obtain the missing image A'. The mask is a binary matrix, 1 represents the area to be covered, and 0 represents the area to be retained.
[0089] The structural edge graph S A It is obtained by inputting the complete image A into the edge detector.
[0090] 2. The missing image A', the foreign image B and the background mask M b Input the image restoration network to obtain the structure prediction map A1 and the restored image C.
[0091] Furthermore, the specific steps of generating the structure prediction image A1 and the repaired image C are as follows:
[0092] (1) Input the missing image A' and the heterogeneous image B into the edge detector respectively to obtain the structural edge map S of the missing image A' A’ and the structural edge map S of the heterogeneous image B B .
[0093] (2) The missing image A' and its structural edge map S A’ Splicing, heterogeneous image B and its structural edge map S BThe splicing is performed, and the splicing results are input into the structure generator G1 to obtain the structure prediction graph A1.
[0094] The specific generation process of the structure prediction graph A1 is as follows:
[0095] The missing image A' and its structural edge map S A’ After splicing, the structure feature extraction module E1 is input to extract the structure feature, and the structure feature f of the missing image A' is obtained. a’ . The heterogeneous image B and its structural edge map S B After splicing, the structure feature extraction module E1 is input to extract the structure feature, and the structure feature f of the heterogeneous image B is obtained. b .
[0096] The structural feature f a’ and structural features f b The input feature fusion module F1 performs feature fusion, and the fused features are input into the structural feature reconstruction module R1 for reconstruction to obtain the structural prediction graph A1.
[0097] (3) The structure prediction image A1, the missing image A' and the background mask M b Input the image generator G2 to obtain the repaired image C.
[0098] The specific generation process of the repaired image C is as follows:
[0099] The missing image A' and the background mask M b After splicing, the structure prediction graph A1 is input into the image feature extraction module E2 to obtain the image feature g of the structure prediction graph A1. a1 and the image feature g of the missing image A' A’ .
[0100] The image feature g a1 and image feature g A’ Input the middle layer to learn the image feature g a1 With image feature g A’ The correlation between them.
[0101] The image feature g a1 and image feature g A’ The correlation between the two features is input into the image feature reconstruction module R2, and the image features of the reference structure prediction image A1 are used to generate the image of the missing area of the missing image A' to obtain the repaired image C.
[0102] 3. The structure prediction graph A1, the missing image A' and the heterogeneous image B are merged based on the channel dimension to obtain the first false data concat(A1, (A', B)); the structure edge graph S A, the missing image A' and the heterogeneous image B are merged based on the channel dimension to obtain the first true data concat(S A , (A', B)). The first false data and the first true data are respectively input into the structure discriminator D1 to obtain a first discriminant value D1(A1, (A', B)) and a second discriminant value D1(S A , (A',B)).
[0103] The repaired image C and the complete image A are respectively input into the image discriminator D2 to obtain a third discriminant value D2 (C) and a fourth discriminant value D2 (A).
[0104] The structure discriminator D1 is constructed by analyzing the structure prediction graph A1 and the structure edge graph S A The structural features of the structure prediction graph A1 generated in the structure generator are judged to obtain a first judgment value and a second judgment value; according to the difference between the first judgment value and the second judgment value, it can be judged whether the structural features of the structure prediction graph A1 generated in the structure generator are true.
[0105] The image discriminator D2 obtains a third discrimination value and a fourth discrimination value by discriminating the image features of the restored image C and the complete image A; and whether the restored image generated in the image generator is authentic can be determined based on the difference between the third discrimination value and the fourth discrimination value.
[0106] In this embodiment, the structure discriminator D1 is a PatchGAN network structure, and the image discriminator D2 is a SM-PatchGAN network structure.
[0107] 4. Based on the structure prediction map A1 and the restored image C, calculate the comprehensive loss function of the image restoration network, update the network parameters of the image restoration network according to the comprehensive loss function, and obtain a pre-trained image restoration network.
[0108] The comprehensive loss function L includes a first loss function L1 and a second loss function L2, which are calculated as follows:
[0109] L=αL1+βL2
[0110] Among them, L1 is the first loss function, L2 is the second loss function, α and β represent the weight parameters of the first loss function and the second loss function respectively.
[0111] The first loss function L1 includes the structure generator G1 loss L G1 and structural loss L ST ;
[0112] L1=λ1L G1 +λ2L ST
[0113] Among them, λ1 and λ2 represent the loss L of the structure generator G1, respectively. G1 and structural loss L ST The weight parameters are set as λ1=1 and λ2=5 in this embodiment.
[0114] The structure generator G1 loses L G1 By calculating the first discriminant value D1(A1, (A', B)) and the second discriminant value D1(S A , (A', B)) is obtained, and the specific calculation method is as follows:
[0115] L G1 =E[log(D1(S A ,Z))]+E[log(1-D1(A1,Z))]
[0116] Among them, L G1 represents the loss of the structure generator G1, Z = {A', B}, D1(S A , Z) represents the first discriminant value, and D1(A1, Z) represents the second discriminant value.
[0117] The structural loss L ST By calculating the structure prediction graph A1 and the structure edge graph S A The structural feature differences between them are obtained, and the specific calculation method is as follows:
[0118]
[0119] Among them, N represents the number of channels of the image, H represents the height of the image, W represents the width of the image, and ||·||1 represents the L1 norm.
[0120] The second loss function L2 includes the reconstruction loss L C 、Image generator G2 loss L G2 , Perceptual loss L perc and style loss L style ;
[0121] L2=γ1L C +γ2L G2 +γ3L perc +γ4L style
[0122] Among them, γ1, γ2, γ3 and γ4 represent the reconstruction loss L C 、Image generator G2 loss L G2 , Perceptual loss L perc and style loss L style The weight parameters are set as follows: in this embodiment, γ1=1, γ2=0.1, γ3=0.1, and γ4=250.
[0123] The reconstruction loss L C It represents the reconstruction loss of the restored image C and the complete image A at the pixel level. It is obtained by calculating the difference in pixel values of each pixel between the restored image C and the complete image A. The specific calculation method is as follows:
[0124]
[0125] Where N represents the number of channels of the image, H represents the height of the image, W represents the width of the image, A represents the complete image, C represents the restored image, and ||·||1 represents the L1 norm.
[0126] Manhattan distance is used to calculate the pixel difference between the repaired image C and the complete image A. The calculation formula is as follows:
[0127]
[0128] n represents the spectral channel dimension of the image (1≤n≤N), i represents the number of rows of image pixels (1≤i≤H), j represents the number of columns of image pixels (1≤j≤W), Represents the pixel value of the pixel at the i-th row and j-th column of the n-th channel of the complete image A. Represents the pixel value of the pixel at the i-th row and j-th column of the n-th channel of the repaired image C.
[0129] The image generator G2 loses L G2 It is obtained by calculating the difference between the third discriminant value D2(C) and the fourth discriminant value D2(A), and the specific calculation method is as follows:
[0130] L G2 =E[(D2(C)-D2(A)) 2 ]
[0131] Among them, L G2 represents the loss of the image generator G2, D2(C) represents the third discriminant value, and D2(A) represents the fourth discriminant value.
[0132] The perceptual loss L perc It is obtained by calculating the difference between the feature map of the repaired image C and the feature map of the complete image A. The specific process is as follows:
[0133] The repaired image C and the complete image A are respectively input into the pre-trained VGG-19 network for feature extraction to obtain the feature maps φ corresponding to the relu1_1, relu2_1, relu3_1, and relu4_1 layers in the VGG-19 network. t (t=1,2,3,4),φ t The size is N t ×H t ×Wt , calculate the difference between the corresponding feature maps obtained by the repaired image C and the complete image A at each convolution layer. The specific calculation method is as follows:
[0134]
[0135] Where t represents the number of layers selected in the VGG network (t = 1, 2, 3, 4); φ t Represents the feature map of the tth layer of the network VGG-19, with a size of N t ×H t ×W t , φ1, φ2, φ3, and φ4 correspond to the feature maps of the relu1_1, relu2_1, relu3_1, and relu4_1 layers of the VGG-19 network, respectively.
[0136] Style loss L style It is obtained by calculating the difference between the feature map mapping matrix of the restored image C and the feature map mapping matrix of the complete image A. The specific process is as follows:
[0137] The repaired image C and the complete image A are respectively input into the pre-trained VGG-19 network for feature extraction to obtain the feature maps φ corresponding to the relu1_1, relu2_1, relu3_1, and relu4_1 layers in the VGG-19 network. t (t=1,2,3,4), the feature map φ t Mapping to get the corresponding Gram matrix Calculate the feature map mapping matrix of the repaired image C Mapping matrix with the feature map of the complete image A The difference between them is calculated as follows:
[0138]
[0139] Where t represents the convolutional layer selected in the VGG network, and t = 1, 2, 3, and 4 correspond to the relu1_1, relu2_1, relu3_1, and relu4_1 layers of the VGG-19 network, respectively; φ t is the feature map corresponding to the tth layer; Represented by the feature map φ t The mapped Gram matrix; x represents a given image, such as a complete image A or a restored image C.
[0140] The process of updating the network parameters of the image restoration network is as follows: updating the parameters of the structure generator according to the first loss function to generate a structure prediction graph A1 with more realistic structural features; and updating the parameters of the image generator according to the second loss function to generate a restoration image C with more realistic image features.
[0141] The present invention also provides a remote sensing image restoration system based on heterogeneous images, comprising the following modules:
[0142] A data acquisition module, used to acquire the image to be restored and the heterogeneous image, wherein the heterogeneous image is an image of the same time and the same area as the image to be restored but from a different source;
[0143] The image restoration module is used to use the image restoration network to restore the image to be restored based on the heterogeneous image to obtain the target image.
[0144] The image restoration network includes: an edge detector, a structure generator and an image generator.
[0145] The edge detector is used to extract useful structural information from the image to be restored and the heterogeneous image respectively, and obtain structural edge maps of the image to be restored and the heterogeneous image respectively.
[0146] The structure generator is used to generate a structure prediction map of the image to be restored according to the structure edge map.
[0147] The image generator is used to generate a complete target image according to the structure prediction map.
[0148] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.
[0149] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.
Claims
1. A remote sensing image restoration method based on heterogeneous images, characterized in that: The following steps are involved: S1: Acquire an image to be restored, a background mask of the image to be restored, and a heterogeneous image, wherein the heterogeneous image is an image of the same time and the same area as the image to be restored but from a different source; S2: using a pre-trained image restoration network to obtain a target image based on the image to be restored, the background mask of the image to be restored, and the heterogeneous image; wherein the image restoration network includes an edge detector, a structure generator, and an image generator; The edge detector is used to extract structural information from the image to be restored and the heterogeneous image respectively, to obtain a structural edge map of the image to be restored and a structural edge map of the heterogeneous image; The structure generator is used to predict and generate a structure prediction map of the image to be restored based on the structure edge maps of the image to be restored and the heterogeneous image; The image generator is used to generate a target image according to the image to be repaired and the structure prediction map.
2. The remote sensing image restoration method based on heterogeneous images according to claim 1 is characterized in that: The specific steps of obtaining the target image using the pre-trained image restoration network based on the image to be restored, the background mask of the image to be restored, and the heterogeneous image are as follows: S21: after splicing the image to be restored and its background mask, the image and the heterogeneous image are respectively input into an edge detector to obtain a structural edge map of the image to be restored and the heterogeneous image; S22: splicing the image to be restored and the heterogeneous image with their respective structure edge maps, and inputting the two spliced results into a structure generator to obtain a structure prediction map of the image to be restored; S23 splices the image to be repaired with the background mask, and then inputs the image and the structure prediction map into an image generator to obtain a target image.
3. The remote sensing image restoration method based on heterogeneous images according to claim 1 is characterized in that: The structure generator includes an encoder and a decoder, the encoder includes a structure feature extraction module and a feature fusion module, and the decoder includes a structure feature reconstruction module; The image generator adopts a U-Net network structure, including an encoder, an intermediate layer and a decoder, and a jump connection is used between the encoder and the decoder; the encoder includes an image feature extraction module, the decoder includes an image feature reconstruction module, and the intermediate layer includes multiple residual blocks.
4. The remote sensing image restoration method based on heterogeneous images according to claim 3 is characterized in that: The structural feature extraction module is used to respectively extract the structural features of the image to be restored and the heterogeneous image after splicing with their respective structural edge maps; The feature fusion module is used to fuse the two structural features extracted by the structural feature extraction module; The structural feature reconstruction module is used to reconstruct the fused features to obtain a structural prediction graph; The image feature extraction module is used to extract the features of the structure prediction map and the missing image respectively; The intermediate layer is used to learn the correlation between two features extracted in the image feature extraction module; The image feature reconstruction module is used to generate an image of the missing area of the missing image according to the correlation between the two features to obtain a target image.
5. The remote sensing image restoration method based on heterogeneous images according to claim 1, characterized in that: The pre-training process of the image restoration network includes: Obtain a training sample set, wherein the training sample set includes multiple training sample groups, each training sample group includes a complete image A, a missing image A', a background mask M b , structural edge graph S A and heterologous image B; The missing image A', the foreign image B and the background mask M b Input the image restoration network to obtain the structure prediction image A1 and the restoration image C; The structure prediction graph A1 is combined with the missing image A' and the heterogeneous image B to obtain the first false data. A Merge the missing image A' and the heterogeneous image B to obtain the first true data, input the first false data and the first true data into the structure discriminator D1 respectively to obtain the first discriminant value and the second discriminant value; input the repaired image C and the complete image A into the image discriminator D2 respectively to obtain the third discriminant value and the fourth discriminant value; Based on the structure prediction graph A1 and the restored image C, the comprehensive loss function of the image restoration network is calculated, and the parameters of the image restoration network are updated according to the comprehensive loss function to obtain a pre-trained image restoration network.
6. The remote sensing image restoration method based on heterogeneous images according to claim 5 is characterized in that: The structure discriminator D1 is constructed by A and the structural features of the structure prediction graph A1 are discriminated, and the authenticity of the structural features in the structure prediction graph A1 is judged according to the discrimination result; The image discriminator D2 discriminates the image features of the complete image A and the restored image C, and determines the authenticity of the image features in the restored image C according to the discrimination result.
7. The remote sensing image restoration method based on heterogeneous images according to claim 5 is characterized in that: The comprehensive loss function includes a first loss function L1 and a second loss function L2; The first loss function L1 includes the structure generator loss L G1 and structural loss L ST ; The structure generator loss L G1 The structural loss L is obtained by calculating the difference between the first discrimination value and the second discrimination value; ST By calculating the structure prediction graph A1 and the structure edge graph S A The differences in structural features between them are obtained; The second loss function L2 includes the reconstruction loss L C , image generator loss L G2 , Perceptual loss L perc and style loss L style ; The reconstruction loss L C The image generator loss L is obtained by calculating the pixel difference between each pixel of the repaired image C and the complete image A; G2 The perceptual loss L is obtained by calculating the difference between the third discrimination value and the fourth discrimination value; perc The style loss L is obtained by calculating the difference between the feature map of the restored image C and the feature map of the complete image A. style It is obtained by calculating the difference between the feature map mapping matrix of the repaired image C and the feature map mapping matrix of the complete image A.
8. A remote sensing image restoration system based on heterogeneous images, characterized in that: The remote sensing image restoration system specifically includes: An image acquisition module is used to acquire the image to be restored and the heterogeneous image, wherein the heterogeneous image is an image of the same time and the same area as the image to be restored but from a different source; An image restoration module, used for using an image restoration network to restore the image to be restored based on the heterogeneous image to obtain a target image; Wherein, the image restoration network includes: an edge detector, a structure generator and an image generator; The edge detector is used to extract structural information from the image to be restored and the heterogeneous image, and obtain structural edge maps of the image to be restored and the heterogeneous image respectively; The structure generator is used to generate a structure prediction map of the image to be restored according to the structure edge map; The image generator is used to generate a complete target image according to the structure prediction map.