A method for intelligent restoration of hidden targets in remote sensing images based on image inpainting

Abstract

This invention provides an intelligent method for restoring hidden targets in remote sensing images based on image inpainting. It constructs a gated convolutional target restoration network and trains an image inpainting model capable of adapting to remote sensing images with significant color variations, fine granularity, and complex features. The gating mechanism provides remote sensing target mask information in the front-end network, while the back-end network acts as an attention mechanism, dynamically learning the pixel distribution characteristics of uncovered target areas and non-target areas in the remote sensing image. This invention uses the original remote sensing image, a combination of a random overlay mask and a target contour mask as input for model training. Guided by labels indicating the hidden target area and the exact location of the remote sensing target, the model can quickly identify key targets in remote sensing images with complex color gradients and large pixel value differences. It then learns the contour features of the remote sensing target and the complex features of its surrounding environment, ensuring that the restored remote sensing target conforms to the features and distribution patterns of ground features in the original image.

Description

Technical Field

[0001] This invention relates to the field of physical technology, specifically to an intelligent method for restoring hidden targets in remote sensing images based on image restoration. Background Technology

[0002] The rapid development of remote sensing technology has provided a vast data foundation for socio-economic and national policy adjustments. Due to national policy requirements, important targets in remote sensing imagery are often concealed to varying degrees and using different technical methods during the mapping process. After the declassification period, these images require technical means to restore the concealed targets. Traditional image encryption and decryption methods, even knowing the target's location, cannot effectively restore it, nor can they handle new declassification and desensitization techniques primarily based on deep learning. Therefore, a target restoration technology is needed that can be based on the surrounding area and can be either manually guided or automated.

[0003] Deep learning-based image inpainting offers a solution to this task. As a branch of image generation, it focuses more on completing images rather than generating them entirely. It is better suited for tasks that modify partial regions of an image, and the restoration of hidden targets is precisely a task that involves processing partial image regions. Currently, image inpainting models are mostly used to repair images with missing or damaged parts, adopting a "remove first, then fill in" approach: first removing the parts that need modification, and then using image inpainting techniques to repair them.

[0004] Hidden targets can be achieved using guided image inpainting techniques. Guided image inpainting involves a deep learning neural network first inferring the features, characteristics, and distribution patterns of ground features based on the pixels in an image. Then, a mask is used as a reference for image restoration, generating an image that fits the mask's features. This technique, by combining a provided target outline, learns the features and distribution patterns of ground features in the scene where the target is located, generating an image that conforms to both the target's location as defined in the guided mask and the scene's features and distribution patterns, thus enabling the task of restoring hidden targets.

[0005] In their paper "Free-Form Image Inpainting with Gated Convolution" (IEEE International Conference on Computer Vision, 2019: 4471-4480), Jiahui Y, Zhe L, Jimei Y, et al. proposed a user-guided image inpainting technique. Based on Deepfillv1, this method replaces all convolutional layers with gated convolutional layers and applies the key concepts of gated convolution to dilated convolutional layers and content attention mechanisms, improving the feature extraction capabilities of each network layer at the channel and pixel levels. The random mask generation method used during training enhances the model's generalization ability on various overlay masks, enabling it to adapt to different types of masks. The model also provides an extension capability: the ability to learn the main contours of a scene. By comparing the contours of pixels in the scene with those provided by the guiding mask during training, the model learns the contour information of objects in the scene as reflected in the training set, and uses the guiding mask as a reference for fitting, achieving excellent results. However, this paper only explores guided image inpainting capabilities for most objects with discernible contours. The datasets used in the paper are all clear, ordinary optical images, including mountains, rivers, oceans, human bodies, and faces. No applications in remote sensing are mentioned. Unlike ordinary optical images, remote sensing images are often high-altitude aerial photographs. If taken by satellite, the objects displayed on the ground often lack very clear contours. Ordinary contour determination methods, such as the Canny operator or the deep learning-based edge detection model mentioned in the paper, cannot adequately meet these contour requirements.

[0006] Therefore, a method for intelligently restoring hidden targets is needed. Summary of the Invention

[0007] This invention addresses the current lack of contour extraction model operators and the absence of precedents for hidden target restoration techniques. It provides an intelligent method for restoring hidden targets in remote sensing images based on image inpainting. The method uses a gated convolutional target restoration network as a guided image inpainting model and combines it with traditional image processing techniques for restoring hidden targets to restore hidden targets in remote sensing images. This enables automated intelligent restoration of de-densified and desensitized targets in remote sensing mapping.

[0008] This invention provides a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting, comprising the following steps:

[0009] S1. Select remote sensing images and corresponding label images, crop and filter them to form a dataset, and obtain remote sensing target area mask and remote sensing target contour mask;

[0010] S2. Construct a gated convolutional target reconstruction network. The gated convolutional target reconstruction network includes a pre-generation network and a fine generation network in series. The pre-generation network generates remote sensing target mask information and outputs it to the fine generation network. The fine generation network uses an attention mechanism to dynamically learn the pixel distribution features of the target area and non-target area of ​​the uncovered remote sensing image and reconstruct the hidden target in the hidden area.

[0011] S3. Use the dataset, remote sensing target region mask and remote sensing target contour mask to train the gated convolutional target reconstruction network. Use gradient descent to iteratively update the parameters of each layer in the gated convolutional target reconstruction network until the total cost function converges to obtain the trained gated convolutional target reconstruction network.

[0012] S4. Use a semantic segmentation network to detect hidden regions in the remote sensing image to detect modified parts of the remote sensing image to be restored and output the label image to be restored containing the outlines of the hidden regions.

[0013] S5. The label image to be restored is the target region mask of the image to be restored. The target region mask of the image to be restored is eroded to obtain the specific target region, and then the contour is extracted to obtain the target contour mask of the remote sensing image to be restored.

[0014] S6. Cover the label image to be restored with the target region mask of the remote sensing image to be restored, and input it together with the target contour mask of the remote sensing image to be restored into the trained gated convolutional target restoration network to obtain a highly restored result. After stitching, the restored remote sensing image is obtained.

[0015] The present invention discloses a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting. In a preferred embodiment, step S1 includes:

[0016] S11. Randomly select at least 30 high-resolution remote sensing images and corresponding label images from the semantic segmentation dataset that have a relatively balanced ratio of foreground to background.

[0017] S12. Crop each high-resolution image and its corresponding label image to a size of 256×256 pixels;

[0018] S13. Select images with a foreground pixel ratio of more than 10% in the cropped label images and the corresponding remote sensing images to form a dataset. The dataset includes a training set, a validation set, and a test set.

[0019] S14. Dilate the random mask to obtain the remote sensing target area mask, and extract the contour of the cropped label image to obtain the remote sensing target contour mask.

[0020] In the present invention, a method for intelligent restoration of hidden targets in remote sensing images based on image restoration is preferred. In step S14, the random mask is a combination of an irregular mask formed by connecting random discrete points and a regular rectangular mask. The random discrete points are generated by a random generation algorithm.

[0021] The present invention provides a method for intelligent restoration of hidden targets in remote sensing images based on image restoration. In a preferred manner, a random mask is expanded with a kernel of size 12×12 and the expansion number is 1 to obtain a mask for the remote sensing target region. The outline of the cropped label image is extracted and a line of thickness 1 is used in the outline reconstruction process to obtain a mask for the outline of the remote sensing target.

[0022] The present invention discloses a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting. In a preferred embodiment, step S2 includes:

[0023] S21. Construct a pre-generated network. The pre-generated network has an 18-layer structure. The pre-generated network includes 12 consecutive gated convolutional layers, upsampling layers, gated convolutional layers, upsampling layers, two consecutive gated convolutional layers and activation layers. The pre-generated network generates remote sensing target mask information and outputs it to the fine generation network.

[0024] S22. Construct a refined generative network. The last eight layers of the refined generative network are identical to the last eight layers of the pre-generative network. The first to tenth layers are two parallel branches.

[0025] Branch 1 is identical to the first ten layers of the pre-generated network;

[0026] Branch 2 has one less layer than Branch 1. The first five layers and the last two layers of Branch 2 correspond to the first five layers and the last two layers of the pre-generated network and have the same number of layers. The sixth layer of Branch 2 adds the ReLU activation function and the seventh layer adds a content attention mechanism.

[0027] The outputs of branch one and branch two are concatenated using concatenation.

[0028] The fine generative network uses an attention mechanism to dynamically learn the pixel distribution features of target regions and non-target regions in uncovered remote sensing images and to reconstruct hidden targets in the hidden regions.

[0029] S23. The pre-generated network and the fine-generated network are concatenated to obtain the gated convolutional target restoration network.

[0030] In the present invention, a method for intelligent restoration of hidden targets in remote sensing images based on image restoration is preferred. In step S21, the kernel size of the first gated convolutional layer is 5×5 and the stride is 1. The stride of the second and fourth gated convolutional layers is 2. The kernel size of the remaining gated convolutional layers is 3×3 and the stride is 1. The seventh to tenth gated convolutional layers use dilated convolution. The activation function of the eighteenth layer is Tanh.

[0031] The present invention provides a method for intelligent restoration of hidden targets in remote sensing images based on image restoration. In a preferred embodiment, in step S3, a training set is input into a gated convolutional target restoration network. First, a remote sensing target region mask is used to remove a portion of the remote sensing image, and then the portion where the remote sensing target contour mask intersects with the random mask is added.

[0032] The input to the training set is:

[0033] Input=Concat(inI,M,guide);

[0034] inI represents the cascaded missing images along the channel dimension, M represents the random overlay mask, and guide represents the remote sensing target contour mask;

[0035] inI=I·(1-M / 255)+guide / 255·M / 255.

[0036] In the present invention, a method for intelligent restoration of hidden targets in remote sensing images based on image restoration is preferred. In step S3, the optimizer is Adam, the learning rate is 0.0001, the optimizer parameters beta1 is 0.5, and beta2 is 0.999.

[0037] The present invention discloses a method for intelligent restoration of hidden targets in remote sensing images based on image restoration. In a preferred embodiment, in step S5, a hidden region is obtained through a semantic segmentation network. The hidden region is the target region mask of the image to be restored. The original target contour is then deduced from the obtained hidden region to obtain the target contour mask of the remote sensing image to be restored.

[0038] In the preferred embodiment of the intelligent restoration method for hidden targets in remote sensing images based on image restoration described in this invention, in step S5, the target area mask of the image to be restored is eroded with a kernel of size 12×12 and the number of erosions is 1 to obtain the specific target area. When extracting the contour of the specific target area, a line with a thickness of 1 is used in the contour reproduction process.

[0039] This invention discloses an intelligent method for restoring hidden targets in remote sensing images based on image inpainting, used for restoring hidden targets in remote sensing mapping. The implementation steps are as follows: 1. Generate a remote sensing target contour dataset; 2. Construct a gated convolutional target restoration network; 3. Train the gated convolutional target restoration network; 4. Use a trained semantic segmentation network to detect target contours and their locations; 5. Generate remote sensing target region masks and contour masks; 6. Highly restore the remote sensing target. This invention constructs a gated convolutional target restoration network and trains an image inpainting model capable of adapting to remote sensing images with significant color changes, fine granularity, and complex features. A gated convolutional network replaces a traditional convolutional network. The gating mechanism provides remote sensing target mask information in the front-end network, while the back-end network acts as an attention mechanism, dynamically learning the pixel distribution features of uncovered target areas and non-target areas in the remote sensing image. This invention uses a combination of random overlay masks and target contour masks, making the trained model more consistent with remote sensing targets with complex shapes and variable positions, thus enhancing the model's versatility. This invention processes labels in a semantic segmentation dataset to obtain a remote sensing target contour mask, which is then applied to the training process. This method facilitates the subsequent use of a manually constructed contour mask to guide the model reconstruction process, compensating for contour errors obtained by the front-end target detection method. It also saves computational power and time on training the contour extraction model, utilizing transfer learning to train the reconstruction of hidden targets. This invention uses the original remote sensing image, a random overlay mask combined with the target contour mask, as input for model training. Guided by labels containing the hidden target region and the exact location of the remote sensing target, the model can quickly identify key targets of interest in remote sensing images with complex color gradients and large pixel value differences. It then learns the contour features of the remote sensing target and the complex features of its surrounding environment, ensuring that the reconstructed remote sensing target conforms to the features and distribution patterns of ground objects in the original image.

[0040] The idea behind achieving the objective of this invention is to construct a gated convolutional target restoration network and train an image restoration model that can adapt to remote sensing images with significant color changes, fine granularity, and complex features. A gated convolutional network is used instead of a convolutional network. The gating mechanism provides remote sensing target mask information in the front-end network and the back-end network serves as an attention mechanism to dynamically learn the pixel distribution features of target areas and non-target areas in the uncovered remote sensing images.

[0041] This invention uses a combination of random overlay mask and target contour mask to make the trained model more consistent with remote sensing targets with complex shapes and variable positions, thus making the model more versatile.

[0042] This invention processes the labels in a semantic segmentation dataset to obtain a remote sensing target contour mask, which is then applied to the training process. This method facilitates the subsequent use of a manually constructed contour mask to guide the model's reconstruction process, compensating for contour errors obtained by the front-end target detection method. It also saves computational power and time on training the contour extraction model, utilizing transfer learning to train the reconstruction of remote sensing hidden targets. The labels used in this invention are all manually simulable and reproducible. In practical applications, target contour labels obtained from a pre-trained semantic segmentation network can be input into the network of this invention to achieve remote sensing target reconstruction.

[0043] This invention uses the original remote sensing image, a random overlay mask combined with a target contour mask as input for model training. Guided by labels with hidden target areas and the exact location of remote sensing targets, the model can quickly discover key targets of interest in remote sensing images with complex color gradients and large differences in pixel values. It also learns the contour features of the remote sensing target and the complex features of its surrounding environment so that the restored remote sensing target conforms to the features and distribution patterns of ground features in the original image.

[0044] The specific steps of this invention include the following:

[0045] Step 1, Generate remote sensing target contour dataset:

[0046] Unlike training image inpainting networks, this part requires additional preparation of contour masks corresponding to remotely sensed targets to provide the model with key learning areas.

[0047] Step 1.1: Randomly select at least 30 high-resolution remote sensing images and their corresponding label images from the semantic segmentation dataset that have a relatively balanced ratio of foreground to background.

[0048] Step 1.2: Crop each high-resolution image and its corresponding label image to a size of 256×256 pixels;

[0049] Step 1.3: Select images with a foreground pixel ratio of more than 10% in the cropped label images and their corresponding remote sensing images to form a dataset;

[0050] Step 2, construct the gated convolutional target reconstruction network:

[0051] Step 2.1, construct the pre-generated network;

[0052] The pre-generated network consists of 12 gated convolutional layers, with the thirteenth layer being an upsampling layer, the fourteenth layer being a gated convolutional layer, the fifteenth layer being an upsampling layer, two consecutive gated convolutional layers, and the eighteenth layer being an activation layer.

[0053] The first convolutional layer has a 5×5 kernel size and a stride of 1. The second and fourth convolutional layers have a stride of 2, and the remaining convolutional layers all have a 3×3 kernel size and a stride of 1. The seventh to tenth gated convolutional layers use dilated convolution. The eighteenth layer uses Tanh activation.

[0054] Step 2.2, construct a refined generative network;

[0055] The last eight layers of the refined generative network have the same structure as the last eight layers of the pre-generative network. Layers one through ten are two parallel branches.

[0056] Branch 1 is the same as the first ten layers of the pre-generated network.

[0057] Branch 2 has one less layer than Branch 1. Its first five layers and last two layers have the same number of layers as the corresponding first five layers and last two layers of the pre-generated network. The sixth layer adds the ReLU activation function, and the seventh layer adds a content attention mechanism.

[0058] The outputs of branch one and branch two are concatenated.

[0059] Step 2.3: Connect the pre-generated network and the fine-grained generation network in series;

[0060] Step 3, train the gated convolutional object reconstruction network:

[0061] The training set is input into a gated convolutional object reconstruction network. The optimizer is Adam, the learning rate is 0.0001, and the optimizer parameters beta1 is 0.5 and beta2 is 0.999. The parameters of each layer in the gated convolutional object reconstruction network are iteratively updated using gradient descent until the total cost function converges, resulting in the trained gated convolutional object reconstruction network.

[0062] The remote sensing target area mask used during training is a random mask. The random overlay mask is a combination of an irregular mask formed by connecting random discrete dots and a regular rectangular mask. The random overlay mask is generated by a random generation algorithm.

[0063] Step 4: Use the trained semantic segmentation network to detect the target contour and its location:

[0064] Use a semantic segmentation network that can detect hidden regions to detect modified parts of the image.

[0065] This step can use a typical semantic segmentation network, but its output must be the contour corresponding to the hidden region.

[0066] Step 5: Generate the remote sensing target area mask and contour mask:

[0067] Based on the mask generation method, this can be divided into the following two cases:

[0068] 1) For the semantic segmentation dataset labels, perform dilation operation with a kernel of size 12×12 and dilation times of 1 to obtain the region mask. For the remote sensing target contour mask, it is necessary to extract the contour of the label and use a line thickness of 1 in the contour reconstruction process to obtain the remote sensing target contour mask.

[0069] 2) For the labels obtained from the semantic segmentation network, these labels serve as region masks. Erosion is performed using a 12×12 kernel with one erosion iteration to obtain the specific target region. For remote sensing target region masks, contour extraction of the specific target region is required, and a line thickness of 1 is used during contour reconstruction to obtain the remote sensing target contour mask.

[0070] Step 6, highly reconstruct the remotely sensed target:

[0071] Step 6.1: Cover the image with the hidden target using the target region mask, and input it together with the corresponding contour mask into the gated convolutional target reconstruction network to obtain a highly reconstructed result;

[0072] Step 6.2: The restored results of the cropped remote sensing images are stitched together sequentially according to the sequence number to obtain the final reconstructed image.

[0073] The present invention has the following advantages:

[0074] (1) This invention employs a gated convolutional target reconstruction network to reconstruct hidden targets in hidden regions. A gated convolutional network replaces the traditional convolutional network. The gating mechanism provides remote sensing target mask information in the front-end network, while the back-end network acts as an attention mechanism, dynamically learning the pixel distribution features of uncovered target and non-target regions in the remote sensing image. Compared to existing models, this model is more adaptable to remote sensing images with significant color variations, fine granularity, and complex features.

[0075] (2) This invention uses a random overlay mask, which is generated by combining a regular rectangular mask with an irregular curved mask to serve as the missing area of ​​the remote sensing image. This simplifies the method of manually providing the overlay mask during the training of existing models. At the same time, the random mask can train the network to repair the missing area with different shapes and positions, which greatly improves the versatility of the model.

[0076] (3) This invention uses traditional image processing methods to create remote sensing target contour masks from the labeled images of the semantic segmentation dataset, and applies them to model training. This saves computational time and effort in training the contour extraction model, and combines transfer learning to restore hidden targets. Existing contour extraction methods are mostly based on gradient changes, which cannot be applied to remote sensing images with fine pixel granularity. However, the method used in this invention can quickly and efficiently construct a targeted target contour dataset. This method is also beneficial for using manually constructed contour masks to guide the model restoration process, compensating for contour errors obtained by front-end target detection methods. The labels used in this invention are all artificially simulated and reproducible. In practical use, the target contour labels can be obtained from the pre-trained semantic segmentation network and input into the network in this invention to restore remote sensing targets.

[0077] (4) The present invention uses the original remote sensing image, random overlay mask combined with target contour mask as input for model training. Under the guidance of labels with hidden target areas and the exact location of remote sensing targets, the model can quickly discover key targets in remote sensing images with complex color gradients and large differences in pixel values, and learn the contour features of the remote sensing target and the complex features of its surrounding environment so that the restored remote sensing target conforms to the features and distribution patterns of ground features in the original image. Attached Figure Description

[0078] Figure 1 A flowchart of a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting;

[0079] Figure 2 This is a schematic diagram of the structure of a gated convolutional target restoration network, which is a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting.

[0080] Figure 3a This is a remote sensing image from the test set of Example 1 of an intelligent restoration method for hidden targets in remote sensing images based on image inpainting;

[0081] Figure 3b Example 1: Semantic segmentation label dilation result of an intelligent restoration method for hidden targets in remote sensing images based on image inpainting;

[0082] Figure 3c Example 1 shows the result after target hiding in a remote sensing image hiding intelligent restoration method based on image inpainting.

[0083] Figure 3d This is a test set of semantic segmentation labels for an embodiment 1 of a remote sensing image hidden target intelligent restoration method based on image inpainting;

[0084] Figure 3eExample 1 shows the result of edge extraction from the segmented label map, which is the result of an intelligent restoration method for hidden targets in remote sensing images based on image inpainting.

[0085] Figure 3f The input model diagram is for Example 1 of an intelligent restoration method for hidden targets in remote sensing images based on image inpainting;

[0086] Figure 3g This is Example 1 of a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting, showing the restoration result of a hidden target under guidance. Detailed Implementation

[0087] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0088] Example 1

[0089] like Figure 1 As shown, a method for intelligent restoration of hidden targets in remote sensing images based on image inpainting includes the following steps:

[0090] Step 1: Generate a remote sensing target contour dataset.

[0091] Unlike training an image inpainting network, this part requires additional preparation of contour masks corresponding to the remote sensing targets to provide the model with key learning areas. The label processing method is the same as in step 3.

[0092] Step 1.1: In this embodiment of the invention, 30 high-resolution 1500×1500 remote sensing images and their corresponding label images are randomly selected from the Massachusetts building dataset, where the foreground and background ratios are relatively balanced.

[0093] Step 1.2: Crop each high-resolution image and its corresponding label image to a size of 224×224 pixels.

[0094] Step 1.3: Select the label images in the cropped label images where the foreground pixels account for more than 10%, and all the cropped remote sensing images to form a training set.

[0095] Step 2, construct the gated convolutional target reconstruction network:

[0096] Reference Figure 2 The gated convolutional target reconstruction network constructed in this invention will be described in further detail. The pre-generated network and the first and second branches of the network in this model are all encoder-decoder structures.

[0097] Step 2.1, construct the pre-generated network;

[0098] The pre-generated network consists of 12 gated convolutional layers, with the thirteenth layer being an upsampling layer, the fourteenth layer being a gated convolutional layer, the fifteenth layer being an upsampling layer, followed by two consecutive gated convolutional layers, and the eighteenth layer being an activation layer.

[0099] The first convolutional layer has a 5×5 kernel size and a stride of 1. The second and fourth convolutional layers have a stride of 2, and the remaining convolutional layers all have a 3×3 kernel size and a stride of 1. The seventh to tenth gated convolutional layers use dilated convolution. The eighteenth layer uses Tanh activation.

[0100] Step 2.2, construct a refined generative network;

[0101] The last eight layers of the refined generative network have the same structure as the last eight layers of the pre-generative network. Layers one through ten are two parallel branches.

[0102] Branch 1 is the same as the first ten layers of the pre-generated network.

[0103] Branch 2 has one less layer than Branch 1. Its first five layers and last two layers have the same number of layers as the corresponding first five layers and last two layers of the pre-generated network. The sixth layer adds the ReLU activation function, and the seventh layer adds a content attention mechanism.

[0104] The outputs of branch one and branch two are concatenated.

[0105] Step 2.3: Connect the pre-generated network and the fine-grained generation network in series;

[0106] Step 3, train the gated convolutional object reconstruction network:

[0107] The training set is input into a gated convolutional object reconstruction network. The optimizer is Adam with a learning rate of 0.0001, and the optimizer parameters beta1 and beta2 are 0.5 and 0.999, respectively. The parameters of each layer in the gated convolutional object reconstruction network are iteratively updated using gradient descent until the total cost function converges, resulting in the trained gated convolutional object reconstruction network.

[0108] The input to the training set is:

[0109] Input=Concat(inI,M,guide);

[0110] The concatenated components in the channel dimension are the missing image inI, the random overlay mask M, and the target contour mask guide. InI can be further represented as:

[0111] inI=I·(1-M / 255)+guide / 255·M / 255;

[0112] First, a random overlay mask is used to remove a portion of the original remote sensing image, and then the portion where the target contour mask intersects with the random overlay mask is added.

[0113] Step 4: Use the trained semantic segmentation network to detect the target contour and its location:

[0114] Use a semantic segmentation network that can detect hidden regions to detect modified parts of the image.

[0115] This step can use a typical semantic segmentation network, but its output must be the contour corresponding to the hidden region.

[0116] Step 5: Generate the remote sensing target area mask and contour mask:

[0117] Based on the mask generation method, this can be divided into the following two cases:

[0118] 1) For the semantic segmentation dataset labels, the random mask is dilated with a kernel of size 12×12, with a dilation count of 1, to obtain the region mask. For the remote sensing target contour mask, the labels need to be extracted for contours, and a line thickness of 1 is used in the contour reconstruction process to obtain the remote sensing target contour mask.

[0119] This situation corresponds to the ideal state, where the original outline of the target is already available, and the tampered area can be inferred from the original outline, thereby obtaining the corresponding area mask.

[0120] 2) For the labels obtained from the semantic segmentation network, these labels serve as region masks. Then, an erosion operation is performed using a 12x12 kernel, with one erosion iteration, to obtain the specific target region. For the remote sensing target contour mask, contour extraction is performed on the specific target region, and a line thickness of 1 is used during contour reconstruction to obtain the remote sensing target contour mask.

[0121] This situation corresponds to the non-cooperative state, where an additional semantic segmentation network is needed to obtain the tampered region. This tampered region corresponds to the remote sensing target region mask, and the original target outline is then deduced from the obtained tampered region to obtain the corresponding outline mask.

[0122] Step 6, highly reconstruct the remotely sensed target:

[0123] Step 6.1: Use a remote sensing target region mask to cover the image that has already hidden the target, and input it together with the corresponding contour mask into the gated convolutional target reconstruction network to obtain a highly reconstructed result.

[0124] The purpose of this step is to remove tampered areas from the modified image and reduce the impact on the remote sensing target restoration process.

[0125] Step 6.2: The hidden target restoration results of the cropped remote sensing image are stitched together sequentially according to the sequence number to obtain the final restoration result.

[0126] The effects of this invention can be further illustrated by the following simulation experiments.

[0127] 1. Simulation experimental conditions:

[0128] The hardware platform for the simulation experiment of this invention is as follows: the processor is an Intel i9-10940X with a main frequency of 3.30GHz, the memory is 64G, the graphics card is an NVIDIA GeForce RTX 2080Ti with 12GB of video memory.

[0129] The software platform for the simulation experiment of this invention is: Windows 10 operating system and Python 3.6.

[0130] The simulation experiment of this invention uses 30 randomly selected images from the Massachusetts remote sensing building dataset. The images were taken in the Boston area. The original dataset contains 151 aerial images with a resolution of 1500×1500, with buildings in the foreground, including buildings of varying sizes and shapes.

[0131] 2. Simulation Experiment Content and Result Analysis:

[0132] The simulation experiment of this invention is carried out according to the following steps.

[0133] The image inpainting network employs a gated convolutional target restoration network.

[0134] The simulation experiment simulates the process from the original image to the hidden target, and then to the restoration of the hidden target.

[0135] Step A: Randomly selected images from the Massachusetts dataset were cropped to a size of 256×256 as the standard image size for this experiment, with a foreground ratio threshold of 10%. The final training set contained 3960 images, the validation set 144 images, and the test set 360 images.

[0136] Step B involves inputting images from the training set into the gated convolutional target reconstruction network. The training process employs a strategy of saving the model every 2000 epochs and conducting trials on the validation set every 2000 epochs.

[0137] Step C: Select the corresponding building outline label of the hidden target image, extract the building outline through simple outline extraction as the outline mask, and use the hidden target area corresponding to the hidden target image as the region mask.

[0138] Step D involves inputting the hidden target image, the random overlay mask, and the remote sensing target contour mask into a gated convolutional target reconstruction network to obtain the target reconstruction result.

[0139] The effects of the present invention will be further described below with reference to the simulation diagram in Figure 3.

[0140] Figure 3a These are remote sensing images from the test set. Figure 3b The result of semantic segmentation label inflation is used as a mask to cover the remote sensing target region. Figure 3c The result after hiding the target. Figure 3d The labels in the test set represent the location and shape of the target, i.e., the semantic segmentation labels, and are also the labels that the model expects the reconstruction results to match the distribution of the target in the labels. Figure 3e The result of extracting edges from the segmented label image is used as a mask for the contour of remote sensing targets. Since the edges are not obvious in the illustration when represented by 1 pixel, they are represented by 2 pixels here. Figure 3f The image is used as the input model. Here, it represents the removal of targets from the remote sensing image. The black area represents the area where the image is missing, and the white area represents the location and shape of the target that is to be restored. Figure 3g This is the result of restoring a hidden target under guidance.

[0141] Compare with the original remote sensing image: Figure 3a Hidden target restoration result image: Figure 3g As can be seen, the hidden target restoration effect of this invention closely approximates the original image. Given a remote sensing image containing a hidden target and its corresponding hidden region, a false target with the same shape and size as the original target can be generated effectively within the hidden region. In practice, the image obtained through de-sensitization and anonymization techniques can be obtained using an anomaly detection model, while the location, shape, and size of the hidden target require human judgment and manual delineation. In some cases, the de-sensitized image may retain some target outlines, which can be manually filled in to obtain the approximate outline of the hidden target. For targets with relatively fixed shapes, such as airplanes and buildings, this outline often closely resembles the original image. Therefore, this invention can be used to restore targets in these situations. Furthermore, this invention can also be used to forge remote sensing images to hide sensitive target information.

[0142] Due to the realism of the images it generates, the images generated by this invention can replace de-densified remote sensing images for remote sensing image analysis.

[0143] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for intelligent restoration of hidden targets in remote sensing images based on image inpainting, characterized in that: Includes the following steps: S1. Select remote sensing images and corresponding label images, crop and filter them to form a dataset, and obtain the remote sensing target area mask and remote sensing target contour mask of the dataset. Step S1 includes: S11. Randomly select at least 30 high-resolution remote sensing images and corresponding label images from the semantic segmentation dataset that have a relatively balanced ratio of foreground to background. S12. Crop each high-resolution image and its corresponding label image into... Pixel size; S13. Select images with a foreground pixel ratio of more than 10% in the cropped label images and the corresponding remote sensing images to form the dataset, which includes a training set, a validation set and a test set; S14. Dilate the random mask to obtain the remote sensing target region mask, and extract the contour of the cropped label image to obtain the remote sensing target contour mask. The random mask is a combination of an irregular mask formed by connecting random discrete points and a regular rectangular mask, and the random discrete points are generated by a random generation algorithm. S2. Construct a gated convolutional target reconstruction network, which includes a pre-generation network and a fine generation network in series. The pre-generation network generates remote sensing target mask information and outputs it to the fine generation network. The fine generation network uses an attention mechanism to dynamically learn the pixel distribution features of the target area and non-target area of ​​the uncovered remote sensing image and reconstruct the hidden target in the hidden area. Step S2 includes: S21. Construct the pre-generated network, which has an 18-layer structure. The pre-generated network includes 12 consecutive gated convolutional layers, upsampling layers, gated convolutional layers, upsampling layers, two consecutive gated convolutional layers, and an activation layer. The pre-generated network generates remote sensing target mask information and outputs it to the fine-generation network. S22. Construct the refined generation network, wherein the last eight layers of the refined generation network are identical to the last eight layers of the pre-generation network, and the first to tenth layers are two parallel branches. Branch 1 is the same as the first ten layers of the pre-generated network; Branch 2 has one less layer than Branch 1. The first five layers and the last two layers of Branch 2 correspond to the first five layers and the last two layers of the pre-generated network and have the same number of layers. The sixth layer of Branch 2 adds the ReLU activation function and the seventh layer adds a content attention mechanism. The outputs of branch one and branch two are concatenated using concatenation. The refined generative network uses an attention mechanism to dynamically learn the pixel distribution features of target regions and non-target regions in uncovered remote sensing images, and then restores the hidden targets in the hidden regions. S23. Connect the pre-generated network and the fine-generated network in series to obtain the gated convolutional target restoration network; S3. Train the gated convolutional target reconstruction network using the dataset, the remote sensing target region mask, and the remote sensing target contour mask. Iteratively update the parameters of each layer in the gated convolutional target reconstruction network using gradient descent until the total cost function converges, and obtain the trained gated convolutional target reconstruction network. S4. Use a semantic segmentation network to detect hidden regions in the remote sensing image to be restored in order to detect the modified parts of the remote sensing image to be restored and output a label image to be restored containing the outlines of the hidden regions. S5. The label image to be restored is the target region mask of the image to be restored. The target region mask of the image to be restored is eroded to obtain the specific target region, and then the contour is extracted to obtain the target contour mask of the remote sensing image to be restored. S6. Cover the remote sensing image to be restored with the target region mask of the remote sensing image to be restored, and input it together with the target contour mask of the remote sensing image to be restored into the trained gated convolutional target restoration network to obtain a highly restored result. After stitching, the restored remote sensing image is obtained.

2. The method for intelligent restoration of hidden targets in remote sensing images based on image inpainting according to claim 1, characterized in that: The random mask is used as The remote sensing target region mask is obtained by expanding the kernel of a certain size and expanding it 1 times. The outline of the cropped label image is extracted and the outline is reproduced by using lines of thickness 1 to obtain the remote sensing target outline mask.

3. The method for intelligent restoration of hidden targets in remote sensing images based on image inpainting according to claim 1, characterized in that: In step S21, the kernel size of the first gated convolutional layer... The stride of the first two gated convolutional layers is 1, the stride of the second and fourth gated convolutional layers is 2, and the kernel size of the remaining gated convolutional layers is 1. Both stride and step size are 1. The seventh to tenth gated convolutional layers use dilated convolution, and the activation function of the eighteenth layer is Tanh.

4. The method for intelligent restoration of hidden targets in remote sensing images based on image inpainting according to claim 1, characterized in that: In step S3, the training set is input into the gated convolutional target restoration network. First, the remote sensing target region mask is used to remove part of the remote sensing image, and then the part where the remote sensing target contour mask intersects with the random mask is added. Input to the training set for: ； For the missing images concatenated along the channel dimension, For random masking, For remote sensing target contour mask; 。 5. The method for intelligent restoration of hidden targets in remote sensing images based on image inpainting according to claim 1, characterized in that: In step S3, the optimizer is Adam, the learning rate is 0.0001, the optimizer parameters beta1 is 0.5, and beta2 is 0.

999.

6. The method for intelligent restoration of hidden targets in remote sensing images based on image inpainting according to claim 1, characterized in that: In step S5, the target region mask of the image to be restored is applied using... The target region is obtained by eroding a kernel of a certain size and performing erosion operation once. When extracting the contour of the target region, a line with a thickness of 1 is used in the contour reconstruction process.

Images