A deep learning-based remote sensing image sensitive target automatic hiding method

Abstract

The present application belongs to the technical field of geographic data desensitization, and particularly relates to a remote sensing image sensitive target automatic hiding method based on deep learning, which comprises the following steps: firstly, a YOLOX algorithm with a high recall parameter is used to identify sensitive targets in a specific scene, and a rectangular mask region with a buffer area is generated to achieve full coverage of the sensitive targets; then, an EdgeConnect algorithm is adopted to perform background completion processing on the generated mask region, and a two-stage conditional generative adversarial network model of edge generation and color texture supplement is used to construct the structural texture features of the missing area; finally, a contrast adjustment module is combined on the basis of the EdgeConnect method to improve the robustness of the completion result. Therefore, the present application solves the problem of poor sensitive target hiding effect caused by incomplete coverage of sensitive targets and their shadows and missing background completion structural features in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of geographic data desensitization technology, specifically relating to a method for automatically hiding sensitive targets in remote sensing images based on deep learning. Background Technology

[0002] With the continuous improvement of my country's capabilities in acquiring, processing, and distributing high-resolution remote sensing imagery, massive amounts of remote sensing imagery have become an indispensable strategic resource for economic and national defense construction. However, data management departments have consistently faced the challenge of sharing and maintaining confidentiality. Traditional manual or semi-automatic methods for hiding sensitive targets are labor-intensive and inefficient, failing to meet the timeliness requirements for the release of massive amounts of imagery. Currently, some public map platforms only provide low-level imagery for sensitive areas, while others do not perform sensitive target hiding processing on high-level imagery of the corresponding areas. Therefore, conducting research on automatic hiding methods for sensitive targets in remote sensing imagery is an urgent need to ensure image data security and safeguard national interests.

[0003] However, research on the automatic hiding of sensitive targets in remote sensing images is limited both domestically and internationally. In 2015, Li Binbin of Nanjing Normal University improved the sample texture synthesis method, proposed a sample matching synthesis method based on color and texture features, and constructed a secure decryption model for sensitive targets in images. This model relies on low-level features and adopts the traditional image completion mode of "search-match-copy," and its hiding effect and processing efficiency need to be improved. In recent years, deep learning technology has achieved good results in remote sensing image processing. For example, in target detection, high accuracy can be achieved through deep learning algorithms represented by convolutional neural networks; while in image generation, by combining generative adversarial networks with encoders and decoders, images with good visual effects and diverse features can be generated, meeting the needs of various generation tasks. For example, Lu Pengjie et al. from Wuhan University proposed a deep learning-based method for automatic hiding of sensitive targets in 2020. This method introduces deep learning technology, uses the Mask R-CNN semantic segmentation model to extract targets and output mask images, and then uses a two-stage Deepfill model for image completion processing to achieve pixel-level hiding of sensitive targets. This method introduces an attention mechanism and optimizes the mask image to obtain a reasonably effective result in hiding sensitive targets. However, using this method, on the one hand, the geometric shape of the image patch covering the sensitive target in the mask image is complex, and the mask image is prone to not completely covering the sensitive target (e.g., the edge of the sensitive target is not covered); on the other hand, for sensitive targets with shadows, the generated sensitive target detection box cannot completely cover the shadow of the sensitive target, resulting in the phenomenon that the shadow of the sensitive target cannot be covered. The above two aspects will lead to the subsequent remote sensing image after hiding the sensitive target still showing the part of the sensitive target or generating artifacts, resulting in poor hiding effect. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic hiding method for sensitive targets in remote sensing images based on deep learning, in order to solve the problem of poor hiding effect of sensitive targets due to incomplete coverage of sensitive targets and their shadows in the prior art.

[0005] To solve the above-mentioned technical problems, the technical solution provided by this invention and the corresponding beneficial effects of the technical solution are as follows:

[0006] The present invention provides an automatic hiding method for sensitive targets in remote sensing images based on deep learning, comprising the following steps:

[0007] 1) Input the remote sensing image containing the sensitive target into the sensitive target detection model to identify the sensitive target and generate a sensitive target detection box; expand the area enclosed by the sensitive target detection box by a buffer of N pixels to obtain the mask area, where N>1;

[0008] 2) Mask the corresponding mask area in the remote sensing image to generate a mask image, and input the mask image into the background completion model to perform background completion processing on the corresponding mask area in the remote sensing image.

[0009] 3) Based on the background-completed remote sensing image, output the remote sensing image that hides sensitive targets.

[0010] The beneficial effects of the above technical solution are as follows: This invention proposes generating a mask region based on a target detection box, using a sensitive target detection model to identify sensitive targets and generate a sensitive target detection box that can completely cover the sensitive target. To ensure complete coverage of the sensitive target and its shadow, a buffer of N pixels is added outside the target detection box to obtain the mask region, thereby guaranteeing the complete coverage of the sensitive target and its shadow. Then, the corresponding mask region in the remote sensing image is masked to generate a mask image. Since the mask region can completely cover the sensitive target and its shadow, complete coverage of the sensitive target is guaranteed. Finally, a background completion model performs background completion processing on the mask region based on the mask image. Based on the remote sensing image after background completion processing, a remote sensing image with the hidden sensitive target is obtained. This solves the problem of poor hiding effect of sensitive targets due to incomplete coverage of sensitive targets and their shadows in the prior art. The method of this invention can guarantee the stability of the hidden target.

[0011] Furthermore, in step 2), before masking the corresponding masked area in the remote sensing image, the remote sensing image needs to be contrast-enhanced; and before outputting the remote sensing image that hides the sensitive target, the contrast needs to be reduced.

[0012] The beneficial effects of the above technical solution are as follows: enhancing contrast improves the image, highlighting useful information while removing or weakening useless information, thus enhancing the image's features. This is beneficial for the background completion model to extract features and reduces the color difference between the completed image and the surrounding environment. To avoid contrast differences caused by contrast enhancement in the final output image, this invention further reduces the contrast of the background-completed image, thereby improving the overall visual effect of target concealment.

[0013] Furthermore, the contrast adjustment formula used when performing contrast enhancement and contrast reduction processing is as follows:

[0014] M'=L*(1-α)+M*α

[0015] Where M is the intensity value of a certain channel in the original image, M' is the intensity value of a certain channel after adjustment, L is the current average gray value of the entire image, and α is the contrast adjustment coefficient. When contrast enhancement is performed, α is the contrast enhancement coefficient, and when contrast reduction is performed, α is the contrast restoration coefficient.

[0016] Furthermore, the contrast adjustment coefficient is determined based on the optimal value obtained from the peak signal-to-noise ratio (PSNR), and the PSNR formula is as follows:

[0017]

[0018] Among them, MAX I The value represents the number of bits in the image, and MSE is the mean square error of the grayscale values ​​of the two images.

[0019] Further, step 3) of outputting the remote sensing image with hidden sensitive targets based on the background-completed remote sensing image includes: according to the mask area, before outputting the remote sensing image with hidden sensitive targets, it is necessary to extract the image within the mask area and overlay the extracted image onto the corresponding position in the remote sensing image, thereby obtaining the remote sensing image with hidden sensitive targets.

[0020] The beneficial effects of the above technical solution are as follows: To reduce information loss in non-masked areas due to contrast adjustment, the image of the masked area with restored contrast is then stitched onto the original remote sensing image to obtain a remote sensing image with complete information. This invention extracts the image of the repaired area from the background-completed image and overlays it onto the corresponding position in the remote sensing image, thereby making the remote sensing image hiding sensitive targets appear more effective.

[0021] Furthermore, the background completion model is an EdgeConnect network; the EdgeConnect network includes a generator and a discriminator for edge prediction, and also includes a generator and a discriminator for color and texture completion; the last layer activation function of the generator and discriminator is the Sigmoid function, and the activation functions of other layers are the ReLU function or the LeakyReLU function.

[0022] The beneficial effects of the above technical solution are as follows: Existing technologies lack structural features during background completion, resulting in a lack of structural texture in the generated background image. This leads to a certain degree of blurring in remote sensing images after sensitive targets are hidden. Therefore, this invention introduces the EdgeConnect algorithm. This algorithm utilizes the prior conditions of edge feature extraction and completion. EdgeConnect first extracts the edge features of the incomplete image and generates edges for the incomplete portion. Then, it supplements the color and texture based on the completed edge feature image and the incomplete image, thereby generating a completed image with rich structural features and good visual effects. To make the output smoother, the last activation function of both the generator and discriminator is the Sigmoid function, while the activation functions of other layers are the ReLU function or the LeakyReLU function.

[0023] Furthermore, the loss functions used for edge prediction and color texture completion are as follows:

[0024]

[0025]

[0026] in To combat the losses, For feature matching loss, For the loss of style, In order to perceive loss, for Loss, λ adv,1 , λ FM , λ adv,2 , λ p and λ s These are the weights of the corresponding loss functions. For edge prediction loss, To compensate for the loss of color and texture.

[0027] Furthermore, in order to fully cover the sensitive target, the sensitive target detection box is a rectangular box, and the mask area is a rectangular area.

[0028] Furthermore, the sensitive target detection model is YOLOX.

[0029] The beneficial effects of the above technical solution are as follows: The present invention adopts the YOLOX algorithm, which uses feature extraction through convolutional neural networks to adjust the coordinates and width and height of the candidate box center point according to the geometric characteristics of sensitive targets, thereby realizing the automatic detection of multi-scale and multi-category targets and is applicable to a wide range of sensitive targets.

[0030] Furthermore, N can be 4, 5, 6, or 7. Attached Figure Description

[0031] Figure 1 This is a flowchart of a method for automatically hiding sensitive targets in remote sensing images based on deep learning, according to an embodiment of the present invention.

[0032] Figure 2 This is a diagram of the YOLOX network structure in an embodiment of the method of the present invention;

[0033] Figure 3 This is a diagram of the EdgeConnect network structure according to an embodiment of the method of the present invention;

[0034] Figure 4 This is a diagram of the EdgeConnect algorithm prediction structure with a contrast optimization module in an embodiment of the method of the present invention.

[0035] Figure 5 This is a comparison diagram of the first target hiding results in an embodiment of the method of the present invention;

[0036] Figure 6 This is a comparison diagram of the second target hiding results in an embodiment of the method of the present invention. Detailed Implementation

[0037] This invention focuses on designing a deep learning-based automatic hiding method for sensitive targets in remote sensing images, addressing both mask image generation and structural feature restoration. The method utilizes a target detection model to automatically identify sensitive targets and draw corresponding detection boxes. A buffer of N pixels is then added outside the area defined by these detection boxes to generate a mask region. Subsequently, an edge inference image completion algorithm combined with a contrast module is used to complete the background of the mask region, improving the hiding visual effect from the perspectives of color difference and image structure while maintaining hiding stability. Specifically, firstly, the YOLOX algorithm with high recall parameters is used to identify sensitive targets in a specific scene and generate a rectangular mask region with a buffer area to achieve full coverage of sensitive targets. Then, the EdgeConnect algorithm is used to complete the background of the generated mask region, utilizing a two-stage conditional generative adversarial network model of edge generation and color texture completion to construct structural texture features of the missing areas. Finally, a contrast adjustment module is combined with the EdgeConnect method to improve the robustness of the completion result.

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0039] Method Implementation Examples:

[0040] An embodiment of the present invention provides an automatic hiding method for sensitive targets in remote sensing images based on deep learning, as follows: Figure 1 As shown, the process includes the following three steps: 1) Inputting the remote sensing image containing the sensitive target into the sensitive target detection model to identify the sensitive target and generate a sensitive target detection box; expanding the area enclosed by the sensitive target detection box by N pixels to obtain a mask region, where N > 1; 2) Masking the corresponding mask region in the remote sensing image to generate a mask image, and inputting the mask image into the background completion model to perform background completion processing on the corresponding mask region in the remote sensing image; 3) Outputting the remote sensing image with the sensitive target hidden based on the background-completed remote sensing image; The sensitive target detection model is trained based on a remote sensing image dataset labeled with sensitive targets; The background completion model is trained based on the incomplete image (mask image) and the corresponding remote sensing image dataset. Preferably, the sensitive target detection model is YOLOX; preferably, the background completion model is the EdgeConnect network. The specific flow of the above three steps is as follows:

[0041] Step 1: Input the remote sensing image containing sensitive targets into the sensitive target detection model. The YOLOX target detection algorithm automatically detects sensitive targets within the scene, generating rectangular detection boxes. Then, a rectangular detection box region is generated based on these boxes, and a buffer of 4, 5, 6, or 7 pixels is added outside the rectangular detection box region to obtain the final rectangular mask region. The number of pixels added outside the rectangular detection box region is adjusted according to the actual situation.

[0042] This invention employs the YOLOX algorithm, which detects targets using bounding boxes. This algorithm utilizes feature extraction from a convolutional neural network to adjust the center point coordinates and dimensions of candidate boxes based on the geometric characteristics of sensitive targets, achieving automatic detection of multi-scale, multi-class targets. The YOLOX algorithm uses YOLOv3 with an SPP module as its base network, which can combine a feature pyramid structure to detect targets at different scales. YOLOX adds stronger Mosaic and Mixup data augmentation methods to YOLOv3, replaces the coupling heads at different scales with decoupling heads, adopts an anchorless box design, and utilizes the SimOTA method for dynamic label allocation. The YOLOX network model structure adopted in this invention is as follows: Figure 2 As shown, the feature extraction module refers to different forms of convolutional layers, SiLU activation functions, and residual modules.

[0043] Given the characteristics of non-overlapping sensitive targets and single target categories in remote sensing images, the Mixup method is not effective in improving network performance and increases training costs. Therefore, this invention does not use this data augmentation method during training.

[0044] When the generated rectangular detection box cannot completely cover the target, background completion output will exhibit artifacts. To avoid this problem, this invention sets a buffer outside the detection area to ensure the complete coverage of the target by the mask image. Typically, the buffer size is set to 5 pixels.

[0045] Step 2: Perform contrast enhancement processing on the remote sensing image to obtain a contrast-enhanced remote sensing image.

[0046] Image enhancement can highlight useful information in an image while removing or weakening useless information, making the processed image more suitable for human visual characteristics or easier for machine recognition and subsequent processing. Contrast adjustment is a commonly used image enhancement method. This invention, while maintaining brightness, alters the tonal differences between bright and dark areas of the image, thus enhancing image contrast and improving image features. This facilitates further feature extraction by the EdgeConnect network and reduces color differences between the generated image and its surrounding environment.

[0047] The contrast adjustment formula is shown in equation (1). In the formula, M is the intensity value of a certain channel in the original image, M' is the intensity value of a certain channel after adjustment, L is the current average gray value of the entire image, and α is the contrast adjustment coefficient. When contrast enhancement processing is performed, α is the contrast enhancement coefficient, and when contrast reduction processing is performed, α is the contrast restoration coefficient.

[0048] M'=L*(1-α)+M*α (1)

[0049] This invention finds the optimal contrast adjustment coefficient based on the PSNR (Peak Signal to Noise Ratio) index, including the contrast adjustment coefficient when contrast is enhanced and the contrast restoration coefficient when contrast is reduced. The PSNR formula is shown in equation (2), where MAX I The image bit depth is represented by α, and MSE is the mean square error of the grayscale values ​​of the two images. A suitable contrast parameter pair (α) is obtained through formula (2). i ,β i ), to perform contrast enhancement and restoration processing on the image to be completed, α i β is the contrast enhancement factor. i This is the contrast restoration coefficient.

[0050]

[0051] Step 3: Based on the mask area, the corresponding mask area in the contrast-enhanced remote sensing image is masked to generate a mask image. The mask image is then input into the background completion model, and the EdgeConnect image inpainting algorithm is used to perform two stages of processing: edge feature completion and color and texture completion. The image with background completion is then output.

[0052] Existing deep learning image completion algorithms focus on image texture and tone generation, but lack the ability to shape the image's structure, often resulting in blurry and unrealistic images. Therefore, this invention introduces the EdgeConnect algorithm, which utilizes prior conditions for edge feature extraction and completion to generate completed images with rich structural features and good visual effects. EdgeConnect is based on the CGAN (Conditional Generative Adversarial Networks) framework. First, it uses the Canny operator to extract edge features from the image to be completed and generates edges for the missing parts. Then, it uses the completed edge feature image and the image to be completed to supplement color and texture.

[0053] The EdgeConnect network consists of an edge prediction generator G1, a discriminator D1, a color and texture supplementation generator G2, and a discriminator D2. The generator primarily uses an encoder, decoder, and a residual module with dilated convolutions. In G1, spectral normalization effectively ensures Lipschitz continuity, thus suppressing abrupt changes in parameters and gradients. In G1 and G2, instance normalization accelerates model convergence while maintaining image independence. The discriminator borrows from the PatchGAN discriminator structure, using a fully convolutional neural network to distinguish between generated and real images, contributing to the generation of high-resolution, high-detail-preserving images. In the last layer of both the generator and discriminator, the activation function is converted from ReLU or LeakyReLU to a Sigmoid function, respectively, resulting in a smoother output. The specific network structure is as follows: Figure 3 As shown.

[0054] To ensure that the generated image achieves clarity, realistic texture, and similar style, different loss functions are used in the edge prediction and color / texture completion stages, as shown in equations (3) and (4). To combat the losses, For feature matching loss, For the loss of style, In order to perceive loss, for Loss, where λ is the weight of the corresponding loss function, i.e., λ adv,1 , λ FM , λ adv,2 , λ p and λ s These are the weights of the corresponding loss functions. For edge prediction loss, To compensate for the loss of color and texture.

[0055]

[0056]

[0057] Step 4: Perform contrast reduction processing on the image after background completion to obtain the remote sensing image with reduced contrast. The contrast reduction coefficient is set based on the image signal-to-noise ratio being the highest and the visual effect being the best after two contrast adjustments.

[0058] like Figure 4 As shown, the structure predicted by the contrast optimization module combined with the EdgeConnect algorithm is illustrated. To avoid contrast differences in the final output image due to contrast adjustment, this invention performs contrast restoration processing (contrast reduction processing) on ​​the output image. During contrast reduction processing, the contrast adjustment formula still uses formula (1), and the contrast adjustment coefficient is obtained using formula (2) to calculate the contrast parameter pair (α). i ,β i The contrast restoration coefficient in formula (1) can be obtained by replacing α with the contrast restoration coefficient when the contrast is reduced.

[0059] Step 5: Based on the masked area, extract the data from the remote sensing image after the contrast has been reduced.

[0060] Step 6: To reduce information loss in non-masked areas due to contrast adjustment, the contrast-restored masked areas are then stitched back onto the original remote sensing image. The extracted image is then overlaid onto the corresponding position in the remote sensing image to obtain a remote sensing image that hides the sensitive target. The de-masked image of the contrast-restored area is then overlaid onto the corresponding area of ​​the original remote sensing image to obtain a remote sensing image with sensitive target masking and background completion.

[0061] In this embodiment, step four, contrast reduction processing of the background-completed image, is performed first to obtain a contrast-reduced remote sensing image. Then, step five, based on the mask region, is performed to perform a cutout operation on the contrast-reduced remote sensing image. In other feasible embodiments, the average brightness of the entire image can be obtained first; then, based on the mask region, a cutout operation can be performed on the background-completed image to obtain the completed image within the mask region; then, based on the obtained average brightness of the entire image, contrast reduction processing is performed on the completed portion of the image within the mask region to obtain the contrast-reduced completed portion of the image overlaid on the remote sensing image.

[0062] like Figure 5 , Figure 6 As shown, the target hiding results for two images are presented. Figure 5 and Figure 6 The leftmost local bounding box in the second row is the original remote sensing image; the middle local bounding box in the second row is the mask image; and the rightmost local bounding box in the second row is the target hiding result of this invention. The leftmost local bounding boxes in the first row and the leftmost local bounding boxes in the third row are two local detail images from the original remote sensing image, respectively. The rightmost local bounding boxes in the first row and the rightmost local bounding boxes in the third row are schematic diagrams of the local target hiding results corresponding to the leftmost local bounding boxes in the first row and the leftmost local bounding boxes in the third row, respectively. It can be seen that the method proposed in this invention achieves automatic target hiding based on accurate detection of sensitive targets. The color difference between the completed result and the background area is small, and the background structural features of the mask area are restored, achieving the expected goal of the hiding process.

[0063] To improve the concealment effect of sensitive targets, this invention focuses on designing a deep learning-based automatic concealment method for sensitive targets in remote sensing images, addressing both mask image generation and structural feature recovery. The main innovations of this invention include: 1) proposing a mask image generation method based on target detection bounding boxes, solving the problem of complex geometric structures and incomplete coverage in mask images generated for complex sensitive targets, thus ensuring the stability of the concealed target; 2) introducing an edge inference image completion algorithm, and designing a contrast adjustment module based on this algorithm, solving the problems of lack of structural image features and significant color difference between the background completion image and the surrounding area, thereby improving the visual effect of target concealment.

Claims

1. A method for automatically hiding sensitive targets in remote sensing images based on deep learning, characterized in that: Includes the following steps: 1) Input remote sensing images containing sensitive targets into the sensitive target detection model to identify the sensitive targets and generate sensitive target detection boxes; Expand the buffer area by N pixels outside the range defined by the sensitive target detection box to obtain the mask area, where N > 1; 2) Perform contrast enhancement processing on the remote sensing image, and then mask the corresponding mask area in the contrast-enhanced remote sensing image to generate a mask image. Input the mask image into the background completion model to perform background completion processing on the corresponding mask area in the remote sensing image. The contrast of the remote sensing image after background completion is reduced. The background completion model employs a two-stage conditional generative adversarial network model based on edge generation and color texture supplementation; the contrast adjustment formula used in contrast enhancement and contrast reduction processing is as follows: M is the intensity value of a certain channel in the original image. The adjusted intensity value of a certain channel, where L is the current average gray value of the entire image. This is the contrast adjustment factor, used when performing contrast enhancement or reduction processing. Corresponding to the enhanced contrast coefficient , restore contrast coefficient The optimal contrast parameters are determined with the goal of maximizing the peak signal-to-noise ratio (PSNR) of the image after two contrast adjustments. ; 3) Based on the mask area, the contrast-reduced remote sensing image is extracted, and the extracted image is overlaid on the corresponding position in the original remote sensing image to obtain the remote sensing image that hides the sensitive target and output it.

2. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 1, characterized in that: The formula for Peak Signal-to-Noise Ratio (PSNR) is: in, Indicates the image bit depth. denoted as the mean square error of the grayscale values ​​of the two images.

3. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 1, characterized in that: The background completion model is the EdgeConnect network; the EdgeConnect network includes a generator and a discriminator for edge prediction, as well as a generator and a discriminator for color and texture completion.

4. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 3, characterized in that: The generator and discriminator both use the Sigmod function as the activation function in the last layer, while the other layers use either the ReLU function or the LeakyReLU function.

5. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 3, characterized in that: The loss functions used for edge prediction and color texture completion are as follows: in, , To combat the losses, For feature matching loss, For the loss of style, In order to perceive loss, for loss, , , , , and These are the weights of the corresponding loss functions. For edge prediction loss, To compensate for the loss of color and texture.

6. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 1, characterized in that: The sensitive target detection box is a rectangle.

7. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to claim 6, characterized in that: The mask area is a rectangular area.

8. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to any one of claims 1-7, characterized in that: The sensitive target detection model is YOLOX.

9. The method for automatic hiding of sensitive targets in remote sensing images based on deep learning according to any one of claims 1-7, characterized in that: N can be 4, 5, 6, or 7.

Images