Remote sensing image sensitive target elimination method and device, equipment and storage medium

Abstract

The invention provides a remote sensing image sensitive target elimination method and device, equipment and a storage medium. Relates to the technical field of remote sensing image processing. The method comprises the following steps: performing block processing on a high-resolution large-size image to obtain a region of interest containing a sensitive target, and converting geographic coordinates of the region of interest into pixel coordinates; performing target detection on each target region by using the trained instance segmentation model based on the target regions, generating a corresponding segmentation mask, dynamically adjusting the expansion degree of the segmentation mask according to the target area, and determining a mask region; and generating filling content, filling the filling content into the mask region to obtain an image after the target is eliminated, and splicing the image after the target is eliminated to the original image to obtain a spliced image. The method meets the requirements of high-efficiency, accurate and traceless elimination of large-scale and high-resolution remote sensing image sensitive targets, and provides a reliable technical scheme for safe sharing of remote sensing image data.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method, device, equipment and storage medium for eliminating sensitive targets in remote sensing images. Background Art

[0002] With the rapid development of remote sensing technology and satellite observation technology, the acquisition of high-resolution remote sensing images has become increasingly convenient. These images have shown great application potential in many fields due to their rich spatial texture features and semantic information. Large-scale, high-resolution satellite remote sensing images have wide spatial coverage, rich details, and accurate geographic information. However, these images often contain sensitive information, such as national key facilities or content information involving citizens' privacy. Remote sensing image data containing sensitive information should be declassified and desensitized according to national standards to meet the needs of public sharing. Therefore, the problem of removing sensitive information from large-scale, high-resolution remote sensing images has gradually become a research focus. How to effectively eliminate sensitive targets while retaining the application value of images has become an urgent problem that needs to be solved.

[0003] In the field of sensitive target removal from high-resolution remote sensing images, traditional methods mainly rely on image processing and machine learning techniques. These methods have played a certain role in early research and can achieve the task of removing sensitive targets to a certain extent. However, with the increasing complexity of high-resolution image application scenarios, their limitations have gradually become apparent. High-resolution images contain rich details and complex scene information. When processing such images, traditional methods often have difficulty accurately identifying the edges of targets, resulting in severe edge blurring. At the same time, due to the lack of effective extraction capabilities for complex texture features, key details are easily lost in the process of removing targets, which greatly reduces the quality of the processed images and makes it difficult to strike a balance between removal accuracy and image quality.

[0004] In recent years, the remarkable progress of deep learning technology in the field of computer vision has brought new opportunities to the field of object detection and image restoration. Currently, commonly used object detection models include the YOLO series, Faster R-CNN, and Transformer-based object detection networks. These models can effectively extract complex texture features in images, thereby greatly improving detection accuracy and efficiency. For example, the paper "CAO Changqing, WANG Bo, ZHANG Wenrui, et al. Animproved faster R-CNN for small object detection[J]. Ieee Access, 2019,7:106838-106846." proposed an improved Faster R-CNN algorithm, which significantly outperformed traditional models in small object detection by optimizing the IoU loss function, enhancing RoI pooling operations, and multi-scale feature fusion. The paper "JIE Xue, ZHENG Yongguo, DONG-YE Changlei, et al. Improved YOLOv5 network method for remote sensing image-based ground objects recognition[J]. Soft Computing, 2022, 26(20): 10879-10889." significantly improves the recognition accuracy of large-scale targets in high-resolution images by improving the backbone network and attention mechanism of the YOLOv5 model. However, with the continuous optimization of the model structure, the increase in computational complexity limits the real-time performance of the algorithm when processing large-scale high-resolution images, which is particularly significant in practical applications. Therefore, how to effectively reduce the computational cost while ensuring detection accuracy and achieve efficient and real-time sensitive target detection and elimination is a difficult problem that needs to be overcome in current research.

[0005] The application of deep learning in the field of image restoration has also made significant progress. Models such as Generative adversarial network (GAN), Convolutional Neural Networks (CNN), and Transformer have shown great potential in improving the quality of image restoration and processing complex scenes. Take the context encoder proposed in the paper "Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting. Proceedings of the IEEE conference on computer vision and pattern recognition [C], 2016: 2536-2544." as an example. By generating the content of any region in the image, it learns to capture the appearance and semantic structure of the image. It is not only used for restoration tasks, but also as an effective pre-training method for CNN models. The deep generative model with contextual attention mechanism proposed in the paper "YU Jiahui, ZHE Lin, YANG Jimei, SHEN Xiaohui, et al. Huang; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition[C], 2018:5505-5514" significantly improved the quality of restoration and training efficiency. However, although these restoration models perform well in general image restoration tasks, they have exposed many problems such as computational efficiency and detail reconstruction when faced with the task of removing sensitive targets in high-resolution remote sensing images.

[0006] In summary, existing methods usually focus on improving the accuracy of target recognition in deep learning model improvement, but in the practical application of large-scale high-resolution remote sensing images, real-time performance and computational efficiency are also crucial, and achieving a balance between recognition accuracy and efficiency becomes a key consideration. In addition, current research rarely combines target recognition with image restoration for sensitive target elimination, while this combination is of great significance in ensuring image availability and security. Summary of the invention

[0007] The present application provides a method, device, equipment and storage medium for eliminating sensitive targets in remote sensing images, aiming to accurately and efficiently detect and remove sensitive content in images. The present application first pre-processes the image in blocks to reduce the computational burden and improve processing efficiency; then uses a deep learning model to identify sensitive targets in the image, and generates a corresponding mask to cover the sensitive area. To ensure that the mask accurately covers the target and reduces interference with the background, the algorithm uses an adaptive expansion strategy to adjust the mask. In response to missed detections and false detections in detection, a manual review link is introduced to ensure that all sensitive targets are accurately marked. After manual confirmation, the elimination operation of sensitive targets is automatically performed, and the processed image blocks are seamlessly spliced ​​back to the original image. This method combines automated detection with manual assisted detection to improve the accuracy and efficiency of sensitive target elimination, providing a feasible solution for remote sensing image security applications.

[0008] In a first aspect, the present application provides a method for eliminating sensitive targets in remote sensing images, comprising: The high-resolution large-size image is processed in blocks to obtain an area of ​​interest containing sensitive targets, and the geographic coordinates of the area of ​​interest are converted into pixel coordinates; Based on the target area, the trained instance segmentation model is used to perform target detection on each target area, and a corresponding segmentation mask is generated. The expansion degree of the segmentation mask is dynamically adjusted according to the target area to determine the mask area; wherein the target area is an area containing sensitive targets manually drawn in the region of interest; Generate a filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

[0009] In a possible design, the geographic coordinates of the region of interest are converted into pixel coordinates by the following formula: ; In the formula, x and y Represent the horizontal and vertical coordinates of the image pixels respectively; X and Y Respectively represent the geographic abscissa and ordinate of the area of ​​interest; a , b , c , d , e and f Both represent affine transformation parameters.

[0010] In one possible design, the trained instance segmentation model.

[0011] In one possible design, the expansion degree of the segmentation mask is dynamically adjusted according to the target area, and the calculation process for determining the mask area is expressed as: ; In the formula, is the expanded core size; area is the mask area; 500 is the scaling factor used to adjust the degree of expansion; max is the maximum function; min is the minimum function; int is the closest distance to the specified edge set.

[0012] In a possible design, generating a filling content and filling it into the mask area to obtain an image after the target is eliminated, and splicing the image after the target is eliminated to the original image to obtain a spliced ​​image, including: The filling content is generated by the following formula and filled into the mask area to obtain the image after the target is eliminated: ; In the formula, The filling content generated by the generator, is the original image; The image after the target is eliminated; is the mask area; The image after the target is eliminated is spliced ​​to the original image using the following formula to obtain a spliced ​​image: ; In the formula, is the stitched image; is the mask of the segmented region; The number of the partition area.

[0013] In a possible design, after obtaining the spliced ​​image, the method further includes: The spliced ​​image is evaluated using evaluation indicators; the evaluation indicators include edge deviation, Hausdorff distance and IoU; wherein the edge deviation and Hausdorff distance are used to evaluate the matching degree between the mask edge and the real target boundary, and the IoU is used to evaluate the consistency of regional overlap; The edge deviation is calculated by the following formula: ; In the formula, is the total number of true edge pixels; and Represent the pixel positions of the true edge and mask edge respectively; express and The Euclidean distance between them; Dedge represents the edge deviation; Indicates the true target edge; The Hausdorff distance is calculated using the following formula: ; In the formula, sup represents the maximum distance in the specified edge set; inf represents the shortest distance to the specified edge set. Represents the true target edge Generate mask edges Hausdorff distance; IoU is calculated using the following formula: ; In the formula, Represents the prediction mask and the true mask The number of intersection pixels; Represents the prediction mask and the true mask The number of pixels in the union.

[0014] In a second aspect, the present application provides a remote sensing image sensitive target elimination device, the device comprising: An image segmentation module is configured to segment high-resolution large-size images to obtain an area of ​​interest containing sensitive targets, and convert geographic coordinates of the area of ​​interest into pixel coordinates; The detection and correction module is configured to perform target detection on each target area based on the target area using the trained instance segmentation model, generate a corresponding segmentation mask, dynamically adjust the expansion degree of the segmentation mask according to the target area, and determine the mask area; wherein the target area is an area containing sensitive targets manually drawn in the region of interest; The elimination and splicing module is configured to generate filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the remote sensing image sensitive target elimination method described in the first aspect and various possible designs of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method for eliminating sensitive targets in remote sensing images as described in the first aspect and various possible designs of the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for eliminating sensitive targets in remote sensing images as described in the first aspect and various possible designs of the first aspect.

[0018] The remote sensing image sensitive target elimination method, device, equipment and storage medium provided in this application have at least the following beneficial effects: This application demonstrates excellent detection and elimination performance in complex scenarios, and can effectively reduce residual and unnatural edge transition phenomena in target elimination, ensuring the integrity and authenticity of the image. This application provides a reliable technical solution for the protection of sensitive information in high-resolution remote sensing images, and is suitable for a variety of practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 A flowchart of a method for eliminating sensitive targets in remote sensing images provided in an embodiment of the present application; Figure 2 A comparison diagram of mask expansion provided in an embodiment of the present application; Figure 3 AOT-GAN model structure diagram provided for the embodiment of the present application; Figure 4 A comparison chart of target elimination effects provided in the embodiments of the present application; Figure 5 The processing efficiency and resource consumption experimental result diagram provided for the embodiment of the present application; wherein, (a), 1.5G image processing resource occupancy (5000 sub-frames); (b), 1.5G image processing resource occupancy (10000 sub-frames); (c), 5G image processing resource occupancy (5000 sub-frames); (d), 5G image processing resource occupancy (10000 sub-frames); Figure 6 The processing efficiency and resource consumption experimental result diagram provided for the embodiment of the present application; wherein, (a), 10G image processing resource occupancy (5000 sub-frames); (b), 10G image processing resource occupancy (10000 sub-frames); (c), 23G image processing resource occupancy (5000 sub-frames); (d), 23G image processing resource occupancy (10000 sub-frames); Figure 7 A schematic diagram of the structure of a remote sensing image sensitive target elimination device provided in an embodiment of the present application.

[0021] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0024] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0025] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0026] At present, there is still a lack of efficient, accurate and intelligent methods for eliminating sensitive targets in large-scale, high-resolution remote sensing images. Traditional methods are difficult to adapt to the characteristics of large data volume and multi-scale targets of remote sensing images due to cumbersome operations such as cropping, splicing, and calibration, and the elimination effect is poor. To this end, an embodiment of the present application provides a method for eliminating sensitive targets in remote sensing images. The method first divides the image into blocks to locate the area where the sensitive target is located, then uses the YOLOv8 instance segmentation model to identify the sensitive target and generate a mask, and proposes a mask adaptive expansion method to optimize the mask quality. Finally, the AOT-GAN model is used to fill the mask area with background consistency to achieve seamless elimination of sensitive targets. Experimental results show that the algorithm meets the requirements of efficient, accurate and seamless elimination of sensitive targets in large-scale, high-resolution remote sensing images, and provides a reliable technical solution for the secure sharing of remote sensing image data.

[0027] Figure 1 A flowchart of a remote sensing image sensitive target elimination method provided in an embodiment of the present application. Figure 1 As shown in the figure, the remote sensing image sensitive target elimination method aims to effectively process sensitive targets in large-scale high-resolution remote sensing images, taking into account both detection accuracy and processing efficiency to meet practical application requirements. To achieve this goal, the algorithm designs modules such as image block preprocessing, manual auxiliary mask correction, sensitive target identification and elimination to ensure accurate elimination of sensitive targets.

[0028] First, the detection process is optimized through image segmentation preprocessing. Before detection, for extremely large remote sensing images, the algorithm will select the region of interest (ROI) and perform regional segmentation to narrow the processing scope, thereby effectively reducing the amount of calculation and improving processing efficiency. In the target recognition stage, the algorithm uses the YOLOv8 instance segmentation model to automatically detect sensitive target areas.

[0029] The quality of target removal is highly dependent on the quality of the mask generated by instance segmentation. If the mask coverage is insufficient, the exposure of the edge of the target area will cause the target to remain after removal or the transition will be abrupt, affecting the removal effect; if the mask area is too large, it may block the background, affecting the realism of the area after removal. To this end, the algorithm performs adaptive expansion processing on the mask area to ensure full coverage of the target while minimizing the background cover to maintain the visual effect of the repair. In addition, the algorithm also provides a manual correction function, which can manually adjust the mask area after detection to ensure the accurate marking of all sensitive targets, providing a reliable basis for subsequent removal steps.

[0030] In the target elimination stage, the algorithm introduces the AOT-GAN model for automatic restoration. AOT-GAN has advantages in processing complex scenes in high-resolution images and can generate filled areas with high realism and rich details, thus ensuring the visual consistency and usability of the restored images.

[0031] In summary, this method provides a feasible technical solution for eliminating sensitive targets in large-size and high-resolution remote sensing images through the organic combination of image block preprocessing, automatic detection, manual correction and automatic repair. It is suitable for performing efficient and accurate sensitive information protection tasks in practical applications.

[0032] Specifically, the remote sensing image sensitive target elimination method includes steps S10 to S30.

[0033] S10: performing block processing on the high-resolution large-size image to obtain an area of ​​interest containing sensitive targets, and converting the geographic coordinates of the area of ​​interest into pixel coordinates.

[0034] Before target detection, high-resolution large-size images are first processed in blocks to reduce computational complexity and improve processing efficiency. For large-size high-resolution remote sensing images (above one billion pixels), the region of interest (ROI) containing sensitive targets is first determined, and then its geographic coordinates are converted to pixel coordinates to ensure spatial reference consistency during sensitive target detection and elimination. Assume that there is the following affine transformation relationship between the geographic coordinates and pixel coordinates of the image: ; In the formula, x and y Represent the horizontal and vertical coordinates of the image pixels respectively; X and Y Respectively represent the geographic abscissa and ordinate of the area of ​​interest; a , b , c , d , e and f Both represent affine transformation parameters, including the image's geographic origin, pixel resolution, and image rotation angle.

[0035] S20: Based on the target area, use the trained instance segmentation model to perform target detection on each target area, generate a corresponding segmentation mask, dynamically adjust the expansion degree of the segmentation mask according to the target area, and determine the mask area; wherein the target area is an area manually drawn in the area of ​​interest containing sensitive targets.

[0036] Step S20 is the sensitive target recognition stage. In order to balance detection accuracy and processing efficiency while retaining more background texture and context information, this paper adopts a sensitive target detection method based on YOLOv8 instance segmentation, and combines the adaptive expansion of the mask and manual assisted correction technology to improve the target elimination effect.

[0037] First, the area containing sensitive targets is manually drawn in the ROI area, and sensitive target mask generation and recognition operations are performed in this area. This strategy aims to ensure that the detection area meets the optimal input size requirements of the YOLO model, thereby maximizing the recognition performance. After determining the target detection area, the trained YOLOv8 instance segmentation model is used to perform target detection in each area and generate the corresponding segmentation mask.

[0038] In some embodiments, in order to optimize the quality of the segmentation mask, an adaptive dilation method based on the target area is proposed. Through the empirical formula (2), this method can dynamically adjust the dilation degree according to the target area: a larger dilation range is used for a larger target, while a moderate dilation is performed for a smaller target, so that the dilation effect has a nonlinear relationship with the target area. This design takes into account both the target coverage rate and the background retention rate, and effectively improves the quality of subsequent repair processing. In addition, the generated mask can also be manually corrected to further ensure the accuracy of the mask. The comparison of the mask before and after dilation is as follows: Figure 2 shown.

[0039] During the expansion process, the minimum value of the expansion kernel is set to 3 (for small targets) and the maximum value is 7 (for large targets). Through this adaptive expansion strategy, the information of the background area can be retained to the greatest extent while completely covering the target, thus improving the effect and naturalness of sensitive target elimination. The formula for adaptive expansion is as follows: ; In the formula, is the expanded core size; area is the mask area; 500 is the scaling factor used to adjust the degree of expansion; max is the maximum function; min is the minimum function; int is the closest distance to the specified edge set.

[0040] S30: Generate filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

[0041] After confirming the mask area, the AOT-GAN (Aggregated Contextual-Transformation GAN) model is introduced in step S30 to automatically fill and repair the sensitive target area. AOT-GAN is a high-resolution image restoration model based on a generative adversarial network, and performs particularly well in the task of filling in large-scale free-form missing areas. The model structure of AOT-GAN is as follows: Figure 3 shown.

[0042] In some embodiments, the infill content generated by AOT-GAN is defined as follows: ; In the formula, The filling content generated by the generator, is the original image; The image after the target is eliminated; is the mask area.

[0043] After the sensitive targets are eliminated, the algorithm will reassemble the filled segmented areas back to the original image and adjust the color difference to ensure that the stitching area is visually consistent with the original image. Figure 1 The re-stitched image is It is expressed as: ; In the formula, is the stitched image; is the mask of the segmented area to ensure that the stitching result is visually consistent with the original Figure 1 To; The number of the partition area.

[0044] Through the organic combination of the above-mentioned image segmentation, automatic detection, mask correction and AOT-GAN elimination, this embodiment achieves efficient and accurate sensitive target elimination in high-resolution image processing of sensitive targets, which is suitable for privacy protection and information security tasks in practical applications.

[0045] In this embodiment, the aircraft is used as a sensitive target for experimentation, and the dataset used is the high-resolution remote sensing image fine-grained target recognition dataset (FAIR1M) independently developed by the Institute of Space and Aerospace. 3,000 images containing aircraft were selected from the dataset to construct the training set of the YOLOv8 instance segmentation model. The annotation of the training set uses the semi-automatic image annotation tool ISAT proposed in the document "JI Shuwei, ZHANG Hongyuan. "ISAT with Segment Anything: An Interactive Semi-Automatic Annotation Tool, 2023." URL: https: / / github. com / yatengLG / ISAT_with_segment_anything, updated on (2023): 06-03." to complete the preliminary instance segmentation mask annotation, and manually correct or re-annotate the incompletely annotated areas due to the similarity of the target color to the background or shadows, etc., to ensure the accuracy and completeness of the annotation.

[0046] In terms of the selection of instance segmentation models, the YOLOv8-seg series provides multiple pre-trained models, whose complexity is from simple to complex, namely YOLOv8n-seg.pt, YOLOv8s-seg.pt, YOLOv8m-seg.pt, YOLOv8l-seg.pt and YOLOv8x-seg.pt. The higher the model complexity, the stronger the parameter quantity and feature extraction capability, thereby improving the detection and segmentation accuracy. This embodiment selects the most complex YOLOv8x-seg.pt as the pre-trained model and performs 100 rounds of training to obtain higher detection accuracy and segmentation quality.

[0047] After the training is completed, the optimal model (best.pt) is used as the final model for instance segmentation, and target detection and mask generation are performed on the above 3,000 remote sensing images to provide the mask data required for AOT-GAN training. Based on these 3,000 remote sensing images and their corresponding segmentation masks, the AOT-GAN model completes the filling training of sensitive target areas after 12,500 iterations. By combining the YOLOv8 instance segmentation and the repair capabilities of the AOT-GAN model, the experimental process of this article achieves accurate recognition and elimination of sensitive targets such as aircraft in high-resolution remote sensing images.

[0048] In the experimental stage, in order to comprehensively evaluate the performance of the algorithm on data of different scales, high-resolution remote sensing images of sizes of 1.5GB, 5GB, 10GB, and 23GB were selected as test samples. The test focuses on mask quality evaluation, target elimination quality evaluation, and analysis of processing efficiency and resource consumption. By testing scenarios with different data volumes and complexities, we can fully understand the performance of the algorithm in all aspects, provide key support for its feasibility in practical applications, and provide data basis for subsequent optimization directions.

[0049] In some embodiments, the stitched image is evaluated using evaluation indicators; the evaluation indicators include edge deviation, Hausdorff distance and IoU; wherein the edge deviation and Hausdorff distance are used to evaluate the degree of matching between the mask edge and the real target boundary, and the IoU is used to evaluate the consistency of regional overlap.

[0050] Edge detail accuracy focuses on the degree of match between the mask edge and the true target boundary, that is, the boundary accuracy of the mask. This indicator is particularly important because the deviation of the mask edge may lead to an obvious transition between the filled content and the background, thus affecting the naturalness of the elimination. Specifically, the edge deviation and Hausdorff distance can be used to quantify the difference between the mask boundary and the true boundary of the target.

[0051] Edge deviation is to calculate the minimum distance from each real edge point to the edge of the mask and take the average of these minimum distances. In this way, the average deviation between the mask and the real target edge can be obtained. The smaller the edge deviation value, the closer the mask edge is to the real edge. Suppose the real target edge is , the mask edge is generated as , the edge deviation is , the formula is as follows: ; In the formula, is the total number of true edge pixels; and Represent the pixel positions of the true edge and mask edge respectively; express and The Euclidean distance between them; Dedge represents the edge deviation; Indicates the true target edge.

[0052] The Hausdorff distance is used to measure the maximum deviation between two boundaries. It is a measure of the farthest distance between two sets of points and is defined as the maximum of all minimum distances from the true edge to the mask edge and the maximum of all minimum distances from the mask edge to the true edge. The formula is as follows: ; In the formula, sup represents the maximum distance in the specified edge set; inf represents the shortest distance to the specified edge set. Represents the true target edge Generate mask edges Hausdorff distance; The consistency of regional overlap can be measured by IoU (Intersection over Union). IoU represents the degree of overlap between the predicted mask area and the true target area. It is defined as the ratio of the intersection area to the union area of ​​the two. It is a common standard for evaluating mask quality in target detection and instance segmentation tasks. The value of IoU ranges from 0 to 1. The closer the value is to 1, the higher the overlap between the predicted mask and the true mask, and the better the matching effect. The IoU formula is as follows: ; In the formula, Represents the prediction mask and the true mask The number of intersection pixels; Represents the prediction mask and the true mask The number of pixels in the union.

[0053] This embodiment aims to quantitatively evaluate the quality of the generated mask, using edge detail accuracy (including edge deviation and Hausdorff distance) and average intersection over union (IoU) as evaluation indicators to systematically compare and evaluate the changes in quality before and after mask expansion. The quality of the mask directly affects the naturalness and accuracy of the sensitive target removal effect, so edge accuracy and area overlap are two key evaluation criteria.

[0054] This example selects four images from the test set as samples, uses the trained YOLOv8 instance segmentation model to detect targets on the images, generates a preliminary mask, and further generates an expanded mask. In order to evaluate the accuracy of the edge details of the mask, we compare the generated mask edge with the manually annotated real target edge and calculate its edge deviation and Hausdorff distance.

[0055] When calculating the edge deviation, according to formula (5), for each true edge point, the minimum distance to the mask edge is calculated and the average is calculated. The Hausdorff distance is calculated according to formula (6) to calculate the maximum deviation distance between the mask edge and the true target edge. For the intersection over union (IoU), according to formula (7), the overlap between the mask area and the true target area is calculated to obtain the IoU value of each sample, as shown in Table 1.

[0056] Table 1 Mask quality evaluation experimental results

[0057] Experimental results show that the mask after adaptive dilation is significantly better than the original generated mask in terms of accuracy of edge details. Specifically, the edge deviation and Hausdorff distance of the expanded mask are both smaller, indicating that the edge of the expanded mask matches the edge of the real target more closely. In addition, the average intersection over union (IoU) of the expanded mask shows higher coverage than the original mask on all targets, indicating that the expanded mask can better cover the real target area. The above results show that mask dilation significantly improves the quality of the mask and provides a more accurate mask basis for subsequent target removal operations, thereby effectively reducing the risk of poor repair results due to mask quality issues.

[0058] This embodiment compares the quality of the repaired image before and after the mask expansion from the perspective of the visual effect of target removal, and conducts a qualitative evaluation. The quality of the target removal operation is evaluated by observing the fusion effect between the repaired area and the surrounding background. In the experiment, the target is first detected on the original image to obtain the target area, and the target area is covered with a mask. Then, the original mask is adaptively expanded to obtain a more accurate mask. The original mask and the expanded mask are used as inputs, and the target is removed through the AOT-GAN model to generate a repaired image and compare them. Figure 4 shown.

[0059] Experimental results show that when the expanded mask is used as input, the visualization effect of the object removal task is significantly better than that of the original mask. The removal results under the expanded mask input are visually more coherent and natural, and the transition between the repaired area and the background area is smoother and there is no obvious splicing trace. In addition, the expanded mask effectively avoids the unnatural transition phenomenon that may occur during the removal process while maintaining a high coverage, further improving the quality of image restoration.

[0060] This example also evaluates the processing efficiency of each image, including the total time required for mask generation, target repair, and stitching. By comparing the processing speed, memory consumption, and CPU usage of images of different scales (1.5GB, 5GB, 10GB, and 23GB), the impact of different image sizes and tile sizes on processing efficiency and resource usage is measured.

[0061] The device parameters used in this embodiment are Windows 10 system, with CPU i7-13700 and running memory 32G. Figure 5 and Figure 6 shown. Figure 5 and Figure 6 In the line graph shown in , the blue line represents the CPU usage (unit: %), and the orange line represents the memory usage (unit: MB). The horizontal axis is the processing time (unit: seconds), the left vertical axis is the CPU usage, and the right vertical axis is the memory usage. The CPU usage exceeds 100% because the device has a multi-core architecture (a single core fully loaded is 100%). Figure 5 and Figure 6 The stages are divided as follows: stage A is the image loading process; the section between A and B with an occupancy rate close to 0 is the manual inspection stage; stage B is the target detection process; stage C is the target elimination process; and stage D is the image saving process.

[0062] This example evaluates the processing efficiency and resource consumption of remote sensing images of different sizes, aiming to verify the actual performance of the proposed algorithm in various scenarios. Figure 5 and Figure 6It can be seen that in terms of processing efficiency, when the overall image size is fixed, the loading time increases with the increase of the frame size, and the time of other processing stages remains unchanged; when the frame size is fixed, the output time increases with the increase of the image size, and the time of other stages is also unaffected. In terms of resource consumption, the peak value of CPU occupancy is stable at around 600%, and is not affected by the image size and frame size; the peak value of memory occupancy increases with the increase of the frame size, but has nothing to do with the overall image size. The experimental results show that the processing efficiency and resource utilization of the two core stages of target recognition and target elimination are not affected by the overall image size or frame size, and the loading and output stages are mainly affected. The method shows stable processing efficiency and resource consumption in different application scenarios, showing good performance and resource adaptability, and can maintain efficient utilization of computing resources while ensuring high processing accuracy.

[0063] In summary, the remote sensing image sensitive target elimination method proposed in the embodiment of the present application can efficiently and accurately detect and remove sensitive information in high-resolution remote sensing images to meet the needs of privacy protection and information security. Through the combination of image block preprocessing, sensitive target recognition, mask correction and automatic filling and repair modules, the algorithm reduces the computing cost while improving the detection efficiency, and realizes high-precision recognition and natural elimination of sensitive targets in large-scale remote sensing images.

[0064] In the specific implementation, the YOLOv8 instance segmentation model is used to automatically detect and mark sensitive target areas. At the same time, the mask adaptive expansion and manual correction functions are combined to further optimize the mask coverage effect and ensure accurate labeling of sensitive areas. Subsequently, the context aggregation capability of the AOT-GAN model is used to automatically repair the target area and generate fill content that seamlessly blends with the surrounding environment, thereby effectively improving the quality and visual consistency of the repaired image.

[0065] Experimental results show that the method proposed in this embodiment shows excellent detection and elimination performance in complex scenes, and can effectively reduce the residual and unnatural edge transition phenomena in target elimination, ensuring the integrity and authenticity of the image. This algorithm provides a reliable technical solution for the protection of sensitive information in high-resolution remote sensing images and is suitable for a variety of practical application scenarios.

[0066] The present application also provides a remote sensing image sensitive target elimination device, such as Figure 7 As shown, the remote sensing image sensitive target elimination device includes: The image segmentation module 701 is configured to segment the high-resolution large-size image into blocks to obtain an area of ​​interest containing sensitive targets, and convert the geographic coordinates of the area of ​​interest into pixel coordinates; The detection and correction module 702 is configured to perform target detection on each target area based on the target area using the trained instance segmentation model, and generate a corresponding segmentation mask, dynamically adjust the expansion degree of the segmentation mask according to the target area, and determine the mask area; wherein the target area is an area containing sensitive targets manually drawn in the region of interest; The elimination and splicing module 703 is configured to generate filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

[0067] In some embodiments, the image segmentation module is further configured to convert the geographic coordinates of the region of interest into pixel coordinates by the following formula: ; In the formula, x and y Represent the horizontal and vertical coordinates of the image pixels respectively; X and Y Respectively represent the geographic abscissa and ordinate of the area of ​​interest; a , b , c , d , e and f Both represent affine transformation parameters.

[0068] In some embodiments, the trained instance segmentation model.

[0069] In some embodiments, the expansion degree of the segmentation mask is dynamically adjusted according to the target area, and the calculation process of determining the mask area is expressed as: ; In the formula, is the expanded core size; area is the mask area; 500 is the scaling factor used to adjust the degree of expansion; max is the maximum function; min is the minimum function; int is the closest distance to the specified edge set.

[0070] In some embodiments, the de-joining module is further configured to: The filling content is generated by the following formula and filled into the mask area to obtain the image after the target is eliminated: ; In the formula, The filling content generated by the generator, is the original image; The image after the target is eliminated; is the mask area; The image after the target is eliminated is spliced ​​to the original image using the following formula to obtain a spliced ​​image: ; In the formula, is the stitched image; is the mask of the segmented region; The number of the partition area.

[0071] In some embodiments, the apparatus further comprises an evaluation module, wherein the evaluation module is configured to: The spliced ​​image is evaluated using evaluation indicators; the evaluation indicators include edge deviation, Hausdorff distance and IoU; wherein the edge deviation and Hausdorff distance are used to evaluate the matching degree between the mask edge and the real target boundary, and the IoU is used to evaluate the consistency of regional overlap; The edge deviation is calculated by the following formula: ; In the formula, is the total number of true edge pixels; and Represent the pixel positions of the true edge and mask edge respectively; express and The Euclidean distance between them; Dedge represents the edge deviation; Indicates the true target edge; The Hausdorff distance is calculated using the following formula: ; In the formula, sup represents the maximum distance in the specified edge set; inf represents the shortest distance to the specified edge set. Represents the true target edge Generate mask edges Hausdorff distance; IoU is calculated using the following formula: ; In the formula, Represents the prediction mask and the true mask The number of intersection pixels; Represents the prediction mask and the true mask The number of pixels in the union.

[0072] An embodiment of the present application provides an electronic device, which may include: a processor and a memory, wherein the processor and the memory may communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0073] The processor executes the computer execution instructions stored in the memory, so that the processor executes the scheme in the above embodiment. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0074] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0075] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0076] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the method for eliminating sensitive targets in remote sensing images of the above embodiment.

[0077] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the remote sensing image sensitive target elimination method in the above embodiment.

[0078] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0079] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0080] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0081] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0082] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0083] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0084] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0085] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0086] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0087] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for eliminating sensitive targets in remote sensing images, characterized in that: The method comprises: The high-resolution large-size image is processed in blocks to obtain an area of ​​interest containing sensitive targets, and the geographic coordinates of the area of ​​interest are converted into pixel coordinates; Based on the target area, the trained instance segmentation model is used to perform target detection on each target area, and a corresponding segmentation mask is generated. The expansion degree of the segmentation mask is dynamically adjusted according to the target area to determine the mask area; wherein the target area is an area containing sensitive targets manually drawn in the region of interest; Generate a filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

2. The remote sensing image sensitive target elimination method according to claim 1, characterized in that: The geographic coordinates of the area of ​​interest are converted to pixel coordinates using the following formula: ； In the formula, x and y Represent the horizontal and vertical coordinates of the image pixels respectively; X and Y Respectively represent the geographic abscissa and ordinate of the area of ​​interest; a , b , c , d , e and f Both represent affine transformation parameters.

3. The remote sensing image sensitive target elimination method according to claim 1, characterized in that: The trained instance segmentation model.

4. The remote sensing image sensitive target elimination method according to claim 1, characterized in that: The expansion degree of the segmentation mask is dynamically adjusted according to the target area, and the calculation process of determining the mask area is expressed as: ； In the formula, is the expanded core size; area is the mask area; 500 is the scaling factor used to adjust the degree of expansion; max is the maximum function; min is the minimum function; int is the closest distance to the specified edge set.

5. The remote sensing image sensitive target elimination method according to claim 1, characterized in that: Generate a filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image, including: The filling content is generated by the following formula and filled into the mask area to obtain the image after the target is eliminated: ； In the formula, The filling content generated by the generator, is the original image; The image after the target is eliminated; is the mask area; The image after the target is eliminated is spliced ​​to the original image using the following formula to obtain a spliced ​​image: ； In the formula, is the stitched image; is the mask of the segmented region; The number of the partition area.

6. The remote sensing image sensitive target elimination method according to claim 1, characterized in that: After obtaining the spliced ​​image, the method further includes: The spliced ​​image is evaluated using evaluation indicators; the evaluation indicators include edge deviation, Hausdorff distance and IoU; wherein the edge deviation and Hausdorff distance are used to evaluate the matching degree between the mask edge and the real target boundary, and the IoU is used to evaluate the consistency of regional overlap; The edge deviation is calculated by the following formula: ； In the formula, is the total number of true edge pixels; and Represent the pixel positions of the true edge and mask edge respectively; express and The Euclidean distance between them; Dedge represents the edge deviation; Indicates the true target edge; The Hausdorff distance is calculated using the following formula: ； In the formula, sup represents the maximum distance in the specified edge set; inf represents the shortest distance to the specified edge set. Represents the true target edge Generate mask edges Hausdorff distance; IoU is calculated using the following formula: ； In the formula, Represents the prediction mask and the true mask The number of intersection pixels; Represents the prediction mask and the true mask The number of pixels in the union.

7. A remote sensing image sensitive target elimination device, characterized in that: The device comprises: An image segmentation module is configured to segment high-resolution large-size images to obtain an area of ​​interest containing sensitive targets, and convert geographic coordinates of the area of ​​interest into pixel coordinates; The detection and correction module is configured to perform target detection on each target area based on the target area using the trained instance segmentation model, generate a corresponding segmentation mask, dynamically adjust the expansion degree of the segmentation mask according to the target area, and determine the mask area; wherein the target area is an area containing sensitive targets manually drawn in the region of interest; The elimination and splicing module is configured to generate filling content and fill it into the mask area to obtain an image after the target is eliminated, and splice the image after the target is eliminated to the original image to obtain a spliced ​​image.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the remote sensing image sensitive target elimination method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the remote sensing image sensitive target elimination method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for eliminating sensitive targets in remote sensing images according to any one of claims 1 to 6 is implemented.

Images