Method and device for automatically removing multi-class targets in remote sensing image

Abstract

The invention provides a remote sensing image multi-category target automatic removal method and device, relates to the technical field of remote sensing image processing, and mainly aims to solve the technical problem that an existing remote sensing image multi-category target removal method cannot implement a removal strategy according to environment characteristics of multi-category targets. According to the main technical scheme, the method comprises the following steps: obtaining a to-be-restored remote sensing image, wherein the to-be-restored remote sensing image comprises a to-be-restored region covering multiple categories to be removed; determining environmental characteristics of a known area around the to-be-restored area in the to-be-restored remote sensing image; according to a removal strategy corresponding to the environmental characteristics, performing multi-category target removal processing on the to-be-repaired area; and if the to-be-repaired area is segmented, performing splicing processing on the to-be-repaired area after removal processing.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for automatically removing multi-category targets from remote sensing images. Background Art

[0002] Remote sensing detection technology is widely used in many fields such as environmental monitoring, agricultural management, and disaster assessment. Images obtained through satellites, drones, and other platforms provide rich geographic information. However, in remote sensing image processing, especially when it comes to data processing in highly sensitive areas such as national security and military secrets, directly disclosing remote sensing images will undoubtedly expose potential security risks. In addition, in the modeling process of remote sensing images, multiple targets (such as bridges, clouds, vehicles, etc.) will interfere with the modeling. Therefore, multi-category target removal has become a key step in remote sensing image processing.

[0003] For the removal of multi-category targets in remote sensing images, current technologies have adopted differentiated processing strategies, with particular emphasis on the differentiated processing of targets in water and land areas. During the processing, water and land are first clearly divided, and then removal methods that match the characteristics of the area are selected based on the specific area where the target is located. For example, for targets in water areas, the removal process needs to fully consider the dynamic characteristics of the water body, such as fluidity and unique reflection effects; while for targets on land, attention needs to be paid to the static characteristics of the surface, such as texture complexity and color diversity, to ensure that the targets can maintain a high degree of harmony and consistency with their respective environments after removal.

[0004] However, although the current remote sensing image target processing method can effectively repair the image to a certain extent, it mainly focuses on the removal strategy at the macro level, that is, according to the overall category of the target area (such as water or land), the target is removed according to the preset rules. However, this method ignores the microscopic characteristics of the specific environment in which the multi-category target is located. For example, when a target is located in a land area, but its environment exhibits reflective characteristics similar to those of aquatic water bodies, if the removal method for the land environment is continued, the processing effect is often unsatisfactory, resulting in the processed remote sensing image appearing unnatural and uncoordinated in environmental fusion, which in turn damages the practicality and aesthetics of the image. Therefore, how to implement the removal strategy based on the environmental characteristics of the multi-category target, so as to achieve high-quality multi-category target removal in remote sensing images, has become a key technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, the main purpose of the present invention is to provide a method and device for automatically removing multi-category targets from remote sensing images, so as to solve the technical problem that the existing multi-category target removal methods for remote sensing images mainly perform multi-category target removal based on macro-region categories, ignoring the microscopic characteristics of the environment in which the multi-category targets are located, resulting in the processed remote sensing images appearing unnatural and uncoordinated in environmental fusion, that is, the removal strategy cannot be implemented according to the environmental characteristics of the multi-category targets.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a method for automatically removing multi-category targets from remote sensing images, the method comprising:

[0008] Acquire a remote sensing image to be restored, where the remote sensing image to be restored includes a region to be restored that covers multiple categories of targets to be removed;

[0009] Determine the environmental characteristics of the known areas surrounding the area to be restored in the remote sensing image;

[0010] According to the removal strategy corresponding to the environmental characteristics, the area to be repaired is subjected to multi-category target removal processing, and the removal strategy at least includes using images in a known area around the area to be repaired to repair the area to be repaired and using a convolutional neural network to repair the output image of the area to be repaired according to the input environmental characteristics;

[0011] If the area to be repaired is segmented, the area to be repaired after the removal process is performed on the segmented area.

[0012] A second aspect of the present invention provides a device for automatically removing multi-category targets from remote sensing images, the device comprising:

[0013] An acquisition unit is used to acquire a remote sensing image to be restored, where the remote sensing image to be restored includes a region to be restored that covers multiple categories of targets to be removed;

[0014] A determination unit, used to determine the environmental characteristics of a known area around the area to be restored in the remote sensing image to be restored in the acquisition unit;

[0015] A removal unit is used to perform multi-category target removal processing on the area to be repaired according to a removal strategy corresponding to the environmental characteristics in the determination unit, wherein the removal strategy at least includes repairing the area to be repaired using images in a known area around the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network according to the input environmental characteristics;

[0016] The splicing unit is used to splice the area to be repaired after the removal process in the removal unit if the area to be repaired is divided.

[0017] The third aspect of the present invention provides an electronic device, which includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute a method for automatically removing multi-category targets from remote sensing images.

[0018] A fourth aspect of the present invention provides a readable storage medium, the storage medium being used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for automatically removing multi-category targets from remote sensing images.

[0019] By means of the above technical scheme, the present invention proposes a method and device for automatically removing multi-category targets from remote sensing images. The core of the method is to flexibly select the most suitable removal means to process the area to be removed according to the specific characteristics of the environment in which the multi-category targets to be removed are located, so as to achieve effective concealment of the multi-category targets. Specifically, first, a remote sensing image containing the multi-category targets to be removed is selected as the "remote sensing image to be repaired" to be processed. Since the positions of the multi-category targets have been identified and marked in advance, it is clear which areas in the remote sensing image to be repaired need to be repaired, that is, the "area to be repaired". Next, the environmental characteristics of the known areas around the area to be repaired in the remote sensing image to be repaired are determined. Finally, according to the identified environmental characteristics, a removal strategy matching it is selected. The selection of this strategy is crucial because it will guide how to remove the multi-category targets in the area to be repaired to ensure that the removal effect is both natural and in line with the characteristics of the environment. If the area to be repaired is segmented and then the removal strategy is used to remove the multi-category targets, then the segmented area to be repaired needs to be spliced ​​to reduce the problem of splicing inconsistency. In summary, the method proposed in the present invention can flexibly select the most suitable removal means according to the removal requirements of multiple categories of targets under different environmental characteristics, thereby ensuring that the removal effect is both natural and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 A schematic diagram of a process for automatically removing multi-category targets from remote sensing images proposed in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a process for selecting a removal strategy according to environmental characteristics proposed in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a process for applying a dynamic classification corresponding removal strategy proposed in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a process for applying a static classification corresponding removal strategy proposed in an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a device for automatically removing multi-category objects from remote sensing images proposed in an embodiment of the present invention;

[0026] Figure 6 A detailed structural diagram of a device for automatically removing multi-category objects from remote sensing images proposed in an embodiment of the present invention;

[0027] Figure 7 A comparison diagram of the effects of a dynamic classification corresponding removal strategy proposed in an embodiment of the present invention;

[0028] Figure 8 This is a comparison diagram of the effects of applying static classification corresponding to the removal strategy proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0031] The design inspiration of the present invention comes from an in-depth analysis of the limitations of existing remote sensing image multi-category target removal technologies. Specifically, existing methods are mainly based on macro-region categories (such as waters and land), ignoring the specific environmental characteristics of the multi-category targets. For example, when the target is located in a land area with special reflective characteristics, the removal method of the ordinary land environment will result in poor processing effect, making the image appear unnatural and uncoordinated. In addition, existing methods usually follow preset fixed rules and lack the ability to flexibly adjust according to environmental characteristics, which limits their scope of application and effect. These limitations not only damage the practicality of the image, but also affect the aesthetics.

[0032] These limitations prompted the inventors to propose a new technical concept to overcome the shortcomings of existing methods. The core of this technical concept is to accurately identify the specific environmental characteristics of the known areas around the area to be repaired and determine the removal strategy corresponding to the environmental characteristics. The advantage of this method is that it does not simply apply preset fixed rules, but flexibly selects the most appropriate removal strategy based on the identified microscopic characteristics. This processing method ensures the naturalness of the removal effect and its coordination with the surrounding environment, thereby improving the practicality and aesthetics of remote sensing image processing.

[0033] In the embodiment of the present invention, a method for automatically removing multi-category targets from remote sensing images is mainly described. The method is executed by a central processing unit (CPU) or a graphics processing unit (GPU). The method obtains a remote sensing image containing multi-category targets to be removed, determines the environmental characteristics of the known area around the area to be repaired, and selects a suitable removal strategy based on these characteristics to process the area to be repaired, thereby achieving the removal of multi-category targets. In short, the removal operation of specific multi-category targets in remote sensing images is guided by environmental characteristic analysis. The specific execution steps are as follows: Figure 1 As shown, including:

[0034] Step 101: Obtain the remote sensing image to be restored.

[0035] In this step, the remote sensing image to be modified is the area to be repaired that includes the multi-category targets to be removed. It should be noted that the area to be repaired not only includes the multi-category target area, but also includes the areas that are affected by the multi-category target area and need to be removed, such as the shadows of the multi-category targets, so as to ensure the natural removal effect.

[0036] Specifically, before obtaining the remote sensing image to be repaired in this step, the initial remote sensing image needs to be preprocessed, and then the trained model is applied or fine-tuned to automatically identify the multi-category targets to be removed from the preprocessed image, generate a preliminary multi-category target frame, and then automatically optimize and adjust the multi-category target frame according to the shadow features to generate an accurate multi-category target frame. Finally, based on the accurate multi-category target frame, a mask of the area to be repaired is generated. Among them, the specific technical process for the identification of multi-category targets and shadows to be removed is as follows:

[0037] In this step, the preprocessed remote sensing image is used as input, and a trained model is used or an existing model is loaded to perform multi-category target recognition and distinguish the multi-category targets to be removed in the remote sensing image, and the multi-category target frames of the multi-category targets to be removed are automatically output.

[0038] After detecting multiple categories of objects, we further identify the shadow areas within or adjacent to the bounding box. Based on the specific form of the shadow (such as direction and length), we intelligently adjust the original bounding box to ensure that multiple categories of objects and their complete shadows are seamlessly included. The steps are as follows:

[0039] Based on the center of the current multi-category target frame, each side of the multi-category target frame is expanded outward in parallel by N pixels. The selection of N needs to be adjusted according to the actual situation, and a corresponding value needs to be determined through experiments or prior knowledge. If the multi-category target frame after expansion exceeds the image boundary, the exceeding part should be regarded as an invalid area and will not participate in subsequent calculations. Determine the range of expansion according to the multi-category target frame before and after expansion, divide the expanded part into four areas, and number them according to the corresponding edges. Obtain the corresponding image according to the four expanded areas, and use the threshold segmentation method through color space conversion to better separate the shadow and background. According to the corresponding number of the shadow area, the corresponding edge of the multi-category target frame is further expanded outward by N2 pixels. N2 can be a different value from N, depending on the size and characteristics of the shadow. Repeat this process of selecting the shadow of the multi-category target until no shadow area is detected outside the boundary of the expanded multi-category target frame.

[0040] In general, in step 101, through a series of operations, the remote sensing image to be restored containing the multi-category targets to be removed and their affected areas is successfully obtained. This step provides the necessary premise and basis for the subsequent multi-category target removal process, ensuring the accuracy and naturalness of the removal operation.

[0041] Step 102: Determine the environmental characteristics of the known area around the area to be restored in the remote sensing image to be restored.

[0042] This step is a key step in the multi-category object automatic removal method. The purpose of this step is to provide the necessary information basis for the subsequent restoration process to ensure that the restoration result is as close to the original image as possible and natural.

[0043] The environmental characteristics here refer to the various physical properties, spectral characteristics, texture characteristics, etc. in the known area around the area to be repaired, which can describe the information of the area.

[0044] In order to accurately obtain these environmental characteristics, the technical means that can usually be referenced include but are not limited to: Image processing algorithms: used to extract textures, edges and other spatial features in images. Machine learning or deep learning models: train models to identify specific types of objects or features. Geographic Information System (GIS) data: use existing geographic data to assist in determining environmental characteristics. Spectral analysis: distinguish different objects by analyzing the spectral information of different bands.

[0045] Once the environmental characteristics are determined, corresponding removal strategies can be developed based on this information. For example, if the area to be repaired is located in a vegetation-covered area, other healthy vegetation pixels in the area can be used as samples to reconstruct the occluded or damaged parts. Alternatively, in the convolutional neural network (CNN) method, input environmental characteristics can help the network better understand the context and generate more realistic repair results.

[0046] In summary, step 102 is an important step in ensuring the quality of restoration, and it provides a solid foundation for subsequent restoration work by carefully analyzing and understanding the environmental characteristics around the area to be restored.

[0047] Step 103: According to the removal strategy corresponding to the environmental characteristics, multi-category target removal processing is performed on the area to be repaired.

[0048] In the process of implementing multi-category object removal, once the environmental characteristics of the area to be repaired are determined in step 102, the next step is to select a suitable removal strategy for the area based on these characteristics. The removal strategy at least includes using images in the known area around the area to be repaired to repair the area to be repaired and using a convolutional neural network to repair the output image of the area to be repaired based on the input environmental characteristics. The importance of this step is that it directly affects the naturalness and fidelity of the image after removal.

[0049] Imagine a specific application scenario: suppose a radar device is located on a smooth and highly reflective surface. Since the deployment of the radar involves military secrets, it needs to be removed from the remote sensing image. To achieve this goal, the environmental characteristics must be considered in depth.

[0050] If the removal strategy chosen ignores the highly reflective nature of the radar support surface in the original remote sensing image, the image after the removal operation may appear unrealistic and fail to correctly represent the light and shadow relationship between the radar and its support surface. In this case, the shape of the radar base may leave a dull trace in the image, while the surrounding area is a highly reflective area. Such an inconsistency can easily make people aware of the presence of the radar.

[0051] Therefore, it is necessary to customize the removal strategy according to the specific characteristics of the radar environment to ensure that the environmental details of the original image without removing the multi-category targets, such as the reflected light effect, can be retained during the image restoration process. In this way, not only the multi-category targets are successfully removed, but also the natural texture and lighting effects of their surroundings are maintained, making the restored remote sensing image visually natural and coherent, thus achieving highly natural and effective multi-category target removal.

[0052] There may be multiple options for the removal strategy determined by environmental characteristics, or only one most suitable solution. When faced with multiple removal strategies, historical data and usage results can be used to evaluate which strategy is superior.

[0053] Specifically, when the environmental characteristics tend to be fluidity, reflectivity, and other characteristics that are easy to change with time or external conditions, it should be noted that this does not mean that the environmental characteristics in the remote sensing image are constantly changing in real time, but that these characteristics belong to the category of dynamic changes. In this case, the reference removal strategies include but are not limited to: selecting the strongest characteristic among these characteristics, and then finding the image matching block that is most similar to the strongest characteristic in the known area around the area to be repaired in the remote sensing image to be repaired, and using this image block to repair the area to be repaired.

[0054] When the environmental characteristics tend to be texture-like and not easy to change with time or external conditions, it should be noted that this means that these characteristics belong to the category of static changes. In this case, the optional removal strategies include but are not limited to: using texture synthesis technology to generate camouflage textures based on the texture characteristics of the surrounding environment and cover them on multi-category targets to achieve removal. Using image fusion technology, the images of multi-category targets are fused with the images of the surrounding environment, so that the multi-category targets are visually integrated with the environment.

[0055] When environmental characteristics are biased, such as transparency, lighting conditions, temperature changes, etc., the optional removal strategies include but are not limited to: For environments with changing lighting conditions, use lighting simulation technology to adjust the lighting effects on the surfaces of multiple categories of targets to match the ambient lighting. For situations where multiple categories of targets need to be removed under different spectra, use multi-spectral camouflage materials or technologies to ensure that multiple categories of targets are difficult to detect in remote sensing images of different spectra.

[0056] In summary, step 103 effectively achieves multi-category target removal while maintaining the visual integrity of the remote sensing image and the authenticity of the environmental information, achieving the purpose of removing multi-category targets while maintaining the naturalness of the environment.

[0057] Step 104: If the area to be repaired is segmented, the area to be repaired after the removal process is performed a splicing process.

[0058] In this step, it is described how to perform stitching processing to generate the final high-quality remote sensing image after the area to be repaired is divided into multiple intermediate areas and these areas are removed.

[0059] Specifically, if an object spans across multiple segmented regions, at the junction of these regions, the same / different removal strategies are used for this object.

[0060] If different removal strategies are used, it is necessary to eliminate the visual discontinuity caused by different removal strategies, such as: decomposing the image into multiple frequency layers, fusing each layer separately, and then resynthesizing; using pre-trained neural network models to automatically learn how to optimally fuse different processing results; using content-aware filling or texture synthesis and other technologies to generate naturally transitioned intermediate images; performing local color correction, contrast adjustment, etc. to further improve the overall quality of the image. If the same removal strategy is used, it is necessary to ensure smooth transitions between areas using the same removal strategy, such as: using feature point detection algorithms such as SIFT and SURF to find common feature points in adjacent areas, and then aligning these feature points through geometric transformations; adjusting the colors of adjacent areas to ensure color consistency; using gradient fusion, feathering effects and other technologies to reduce hard edge phenomena at the boundaries; using weighted average methods at the junctions to smooth transitions, such as Laplace pyramid blending.

[0061] After the stitched image is generated using the above steps, a comprehensive quality check is performed on the entire stitched image to find and correct any unnatural edges or color mutations. If slight inconsistencies or stitching marks are found, local adjustments can be made, such as color correction, contrast adjustment, local smoothing, etc., to ensure that the stitched image has a smooth transition visually and no obvious stitching marks. The processed image is accurately replaced back to its original position.

[0062] The processed image is accurately replaced back to its original location and the relevant metadata record is updated to ensure that future users will know that the part has been specially processed.

[0063] Through the above steps, high-quality image stitching processing can be effectively achieved to generate the final high-quality remote sensing image. This method not only improves the technical level of image processing, but also provides a more reliable data foundation for subsequent applications.

[0064] Based on the above Figure 1It can be seen from the implementation method that the present invention proposes a method and device for automatically removing multi-category targets from remote sensing images. The core of the method is to flexibly select the most suitable removal means to process the area to be removed according to the specific characteristics of the environment in which the multi-category targets to be removed are located, so as to achieve effective concealment of the multi-category targets. Specifically, first, a remote sensing image containing multi-category targets to be removed is selected as the "remote sensing image to be repaired" to be processed. Since the positions of the multi-category targets have been identified and marked in advance, it is clear which areas in the remote sensing image to be repaired need to be repaired, that is, the "area to be repaired". Next, the environmental characteristics of the known areas around the area to be repaired in the remote sensing image to be repaired are determined. Finally, according to the identified environmental characteristics, a removal strategy matching it is selected. The selection of this strategy is crucial because it will guide how to remove the multi-category targets in the area to be repaired to ensure that the removal effect is both natural and in line with the characteristics of the environment. In summary, the method proposed by the present invention can flexibly select the most suitable removal means according to the removal requirements of multi-category targets under different environmental characteristics, thereby ensuring that the removal effect is both natural and effective. If the area to be repaired is segmented and then a removal strategy is used to remove multiple categories of targets, the segmented area to be repaired needs to be spliced ​​to reduce the problem of splicing inconsistency.

[0065] Furthermore, in the above embodiment, a method for removing multiple categories of targets by determining a removal strategy based on the characteristics of the environment in which the multiple categories of targets to be removed are located has been outlined. The core of this method is to select or design the most suitable removal strategy based on the identified environmental characteristics. These strategies can adapt to changes in the environment, so that the removed area blends in with the surrounding environment and avoids obvious visual differences. Now, based on the above discussion, Figure 2 As shown, it specifically explains how to select the corresponding removal strategy according to the environmental characteristics.

[0066] Step 201: traverse the area to be repaired and obtain the corresponding four-boundary range.

[0067] This step is used to clarify the boundaries of the specific area that needs to be processed. The following is a specific analysis of this step:

[0068] Traversing the area to be repaired actually accesses the target frame of each area to be repaired in the remote sensing image to be repaired. Each target frame is a rectangle, and these areas have been marked or identified as parts that need to be repaired. By traversing these target frames, the location and scope of all areas to be repaired can be fully understood to ensure that nothing is missed.

[0069] The four-boundary range refers to the rectangular bounding box formed by the farthest points in the east, south, west, and north directions of all the areas to be repaired. This bounding box corresponds to the positive XY axis, even if a single target box may be a tilted rectangle or other shape. By calculating these boundaries, a minimum rectangular box containing all the areas to be repaired can be determined for subsequent fixed-point cropping.

[0070] Accurately obtaining the boundaries of the area to be repaired is very important for subsequent steps, because it directly affects how to correctly select and analyze the surrounding known areas, and then affects the selection and application of the removal strategy. In this way, the consistency and authenticity of the repair result with the original image can be guaranteed, avoiding unnatural transitions or erroneous repair results.

[0071] Step 202: Buffer N pixels outward according to the range to obtain an image block to be processed.

[0072] This step is intended to ensure that the environmental characteristics of the surrounding area are utilized during the restoration process, thereby improving the authenticity of the restoration.

[0073] Once the bounds are determined in step 201, the next step is to expand the bounds by a certain number of pixels (e.g., N pixels) on this basis. This means that N pixels are added to the top, bottom, left, and right of the original rectangular frame. This is done to ensure that the image is not limited to the original area to be repaired, but includes a certain width of surrounding environment information.

[0074] To ensure that the repaired image can smoothly transition into the surrounding known environment, based on the rectangular frame obtained in the previous step, N pixels are expanded outward to form a new rectangular frame. The new rectangular frame is used to crop the area to be repaired to obtain a processed image block that includes the area to be repaired and its surrounding environment. This value N should be large enough to capture enough context information, but not too large to avoid introducing unnecessary complexity or interference information.

[0075] It should be noted that the specific number of pixels N to be expanded outward is determined. The selection of this value depends on multiple factors, including but not limited to the size and type of the object to be repaired and the amount of context information required. For example, for larger objects or complex scenes, a larger N value may be required to capture more background information; while for smaller objects or simple scenes, a smaller N value may be selected to reduce the computational burden.

[0076] After completing the above expansion, a new, larger rectangular area will be obtained. This new area is called the "image block to be processed", which includes the original area to be repaired and a part of the known area around it.

[0077] Step 203: Determine the environmental characteristics of the known area in the image block to be processed.

[0078] In the image block to be processed that has been acquired in the above step 202, since there is a part of known area, the environmental characteristics in this part of known area are acquired, which can be used to guide the subsequent selection and implementation of the removal strategy.

[0079] Extract various environmental characteristics from the known area. This step can be achieved through the following methods: Spectral analysis: Use different band information of remote sensing images (such as visible light, near infrared, short-wave infrared, etc.) to analyze the spectral reflectance characteristics of the objects. Different types of objects usually have unique spectral response patterns. Spectral analysis can effectively distinguish vegetation, water bodies, buildings, etc. Texture analysis: Use image processing algorithms to extract texture features in a known area, such as gray-level co-occurrence matrix (GLCM), local binary pattern (LBP), etc. Texture features can reflect the regularity and complexity of the surface structure of objects, and are very useful for identifying objects such as forests, grasslands, and urban buildings. Shape and spatial relationship: Analyze the shape characteristics of objects and their distribution patterns in space. For example, rivers usually present linear distribution, while lakes are mostly closed surface distribution; buildings may present regular rectangular or square layouts. Geographic Information System (GIS) data integration: Combine existing geographic information data, such as land cover maps, topographic maps, etc., to assist in determining the type of objects and other environmental characteristics in a known area. GIS data provides rich background information, which helps to improve the accuracy of environmental feature identification.

[0080] In summary, this step not only provides the necessary background information and support for subsequent restoration work, but also ensures that the restoration process not only relies on the data of the area to be restored itself, but also makes full use of the information of the surrounding environment, thereby improving the quality and authenticity of the restoration effect.

[0081] Step 204: If there are multiple environmental characteristics in the area to be repaired, the multiple environmental characteristics are classified according to a preset characteristic classification standard, and the total weight value of all environmental characteristics in each classification is calculated respectively.

[0082] In this step, when dealing with the various environmental characteristics of the area to be restored, it is first necessary to classify them according to the preset characteristic classification standards. The classification standards should be scientific and reasonable, and be able to effectively distinguish between dynamic and static characteristics. Dynamic characteristics usually include light reflection characteristics, water flow ripple characteristics, reflection characteristics, etc. These characteristics are prone to change over time or external conditions. Although they may not appear as actual movement in the image to be restored, they are crucial to maintaining the natural feeling of the restored area and the surrounding environment, so as to avoid unrealistic situations after restoration, such as no reflected light in the restored area and reflected light in the surrounding area.

[0083] Static characteristics include texture characteristics, clear structure characteristics, etc. These characteristics are relatively stable. When performing regional repair for such characteristics, the focus of the repair is to restore the real ground texture and structure.

[0084] The weight calculation method can be based on multiple factors such as the distribution frequency of color in the area, the complexity of the texture, and the lighting and reflection characteristics. When setting the weight, each characteristic should be weighted in advance, and the weights corresponding to different degrees of characteristics should also be different.

[0085] Next, the environmental characteristics are classified according to the preset classification standards, such as dynamic classification, static classification, regional classification (i.e., the area to be repaired is divided into multiple blocks, and the characteristics in each block are classified into one category), etc., and their combination. After the classification is completed, the total weight value of all the characteristics in each category needs to be calculated.

[0086] If there are multiple classifications, in order to determine which classification plays a leading role in the restoration of the area to be restored, that is, to select the effective removal strategy corresponding to the classification to ensure the natural restoration effect, multiple classifications need to be evaluated. Of course, there are also cases where the removal strategies corresponding to multiple classifications have similar effects after application. At this time, it is necessary to determine which removal strategy corresponding to the environmental characteristics should be selected according to the actual situation to achieve the natural removal of multi-category targets.

[0087] Through this step, the various environmental characteristics of the area to be repaired can be dealt with clearly and methodically, ensuring the naturalness and authenticity of the restoration effect.

[0088] Step 205: Compare the total weight values ​​and select a removal strategy based on the difference between the first two categories with the largest total weight values.

[0089] In this step, since the above step 204 has calculated the total weight value of all environmental characteristics in various categories, it is necessary to set the threshold reasonably when comparing various weights to ensure that the threshold can effectively distinguish the dominance of different categories and handle the situation where the total weight values ​​are similar to avoid misjudgment.

[0090] Specifically, determine a reasonable threshold to distinguish the dominance of different categories. This threshold should be set based on the importance of environmental features, the goal of repairing multiple categories, and the possible error range. The threshold should be sensitive enough to identify significant differences between total weight values, but also loose enough to avoid misjudgment when the total weight values ​​are similar.

[0091] The total weight values ​​of each category calculated in step 204 are compared. Before determining which category has the largest total weight, the categories are arranged in order from high to low according to the total weight value, so as to identify the dominant category. Among them, since when the weight values ​​of multiple categories are similar, if only the removal strategy corresponding to one category is used to repair the area to be repaired, the repair effect at this time may not be better than the repair effect of the area to be repaired by the removal strategy corresponding to multiple categories. Therefore, a method for selecting the corresponding removal strategy according to the difference between the categories is referred to here: the difference between the first two categories with the largest total weight value is calculated, and this difference will help determine whether there is a significant gap between the two categories. If the difference is large, it means that the dominant category is very obvious, and the removal strategy corresponding to the dominant category can be directly selected; if the difference is small, it means that the importance of the first two categories is similar, and further analysis or adjustment of the threshold is required to determine whether it is necessary to combine the removal strategies of the two categories, or to choose a more general strategy to balance the two. It should be noted that when the difference between the first two categories with the largest total weight value is small, this may mean that both categories are crucial to the selection of the removal strategy. At this time, the importance of other categories should not be ignored. Therefore, continue to determine whether other classifications also have a significant impact on the restoration effect. The specific operation is as follows: calculate the difference between the third largest total weight value and the first two total weight values, and determine whether the difference is greater than or equal to the preset threshold. If the difference is less than the threshold, it means that the classification with the third largest total weight value is equally important, and its removal strategy needs to be considered in the restoration plan. This process can be continued, comparing the difference between the classification with a smaller total weight value and the previous classification to determine whether there are other classifications that need to be considered. In this way, it can be ensured that all important environmental characteristic classifications are taken into account when repairing the area to be repaired, thereby achieving a more natural and coordinated restoration effect.

[0092] In the case of similar total weight values, you can consider the following strategies: Combine the removal strategies of the two categories to create a new strategy. Choose a more flexible or universal removal strategy to adapt to a variety of environmental characteristics. If possible, consult experts or conduct user testing to determine the best strategy.

[0093] Through these steps, it is ensured that the selection of removal strategy is both scientific and reasonable, and can maintain the naturalness and consistency of the restored area with the surrounding environment to the greatest extent.

[0094] Since step 205 has described in detail how to select the corresponding removal strategy according to the difference in total weight values, in order to deeply understand its specific implementation process, dynamic and static classifications will be used as examples to explain in detail how to select the corresponding removal strategy according to the difference in total weight values.

[0095] Assume that the total weight values ​​of the dynamic classification and the static classification have been calculated and arranged in descending order. The following are specific steps for implementing step 205:

[0096] Step 1: Calculate the total weight values ​​of all environmental characteristics in the dynamic classification and static classification respectively.

[0097] In this step, the total weight values ​​of the dynamic classification and the static classification are compared. If the total weight value of the dynamic classification is higher than that of the static classification, then the dynamic characteristics occupy a more important position in the area to be repaired; otherwise, the static characteristics are more important.

[0098] Step 2: Calculate the difference between the total weight values ​​corresponding to the dynamic classification and the static classification, and determine whether the difference is greater than or equal to the threshold.

[0099] In this step, the difference between the category with the highest total weight value (assuming it is a dynamic category) and the category with the second highest total weight value (assuming it is a static category) is calculated. This difference will help determine whether there is a significant difference between the two categories.

[0100] Step 3: Select the removal strategy based on the category corresponding to the maximum total weight value.

[0101] If the judgment in step 2 is greater than or equal to the threshold, that is, the difference is large, this indicates that the dynamic classification clearly dominates the area to be repaired. Therefore, a removal strategy focusing on dynamic characteristics can be selected, for example, using dynamic blur effects to simulate water flow ripples or changes in light reflection.

[0102] Step 4: In the area to be repaired, the area to be repaired is divided into multiple sub-areas, and the category to which each sub-area belongs is determined based on all environmental characteristics in each sub-area, and a removal strategy corresponding to each sub-area is selected according to the classification.

[0103] If the judgment in step 2 is less than the threshold, that is, the difference is small, this indicates that the importance of dynamic and static characteristics in the area to be repaired is almost equal, and there is no obvious advantage classification. In this case, in order to process the area to be repaired more accurately, the following measures can be taken:

[0104] First, since both dynamic and static characteristics have a significant impact on the area, the area to be repaired can be divided into several small areas. This segmentation allows for a more detailed analysis of the environmental characteristics within each small area.

[0105] Next, for each small area, the corresponding removal strategy will be selected according to its specific environmental characteristics classification. For specific operations, please refer to: In small areas where dynamic characteristics are more significant, removal strategies targeting dynamic characteristics will be given priority, such as simulating changes in light reflection or the effect of water flow. In small areas where static characteristics are more significant, the focus will be on removal strategies for static characteristics, for example, adjusting color and texture to match the surrounding stable elements. For small areas where both dynamic and static characteristics are relatively balanced, removal strategies for both characteristics will be combined, or a general strategy that can handle both dynamic and static characteristics will be adopted. In this way, it can be ensured that each small area is properly treated, so that the entire area to be repaired remains natural and coordinated with the surrounding environment after repair.

[0106] Through the above detailed application process, it can be ensured that when repairing the area to be repaired, the removal strategy selected can fully consider the importance of dynamic and static characteristics, as well as their relationship with each other, so as to achieve a natural and coordinated repair effect.

[0107] Further, for step 3 in step 205 above, if the category corresponding to the maximum total weight value is a dynamic category, in order to better describe the specific process of how the removal strategy corresponding to the category repairs the area to be repaired, such as Figure 3 As shown, an application process for dynamic classification corresponding removal strategy is provided:

[0108] Step 301: Obtain the repair priority of each boundary pixel in the area to be repaired.

[0109] In this step, the boundary of the area to be repaired is clarified. Boundary pixels usually refer to the pixel points where the area to be repaired touches the surrounding known area. For each boundary pixel, calculate its repair priority. The calculation of repair priority can be based on the following factors: the position of the pixel, pixels at certain positions may have a greater impact on the overall visual effect, such as corner points or edge pixels; environmental characteristics, considering the environmental characteristics of the area where the pixel is located, such as dynamic characteristics may require a higher priority; visual saliency, the visual saliency of the pixel may affect its priority; contextual information, contextual information around the pixel, such as consistency of texture, color, etc. Different algorithms or models can be used to calculate the repair priority, such as saliency detection based on image content, graph-based algorithms, or machine learning models.

[0110] Here is a priority calculation method worthy of reference: the boundary pixels of the area to be repaired are formed into a linked list, and the data item and confidence of each boundary pixel are obtained. The data item is used to represent the similarity value between the boundary pixel of the area to be repaired and the known pixels in the adjacent known area, and the confidence is used to represent the degree of trust in the boundary pixel value of the area to be repaired; according to the relationship between the confidence and the preset numerical value, a preset formula combining the confidence and the data item is selected, and the preset formula is used to calculate the repair priority of each boundary pixel in the linked list.

[0111] The following is a preset formula for calculating the repair priority of the boundary pixels of the area to be repaired:

[0112] Assume that P(i) is the restoration priority of boundary pixel i, D(i) is the data item of boundary pixel i, and C(i) is the confidence of boundary pixel i. The preset formula can be expressed as: P(i) = f(D(i), C(i)) where the function f can be in the following form: f(D(i), C(i)) = α·C(i)+(1-α)·D(i) Here, α is a weight coefficient between 0 and 1, which is used to balance the influence of confidence and data items on the restoration priority. The specific value can be adjusted according to the actual situation. Data item D(i): represents the similarity value between boundary pixel i and the known pixels in its adjacent known area. The similarity value can be calculated by comparing the color, brightness, texture and other features of the pixel. For example, the data item can be calculated using the following formula: D(i) = 1 / (1+d(i,j)) where d(i,j) is the distance measure (e.g., Euclidean distance) between boundary pixel i and its nearest known area pixel j. Confidence C(i): represents the degree of trust in the value of boundary pixel i in the area to be restored. The confidence can be based on the visibility, stability, or reliability score of the pixel given by some image processing algorithm. For example, the confidence can be calculated using the following formula: C(i) = 1 / (1+σ(i)) where σ(i) is the uncertainty or noise level of the boundary pixel i. Substituting these values ​​into the preset formula, the restoration priority P(i) of each boundary pixel can be obtained. In practical applications, the parameters and calculation methods in the formula can be adjusted according to the specific situation to achieve the best restoration effect.

[0113] Step 302: After identifying the multi-category target pixel with the maximum restoration priority, analyzing the region type where the multi-category target pixel is located.

[0114] After the restoration priority of all boundary pixels is obtained in step 301, the multi-category target pixel with the maximum restoration priority is identified. This pixel will be the starting point for the next restoration process. In this step, the region type where the multi-category target pixel is located is divided into a flat region, an edge region, and a corner region.

[0115] Specifically, the purpose of doing this is that if the multi-category target pixels are located in a flat area, it usually means that the texture and color changes in the area are small, and a smaller image block can be selected for repair. If the multi-category target pixels are located in an edge area, the continuity of the edge needs to be considered during repair. It may be necessary to select a larger image block that can cover the edge and use an edge-guided repair algorithm. If the multi-category target pixels are located in an edge area, the continuity of the edge needs to be considered during repair. It may be necessary to select a larger image block that can cover the edge and use an edge-guided repair algorithm.

[0116] Such analysis ensures that restoration work is more accurate while maintaining the naturalness and continuity of the image.

[0117] Step 303: According to the correspondence between the preset region type and the size of the image block to be restored, the size of the image block to be restored is determined based on the region type where the multi-category target pixel points are located.

[0118] Since the region category where the multi-category target pixels are located has been determined in step 302, in this step, the size of the image block to be restored can be determined based on the correspondence between the region category and the size of the image block to be restored.

[0119] Specifically, first, the correspondence between the region type and the size of the image block to be restored needs to be preset according to the requirements of the restoration algorithm and the image characteristics. This correspondence is usually determined based on experience and experimental results. In step 302, the region type (flat region, edge region or corner region) where the multi-category target pixels are located has been determined.

[0120] Next, the preset correspondence table is queried based on this information. Based on the query results, the size of the image block to be repaired is determined for the type of area where the multi-category target pixels are located. The following are examples of possible correspondences: Flat area: Since flat areas do not change much, a smaller image block size can be selected, such as 3x3 or 5x5 pixels. Edge area: The edge area requires more contextual information to maintain the continuity of the edge, so a medium-sized image block can be selected, such as 7x7 or 9x9 pixels. Corner area: The information contained in the corner area is the most complex, and a larger image block, such as 11x11 or larger, may be required to ensure that the features of the corner points are properly preserved.

[0121] After the image block size is determined, the image block is extracted from the area to be repaired according to the size, with the multi-category target pixel as the center. This image block will be used in the subsequent search and matching process.

[0122] In this way, step 303 ensures that each area to be repaired can select the most appropriate image block size according to its specific characteristics, so that the structure and details of the image can be more effectively maintained in the subsequent repair process.

[0123] Step 304: Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block with the highest similarity to the image block to be repaired in a known area around the area to be repaired.

[0124] In this step, since the size of the image block to be repaired has been constructed in step 303, it is necessary to find a matching block that can match the image block to be repaired in the surrounding known area. When determining the search area, that is, the known area around the area to be repaired, this area should be large enough to ensure that a matching block with a high similarity to the image block to be repaired can be found. A local search strategy is adopted, which means that the search will be concentrated on the area around the multi-category target pixel, rather than the entire image. This strategy can reduce the amount of calculation and improve the search efficiency. For the image block to be repaired centered on the multi-category target pixel, calculate the similarity between it and the candidate matching block in the known area. The similarity calculation can be based on a variety of factors, such as color, texture, structure, etc. For the image block to be repaired centered on the multi-category target pixel, calculate the similarity between it and the candidate matching block in the known area. The similarity calculation can be based on a variety of factors, such as color, texture, structure, etc.

[0125] The specific implementation process can be found as follows:

[0126] Step 1: Get the length and width of the remote sensing image to be restored, take the center of the image block to be restored as the base point, set the search radius, and take the smaller value of the length and width of the remote sensing image to be restored, 1 / n, where n is a preset positive integer.

[0127] In this step, the size information of the remote sensing image to be restored, that is, the length and width of the image, is obtained. Then, taking the center of the image block to be restored as the base point, an initial search radius is set according to the size of the image. This search radius is usually 1 / n of the smaller value of the image length and width, where n is a preset positive integer. The value of n can be adjusted according to actual conditions to balance the search efficiency and matching accuracy. Setting the search radius is the key to ensuring that the search area is large enough to find high-quality matching blocks. By taking 1 / n of the smaller value of the image length and width as the initial radius, it can be ensured that the search area is neither too large to cause a sharp increase in calculations, nor too small to find enough matching blocks.

[0128] Step 2: Based on the set search radius, check whether there is a matching block in the search area whose similarity with the image block to be repaired meets the matching requirements;

[0129] In this step, within the set search radius, all candidate matching blocks are traversed to calculate the similarity between them and the image block to be repaired. The similarity calculation can be based on multiple factors such as color, texture, structure, etc. In this embodiment, based on the traditional criminisi algorithm, a spatial feature, namely distance similarity, is introduced to measure the similarity between image blocks, and the best matching block is selected for filling according to the matching principle of the following formula:

[0130] ψ q′ =argminSM(ψ p ,ψ q ),ψ q ∈φ

[0131] SM(ψ p ,ψ q )=μ*SSD(ψ p ,ψ q )+ν*Dist(ψ p ,ψ q )

[0132] SSD(ψ p ,ψ q )=∑[(I r -I′ r ) 2 +(I g -I′ g ) 2 +(I b -I′ b ) 2 ]

[0133]

[0134] Where μ and ν are constants, μ+ν=1. φ represents the intact area. p and q represent the pixel positions of the area to be repaired and the candidate matching block, ψ p ,ψ q They represent the image block and candidate matching block of the area to be repaired, respectively, q′ is the best candidate matching block. r ,I g ,I b is the pixel ″′ in the multi-category target block

[0135] RGB value, I r ,I g ,I b is the RGB value of the pixel in the matching sample block. i and B i It is a vector of multi-category target image blocks and blocks to be matched.

[0136] If a matching block whose similarity meets the matching requirements is found, the next step, i.e., step 305, is entered. The local search strategy plays a key role in this step. By searching for matching blocks in a smaller search area, the amount of calculation can be significantly reduced and the search efficiency can be improved. At the same time, the calculation of similarity is the basis for ensuring that high-quality matching blocks are found.

[0137] Step 3: Increase the search radius until at least one matching block whose similarity with the image block to be repaired meets the matching requirements is found in the search area, and determine the matching block with the highest similarity as the multi-category target matching block.

[0138] In this step, if no matching block that meets the matching requirements is found within the initial search radius, the search radius is gradually increased and the search process of step 2 is repeated. Until at least one matching block that meets the matching requirements is found in the search area. Then, the one with the highest similarity is selected from these matching blocks that meet the requirements as the multi-category target matching block. By gradually increasing the search radius, it is possible to ensure that matching blocks are found in a larger range, thereby increasing the chance of finding high-quality matching blocks. At the same time, selecting the matching block with the highest similarity as the multi-category target matching block can ensure that the restored image is consistent with the surrounding area in terms of color and texture.

[0139] In summary, step 304 efficiently searches for multi-category target matching blocks with the highest similarity to the image block to be repaired in the surrounding known area by setting a reasonable search area, adopting a local search strategy, and gradually increasing the search radius. This provides an important reference and basis for subsequent image restoration work.

[0140] Step 305: fill the multi-category target matching block into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after repair.

[0141] In this step, the main task is to fill the previously found multi-category target matching blocks (i.e. the known area blocks with the highest similarity to the image blocks to be repaired) to the location of the image blocks to be repaired. The purpose of this is to make the repaired image blocks consistent with the surrounding known areas in terms of color, texture, structure, etc., so as to achieve visual coherence and naturalness.

[0142] First, the position of the image block to be restored in the image needs to be accurately located, which is usually achieved by determining the coordinates of the image block to be restored in step 303 .

[0143] Next, the previously found multi-category target matching block is extracted from the known region. This matching block is found by the local search strategy in step 304 and has the highest similarity with the image block to be repaired.

[0144] Then, the extracted multi-category target matching block is filled into the position of the image block to be repaired. This step usually involves pixel-level operations, that is, copying each pixel value in the multi-category target matching block to the corresponding position of the image block to be repaired.

[0145] In some cases, in order to further improve the restoration effect, it may be necessary to perform transition processing on the filled edge. This can be achieved by smoothing filtering, gradient processing or other image processing techniques to reduce the abruptness between the filled area and the surrounding known area.

[0146] Finally, the restored image needs to be verified to ensure that the restoration effect is as expected. This can be achieved by visual inspection, calculating the similarity between the restored area and the surrounding area, or using other quality assessment metrics.

[0147] Through this step, the position of the image block to be repaired can be filled with known area blocks similar to the multi-category target matching blocks, so that the repaired image is more visually coherent and natural. Figure 7 As shown, the application Figure 3 The effect comparison of the dynamic classification corresponding removal strategy proposed in the above is shown in Figure 2. This strategy performs particularly well in dealing with environments with dynamic characteristics such as water areas, and can effectively remove targets such as ships. It is worth noting that the "dynamic classification corresponding removal strategy" in this embodiment is not limited to removing Figure 7 The example is not limited to the ships, but is applicable to multiple categories of target removal.

[0148] Further, for step 3 in step 204 above, if the category corresponding to the maximum total weight value is a static category, in order to better describe the specific process of how the removal strategy corresponding to the category repairs the area to be repaired, such as Figure 4 As shown, an application process for the removal strategy corresponding to static classification is provided:

[0149] Step 401: Obtain the priority of each environmental characteristic on the visual restoration effect in the static classification, and identify the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect.

[0150] In this step, first, all relevant environmental characteristics information in static classification needs to be collected. These environmental characteristics may include color, texture, texture, etc.

[0151] Next, it is necessary to evaluate the specific impact of each environmental characteristic on the visual restoration effect. This can be achieved through experimental analysis, expert evaluation or historical data comparison. During the evaluation process, factors such as the importance of the environmental characteristic in the restoration process, its contribution to the restoration effect, and its interaction with other environmental characteristics need to be considered. Based on the evaluation results, a priority can be assigned to each environmental characteristic. Priority is usually a numerical value or a level that indicates the importance of the environmental characteristic in the visual restoration process. Priority can be determined based on a variety of factors, such as the degree of impact of the environmental characteristic, the necessity of the restoration process, the degree of improvement in the restoration effect, etc.

[0152] Finally, the environment feature with the highest priority is identified as the first environment feature that has the greatest impact on the visual restoration effect. This feature is usually the one that needs to be considered and processed most during the restoration process.

[0153] Through this step, the priority of each environmental characteristic in the static classification on the visual restoration effect can be clarified, and the first environmental characteristic with the greatest impact on the visual restoration effect can be determined, providing an important reference for subsequent processing steps.

[0154] Step 402: extracting first environmental characteristics in the remote sensing area to be repaired and the surrounding known areas.

[0155] In this step, first, the remote sensing area to be repaired and the surrounding known areas need to be accurately located. This is usually achieved through preprocessing steps of remote sensing images (such as image registration, segmentation, etc.). The area to be repaired is the part of the image that needs to be repaired or replaced, and the surrounding known areas are the intact or less damaged parts used to provide repair references.

[0156] Next, it is necessary to select a method suitable for extracting the first environmental characteristic. This depends on the specific type of the first environmental characteristic, such as color, texture, structure, etc. For example, if the first environmental characteristic is color, color space conversion, color histogram statistics, etc. can be used to extract color information. If the first environmental characteristic is texture, texture analysis, texture synthesis, etc. can be used to extract texture features.

[0157] After selecting a suitable extraction method, the first environmental characteristics are extracted from the area to be repaired and the surrounding known area respectively.

[0158] Finally, it is necessary to verify whether the extracted first environmental characteristics are accurate, complete, and meet the requirements of subsequent processing steps. This can be achieved by comparing the extracted results with the original image, using feature matching algorithms, etc.

[0159] Through this step, the first environmental characteristics can be extracted from the remote sensing area to be restored and the surrounding known areas, providing important input information for the subsequent step 403 using a convolutional neural network to perform image restoration.

[0160] Step 403: Input the extracted first environmental characteristics into a convolutional neural network, where the convolutional neural network has a large receptive field characteristic.

[0161] In this step, the large receptive field feature is used to capture the information of the input image and predict and generate the missing image content.

[0162] First, the extracted first environmental characteristics need to be organized into a format suitable for convolutional neural network input. This may include steps such as adjusting the data size, normalization, and data enhancement. The input data usually includes the first environmental characteristics information of the area to be repaired and its surrounding known areas, which will be used as the input of the convolutional neural network.

[0163] Next, you need to select a convolutional neural network model with a large receptive field. A large receptive field means that the network can capture a wider range of input image information, which is crucial for predicting and generating missing image content. Common convolutional neural network models include VGG, ResNet, Inception, etc., but the specific model to choose depends on the complexity of the task, the availability of computing resources, and the requirements for model performance.

[0164] After selecting a suitable convolutional neural network model, you need to configure the network's parameters, such as learning rate, batch size, number of iterations, etc. These parameters will affect the network's training effect and convergence speed. Input the prepared input data into the convolutional neural network and start the training process. During the training process, the network will learn how to predict and generate the missing image content from the input first environmental characteristics. The training process may involve multiple iterations, and in each cycle the network will adjust its weights and bias parameters according to the error calculated by the loss function.

[0165] Step 404: Verify whether the image content is visually consistent with the surrounding known area and meets the restoration standard.

[0166] After the training in step 403 is completed, in this step, the performance of the network needs to be verified. This can be achieved by applying the network to a validation dataset and evaluating its restoration effect. Verification indicators may include visual inspection, calculation of image similarity indicators (such as PSNR, SSIM, etc.), comparison of the difference between the repaired area and the surrounding known area, etc. Once the network training is completed and its performance is verified, it can be used to generate the missing image content. This usually involves inputting the first environmental characteristics of the area to be repaired into the network and outputting the predicted image content.

[0167] Step 405: Gradually add other environmental characteristics according to priority as input, and repeat the neural network processing until the repair effect meets the preset requirements.

[0168] When the verification result of step 404 is negative, further measures need to be taken to improve the restoration effect, including: gradually adding other environmental characteristics in order of priority as input, which are determined in step 401, but initially only the first environmental characteristic is used. Repeat the neural network processing process of step 403, input the new combination of environmental characteristics into the network, and generate a new restoration image. Repeat the verification process of step 404 (implicitly) to evaluate whether the newly generated restoration image meets the preset requirements. This iterative process will continue until the restoration effect meets the preset quality requirements or cannot be improved by adding environmental characteristics.

[0169] Through this iterative process, the restoration effect can be gradually improved until the preset quality requirements are met. This method makes full use of the priority information of environmental characteristics and can minimize the consumption of computing resources while ensuring the restoration quality.

[0170] like Figure 8 As shown, the application Figure 4 The static classification corresponding removal strategy proposed in the previous section is compared. This strategy performs particularly well in dealing with environments with static characteristics such as airports, and can effectively remove targets such as Figure 8 The example is not for the aircraft, but for a variety of target removals.

[0171] Further, for the stitching process in step 104, in order to better describe how to stitch the segmented area to be repaired, as shown below, a process for selecting the corresponding image stitching algorithm according to the removal strategy is provided: Step 1: When the area to be repaired is divided into multiple intermediate areas, identify the same multi-category target spanning multiple intermediate areas. Step 2: At the junction of multiple intermediate areas, determine the removal strategy corresponding to the multiple intermediate areas that constitute the junction of the areas. Step 3: According to the corresponding removal strategy, select the corresponding image stitching algorithm to stitch the multiple intermediate areas, generate a smooth intermediate image, and update the original image data. Among them, the stitching algorithm is selected according to the removal strategy, including but not limited to the following selection methods: If the removal strategy is to simply remove the boundary or edge part, then it may be necessary to use a stitching technology that can handle obvious boundaries, such as a method based on graph cuts. If the removal strategy is to achieve smooth transition through gradient or other methods, then it may be appropriate to adopt a stitching method based on weighted average, multi-scale decomposition and synthesis, or content-awareness.

[0172] Of course, in addition to selecting the corresponding image stitching algorithm according to the removal strategy, the image stitching algorithm can also be selected according to the target characteristics, including but not limited to the following selection methods: For scenes with high requirements for texture and color consistency (such as natural landscapes), it may be more appropriate to use stitching techniques based on multi-resolution analysis, such as Laplace pyramid blending. When it is important to maintain the consistency of fine structures, such as in medical images, you can consider using a stitching method based on feature point matching to ensure that the fine structure is not lost or deformed due to stitching. Select the image stitching algorithm according to the characteristics of the junction, including but not limited to the following selection methods: If the adjacent regions are very similar, you can consider a simple linear blending or a stitching method based on color correction. For regions with large differences, more complex stitching methods may be required, such as image fusion models based on deep learning, which can learn how to best connect regions with different characteristics. Select the image stitching algorithm according to the algorithm complexity and performance requirements, including but not limited to the following selection methods: For applications with real-time processing requirements, you may need to choose a computationally efficient algorithm, such as a stitching method based on fast Fourier transform (FFT). If the accuracy requirement is extremely high, you can sacrifice some computational efficiency and choose a more accurate but more computationally intensive algorithm, such as a stitching method based on global optimization.

[0173] This series of steps constitutes a closed-loop feedback mechanism to ensure that the restoration effect of remote sensing images meets the expected standards. By continuously evaluating and adjusting the removal strategy, the restoration results can be gradually optimized until specific quality or performance requirements are met.

[0174] Further, based on the same inventive concept, as a Figure 1 In order to realize the method embodiment, the embodiment of the present invention also provides a device for automatically removing multiple categories of targets from remote sensing images. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 5 As shown, the device includes: an acquisition unit 51, an analysis unit 52, a removal unit 53 and a splicing unit 54.

[0175] An acquisition unit 51 is used to acquire a remote sensing image to be restored, where the remote sensing image to be restored includes a region to be restored that covers multiple categories of targets to be removed;

[0176] A determination unit 52, used to determine the environmental characteristics of the known area around the area to be restored in the remote sensing image to be restored in the acquisition unit 51;

[0177] A removal unit 53 is used to perform multi-category target removal processing on the area to be repaired according to the removal strategy corresponding to the environmental characteristics in the determination unit 52, wherein the removal strategy at least includes repairing the area to be repaired using images in a known area around the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network according to the input environmental characteristics;

[0178] The splicing unit 54 is used to splice the area to be repaired after the removal process in the removal unit 53 if the area to be repaired is segmented.

[0179] Furthermore, the embodiment of the present invention also provides a remote sensing image multi-category target automatic removal device for the above Figure 1-4 The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Figure 6 As shown, the device includes: a receiving unit 51, a matching unit 52, a selecting unit 53 and a splicing unit 54.

[0180] In an optional implementation, the analysis unit 52 includes:

[0181] The traversal module 521 is used to traverse the area to be repaired and obtain the corresponding four-dimensional range;

[0182] An acquisition module 522 is used to buffer N pixels outward according to the four ranges in the traversal module 521 to acquire an image block to be processed;

[0183] A first determination module 523, used to determine the environmental characteristics of the known area in the image block to be processed in the acquisition module 522;

[0184] The calculation module 524 is used to classify the multiple environmental characteristics according to the preset characteristic classification standard if it is determined in the first determination module 523 that the area to be repaired has multiple environmental characteristics, and respectively calculate the total weight value of all environmental characteristics in each classification;

[0185] The comparison module 525 is used to compare the total weight values ​​in the calculation module 524 and select a removal strategy according to the difference between the first two categories with the largest total weight values.

[0186] In an optional implementation manner, the first two categories with the largest total weight values ​​in the comparison module 525 are dynamic and static categories, and the comparison module 525 includes:

[0187] A calculation submodule 5251 is used to calculate the total weight values ​​of all environmental characteristics in the dynamic classification and the static classification respectively;

[0188] The judgment submodule 5252 is used to calculate the difference between the total weight values ​​corresponding to the dynamic classification and the static classification in the calculation submodule 5251, and judge whether the difference is greater than or equal to a threshold value, and the threshold value is used to determine the dominant position of the dynamic classification and the static classification in selecting the removal strategy for the repair area;

[0189] The first selection submodule 5253 is used to select a removal strategy according to the category corresponding to the maximum total weight value if the judgment result of the judgment submodule 5253 is yes;

[0190] The second selection submodule 5254 is used to divide the area to be repaired into multiple sub-areas within the area to be repaired if the judgment result of the judgment submodule 5253 is no, determine the category to which each sub-area belongs based on all environmental characteristics within each sub-area, and select the removal strategy corresponding to each sub-area based on the classification.

[0191] Among them, when the classification corresponding to the maximum total weight value in the first selection submodule 5253 is a dynamic classification, a multi-category target removal process is performed on the area to be repaired according to the removal strategy corresponding to the environmental characteristics, including: obtaining the repair priority of each boundary pixel of the area to be repaired; after identifying the multi-category target pixel with the maximum repair priority, analyzing the area type where the multi-category target pixel is located, the type includes a flat area, an edge area and a corner area; according to the correspondence between the preset area type and the size of the image block to be repaired, the size of the image block to be repaired that should be constructed is determined based on the area type where the multi-category target pixel is located; based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block with the highest similarity to the image block to be repaired in the known area around the area to be repaired; and the multi-category target matching block is filled into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after repair.

[0192] Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for the multi-category target matching block with the highest similarity to the image block to be repaired in the known area around the area to be repaired, including: obtaining the length and width of the remote sensing image to be repaired, taking the center of the image block to be repaired as the base point, setting a search radius, and the radius is 1 / n of the smaller value of the length and width of the remote sensing image to be repaired, where n is a preset positive integer; based on the set search radius, checking whether there is a matching block in the search area whose similarity meets the matching requirements with the image block to be repaired; if not, increasing the search radius until at least one matching block whose similarity meets the matching requirements with the image block to be repaired is found in the search area, and determining the matching block with the highest similarity as the multi-category target matching block.

[0193] Among them, when the classification corresponding to the maximum total weight value in the first selection submodule 5253 is a static classification, multi-category target removal processing is performed on the area to be repaired according to the removal strategy corresponding to the environmental characteristics, including: obtaining the priority of each environmental characteristic in the static classification on the visual restoration effect, and confirming the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect; extracting the first environmental characteristic in the remote sensing area to be repaired and the surrounding known area; inputting the extracted first environmental characteristic into a convolutional neural network, which has a large receptive field characteristic and is used to capture the information of the input image, predict and generate missing image content; verifying whether the image content is visually consistent with the surrounding known area and meets the restoration standard; if not, gradually adding other environmental characteristics according to priority as input, and repeating the neural network processing until the restoration effect meets the preset requirements.

[0194] In an optional implementation, the splicing unit 54 includes:

[0195] An identification module 541 is used to identify the same multi-category target spanning multiple middle regions when the area to be repaired is divided into multiple middle regions;

[0196] The second determination module 542 is used to determine the removal strategies corresponding to the plurality of middle regions constituting the region junction at the region junction of the plurality of middle regions in the identification module 541;

[0197] The stitching module 543 is used to select a corresponding image stitching algorithm to stitch multiple intermediate areas according to the corresponding removal strategy in the second determination module 542, generate a smooth intermediate image, and update the original image data.

[0198] Furthermore, an embodiment of the present invention also provides an electronic device, which includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute a method for automatically removing multi-category targets from remote sensing images.

[0199] Furthermore, an embodiment of the present invention also provides a readable storage medium, which is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute a method for automatically removing multi-category targets from remote sensing images.

[0200] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for automatically removing multi-category targets from remote sensing images, characterized in that: The method comprises: Acquire a remote sensing image to be restored, wherein the remote sensing image to be restored includes a region to be restored covering multiple categories of objects to be removed; Determining environmental characteristics of known areas around the area to be restored in the remote sensing image to be restored; According to the removal strategy corresponding to the environmental characteristics, the area to be repaired is subjected to multi-category target removal processing, wherein the removal strategy at least includes repairing the area to be repaired using images in a known area around the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network according to the input environmental characteristics; If the area to be repaired is segmented, the area to be repaired after the removal process is performed with a splicing process.

2. The method according to claim 1, characterized in that Determining the environmental characteristics of the known area around the area to be restored in the remote sensing image to be restored, including: Traverse the area to be repaired and obtain the corresponding four-boundary range; Buffer N pixels outward according to the four-to-range to obtain an image block to be processed; Determining environmental characteristics of a known area in the image block to be processed; If there are multiple environmental characteristics in the known area, the multiple environmental characteristics are classified according to a preset characteristic classification standard, and the total weight value of all the environmental characteristics in each classification is calculated respectively; The total weight values ​​are compared, and the removal strategy is selected according to the difference between the first two categories with the largest total weight value.

3. The method according to claim 2, characterized in that If the first two categories with the largest total weight value are dynamic and static categories, dynamic categories refer to characteristics that are easily changed with time or external conditions, and static categories refer to relatively stable characteristics; The comparing the total weight values ​​and selecting the removal strategy according to the difference between the first two categories with the largest total weight value includes: Calculate the total weight values ​​of all environmental characteristics in the dynamic classification and the static classification respectively; Calculating the difference between the total weight values ​​corresponding to the dynamic classification and the static classification, and determining whether the difference is greater than or equal to a threshold, wherein the threshold is used to determine the dominant position of the dynamic classification and the static classification in selecting a removal strategy for the repaired area; If yes, then selecting the removal strategy according to the category corresponding to the maximum total weight value; If not, then within the area to be repaired, divide the area to be repaired into multiple sub-areas, determine the category to which each sub-area belongs based on all the environmental characteristics within each sub-area, and select the removal strategy corresponding to each sub-area based on the classification.

4. The method according to claim 3, characterized in that When the classification corresponding to the maximum total weight value is a dynamic classification; the removal strategy corresponding to the environmental characteristics is used to remove multiple categories of targets from the area to be repaired, including: Obtaining the repair priority of each boundary pixel of the area to be repaired; After identifying the multi-category target pixels with the maximum restoration priority, analyzing the region types where the multi-category target pixels are located, the regions including flat regions, edge regions, and corner regions; According to the correspondence between the preset region type and the size of the image block to be restored, the size of the image block to be restored is determined based on the region type where the multi-category target pixel points are located; Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block having the highest similarity with the image block to be repaired in a known area around the area to be repaired; The multi-category target matching block is filled into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after being repaired.

5. The method according to claim 4, characterized in that Based on the image block to be repaired with the multi-category target pixel as the center, a local search strategy is adopted to search for a multi-category target matching block having the highest similarity with the image block to be repaired in a known area around the area to be repaired, including: The length and width of the remote sensing image to be restored are obtained, and a search radius is set with the center of the image block to be restored as the base point. The radius is 1 / n of the smaller value of the length and width of the remote sensing image to be restored, where n is a preset positive integer; Based on the set search radius, checking whether there is a matching block in the search area whose similarity with the image block to be repaired meets the matching requirement; If not, the search radius is increased until at least one matching block whose similarity with the image block to be restored meets the matching requirement is found in the search area, and the matching block with the highest similarity is determined as the multi-category target matching block.

6. The method according to claim 3, characterized in that When the classification corresponding to the maximum total weight value is a static classification; the removal strategy corresponding to the environmental characteristics is used to remove multiple categories of targets from the area to be repaired, including: Obtaining the priority of each environmental characteristic in the static classification on the visual restoration effect, and identifying the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect; Extracting first environmental characteristics from the remote sensing area to be repaired and the surrounding known areas; Inputting the extracted first environmental characteristics into a convolutional neural network, wherein the convolutional neural network has a large receptive field characteristic and is used to capture information prediction of the input image and generate missing image content; Verify that the image content is visually consistent with the surrounding known area and meets the restoration standards; If not, other environmental characteristics according to priority are gradually added as inputs, and the neural network processing is repeated until the repair effect meets the preset requirements.

7. The method according to claim 1, characterized in that If the area to be repaired is segmented, the area to be repaired after the removal process is performed a splicing process, including: When the area to be repaired is divided into a plurality of intermediate areas, identifying that the same multi-category target spans across the plurality of intermediate areas; At the region boundary of the plurality of the middle regions, determining removal strategies corresponding to the plurality of the middle regions constituting the region boundary; According to the corresponding removal strategy, a corresponding image stitching algorithm is selected to stitch the plurality of intermediate areas to generate a smooth intermediate image, and the original image data is updated.

8. A remote sensing image multi-category target automatic removal device, characterized in that: The device comprises: An acquisition unit, used for acquiring a remote sensing image to be restored, wherein the remote sensing image to be restored includes a region to be restored covering multiple categories of targets to be removed; A determination unit, used to determine the environmental characteristics of a known area around the area to be restored in the remote sensing image to be restored in the acquisition unit; a removal unit, configured to perform multi-category target removal processing on the area to be repaired according to a removal strategy corresponding to the environmental characteristics in the determination unit, wherein the removal strategy at least includes repairing the area to be repaired using images in a known area around the area to be repaired and repairing an output image of the area to be repaired using a convolutional neural network according to input environmental characteristics; The splicing unit is used to splice the area to be repaired after the removal process in the removal unit if the area to be repaired is divided.

9. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call the program instructions in the memory to execute the method for automatic removal of multi-category targets in remote sensing images as described in any one of claims 1-7.

10. A readable storage medium, characterized in that: The storage medium is used to store a computer program, wherein when the computer program is running, it controls the device where the storage medium is located to execute the method for automatically removing multi-category targets from remote sensing images as described in any one of claims 1 to 7.

Images