A method and device for automatically removing multi-category targets from remote sensing images

Abstract

The present invention provides a method and device for automatically removing multi-category targets from remote sensing images, which relates to the technical field of remote sensing image processing. The main purpose is to solve the technical problem that existing multi-category target removal methods for remote sensing images cannot implement a removal strategy based on the environmental characteristics of the multi-category targets. The main technical solution of the present invention is as follows: obtaining a remote sensing image to be repaired, the remote sensing image to be repaired containing a repaired area covering multiple categories to be removed; determining the environmental characteristics of known areas surrounding the repaired area in the repaired remote sensing image; performing multi-category target removal processing on the repaired area according to the removal strategy corresponding to the environmental characteristics; and if the repaired area is segmented, performing a splicing process on the repaired area after the removal processing.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for automatically removing multi-category targets from remote sensing images. Background Art

[0002] Remote sensing technology is widely used in fields such as environmental monitoring, agricultural management, and disaster assessment. Imagery acquired through platforms like satellites and drones provides a wealth of geographic information. However, in remote sensing image processing, especially when processing data in highly sensitive areas such as national security and military secrets, directly disclosing remote sensing imagery undoubtedly exposes potential security risks. Furthermore, during remote sensing image modeling, various objects (such as bridges, clouds, and vehicles) can interfere with the modeling process. Therefore, multi-class object removal has become a critical step in remote sensing image processing.

[0003] Current technologies employ differentiated processing strategies for removing multi-category targets from remote sensing imagery, with a particular focus on distinguishing between targets in water and land areas. During the processing, a clear distinction is first made between water and land. Then, based on the specific area where the target is located, a removal method is selected that matches the characteristics of that area. For example, for targets in water, removal must fully consider the dynamic characteristics of the water, such as fluidity and unique reflective effects. For targets on land, attention must be paid to the static characteristics of the surface, such as texture complexity and color diversity, to ensure that the removed target maintains a high degree of harmony and consistency with its respective environment.

[0004] However, although the current remote sensing image target processing method can effectively repair the image to a certain extent, it mainly focuses on the removal strategy at the macro level, that is, according to the overall category of the target area (such as water or land), the target is removed according to the preset rules. However, this method ignores the microscopic characteristics of the specific environment in which the multi-category target is located. For example, when a target is located in a land area, but its environment exhibits reflective characteristics similar to those of a water body, if the removal method for the land environment is continued to be used, the processing effect is often unsatisfactory, resulting in the processed remote sensing image appearing unnatural and uncoordinated in environmental fusion, thereby damaging the practicality and aesthetics of the image. Therefore, how to implement a removal strategy based on the environmental characteristics of the multi-category target to achieve high-quality multi-category target removal in remote sensing images has become a key technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the above problems, the main purpose of the present invention is to provide a method and device for automatically removing multi-category targets from remote sensing images, so as to solve the technical problem that the existing multi-category target removal methods of remote sensing images mainly perform multi-category target removal based on macro-region categories, ignoring the microscopic characteristics of the environment in which the multi-category targets are located, resulting in the processed remote sensing images appearing unnatural and uncoordinated in environmental fusion, that is, it is impossible to implement a removal strategy based on the environmental characteristics of the multi-category targets.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a method for automatically removing multi-category targets from remote sensing images, the method comprising:

[0008] Acquire a remote sensing image to be restored, where the remote sensing image to be restored includes an area to be restored that covers multiple categories of targets to be removed;

[0009] Determine the environmental characteristics of the known areas surrounding the area to be restored in the remote sensing image;

[0010] Perform multi-category target removal on the area to be repaired based on a removal strategy corresponding to the environmental characteristics. The removal strategy at least includes repairing the area to be repaired using images within a known area surrounding the area to be repaired and repairing the output image of the area to be repaired based on the input environmental characteristics using a convolutional neural network.

[0011] If the area to be repaired is segmented, the area to be repaired after the removal process is performed on the segmented area.

[0012] A second aspect of the present invention provides a device for automatically removing multiple categories of objects from remote sensing images, the device comprising:

[0013] An acquisition unit is used to acquire a remote sensing image to be restored, where the remote sensing image to be restored includes an area to be restored that covers multiple categories of targets to be removed;

[0014] a determination unit, for determining environmental characteristics of a known area surrounding the area to be restored in the remote sensing image to be restored in the acquisition unit;

[0015] a removal unit, configured to perform multi-category target removal processing on the area to be repaired based on a removal strategy corresponding to the environmental characteristics in the determination unit, wherein the removal strategy at least includes repairing the area to be repaired using images within a known area surrounding the area to be repaired and repairing the output image of the area to be repaired based on the input environmental characteristics using a convolutional neural network;

[0016] The splicing unit is used to splice the area to be repaired after the removal process in the removal unit if the area to be repaired is divided.

[0017] The third aspect of the present invention provides an electronic device, which includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute a method for automatically removing multi-category targets in remote sensing images.

[0018] A fourth aspect of the present invention provides a readable storage medium, which is used to store a computer program. When the computer program is running, it controls the device where the storage medium is located to execute a method for automatically removing multi-category targets from remote sensing images.

[0019] Based on the above technical solution, the present invention proposes a method and apparatus for automatically removing multi-category objects from remote sensing images. The core of this method lies in flexibly selecting the most appropriate removal method for the area to be removed based on the specific characteristics of the environment in which the multi-category objects are located, thereby effectively concealing the multi-category objects. Specifically, a remote sensing image containing the multi-category objects to be removed is first selected as the "remote sensing image to be repaired." Because the locations of the multi-category objects have been identified and marked in advance, it is clear which areas in the remote sensing image to be repaired require repair, namely the "area to be repaired." Next, the environmental characteristics of the known areas surrounding the area to be repaired in the remote sensing image are determined. Finally, a removal strategy that matches the identified environmental characteristics is selected. The selection of this strategy is crucial, as it guides how to remove the multi-category objects from the area to be repaired, ensuring a natural removal effect that is consistent with the characteristics of the environment. If the area to be repaired is segmented and then the removal strategy is used to remove the multi-category objects, the segmented areas to be repaired must be spliced to reduce splicing inconsistencies. In summary, the method proposed in the present invention can flexibly select the most suitable removal means according to the removal requirements of multiple categories of targets under different environmental characteristics, thereby ensuring that the removal effect is both natural and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 A flowchart of a method for automatically removing multi-category objects from remote sensing images proposed in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a process for selecting a removal strategy based on environmental characteristics, proposed in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a process for applying a dynamic classification corresponding removal strategy proposed in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a process for applying a static classification corresponding removal strategy proposed in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of a device for automatically removing multi-category objects from remote sensing images proposed in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the detailed structure of a device for automatically removing multi-category objects from remote sensing images proposed in an embodiment of the present invention;

[0027] Figure 7 A comparison chart of the effects of a dynamic classification and corresponding removal strategy proposed in an embodiment of the present invention;

[0028] Figure 8 This is a comparison chart of the effects of applying static classification corresponding removal strategies proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0030] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which this application belongs.

[0031] The design inspiration of this invention comes from an in-depth analysis of the limitations of existing remote sensing image multi-category target removal technologies. Specifically, existing methods are mainly based on macro-region categories (such as water areas and land), ignoring the specific environmental characteristics of the multi-category targets. For example, when the target is located in a land area with special reflective characteristics, the removal method of an ordinary land environment will result in poor processing effect, making the image appear unnatural and uncoordinated. In addition, existing methods usually follow preset fixed rules and lack the ability to flexibly adjust according to environmental characteristics, which limits their scope of application and effect. These limitations not only undermine the practicality of the image, but also affect the aesthetics.

[0032] These limitations prompted the inventors to propose a novel technical concept designed to overcome the shortcomings of existing methods. The core of this technical concept lies in precisely identifying the specific environmental characteristics of the known areas surrounding the area to be restored and determining a removal strategy that corresponds to these characteristics. The advantage of this approach lies in its flexibility in selecting the most appropriate removal strategy based on identified microscopic characteristics, rather than simply applying a fixed set of preset rules. This approach ensures a natural removal effect that is consistent with the surrounding environment, thereby improving the practicality and aesthetics of remote sensing image processing.

[0033] In the embodiment of the present invention, a method for automatically removing multi-category targets from remote sensing images is mainly described. The method is executed by a central processing unit (CPU) or a graphics processing unit (GPU). The method obtains a remote sensing image containing multi-category targets to be removed, determines the environmental characteristics of the known area around the area to be repaired, and selects a suitable removal strategy based on these characteristics to process the area to be repaired, thereby achieving the removal of multi-category targets. In short, it is to guide the removal operation of specific multi-category targets in remote sensing images through environmental characteristic analysis. The specific execution steps are as follows: Figure 1 Shown, including:

[0034] Step 101: Acquire the remote sensing image to be restored.

[0035] In this step, the remote sensing image to be modified is an area to be repaired that covers multiple categories of targets to be removed. It should be noted that the area to be repaired not only includes the multi-category target area, but also includes areas that are affected by the multi-category target area and therefore need to be removed, such as multi-category target shadows, so as to ensure a natural removal effect.

[0036] Specifically, before obtaining the remote sensing image to be repaired in this step, the initial remote sensing image needs to be preprocessed. Then, the trained model is applied or fine-tuned to automatically identify the multi-category targets to be removed from the preprocessed image, generate a preliminary multi-category target frame, and then automatically optimize and adjust the multi-category target frame based on the shadow characteristics to generate a precise multi-category target frame. Finally, based on the precise multi-category target frame, a mask of the area to be repaired is generated. Among them, the specific technical process for the identification of multi-category targets to be removed and shadows is as follows:

[0037] In this step, the preprocessed remote sensing image is used as input, and a trained model is used or an existing model is loaded to perform multi-category target recognition and distinguish the multi-category targets to be removed in the remote sensing image, and the multi-category target frames of the multi-category targets to be removed are automatically output.

[0038] After detecting multiple categories of objects, we further identify shadow areas within or immediately adjacent to the bounding box. Based on the specific shape of the shadows (such as direction and length), we intelligently adjust the original bounding box to ensure that the multiple categories of objects and their complete shadows are seamlessly included. The steps are as follows:

[0039] Based on the center of the current multi-class target bounding box, each side of the multi-class target bounding box is expanded outward in parallel by N pixels. The choice of N needs to be adjusted based on actual conditions and needs to be determined through experimentation or prior knowledge. If the expanded multi-class target bounding box exceeds the image boundary, the excess area is considered invalid and excluded from subsequent calculations. The expansion range is determined based on the multi-class target bounding box before and after expansion. The expanded area is divided into four regions and numbered according to the corresponding edges. The corresponding image is obtained based on the four expanded regions. A threshold segmentation method is used to better separate shadows from background through color space conversion. Based on the corresponding shadow region number, the corresponding multi-class target bounding box edge is further expanded outward by N2 pixels. N2 can be a different value from N, depending on the size and characteristics of the shadow. This process of selecting the multi-class target shadow is repeated until no shadow region is detected outside the expanded multi-class target bounding box boundary.

[0040] Overall, in step 101, through a series of operations, a remote sensing image containing the multi-category objects to be removed and their affected areas is successfully acquired. This step provides the necessary premise and foundation for the subsequent multi-category object removal process, ensuring the accuracy and naturalness of the removal operation.

[0041] Step 102: Determine the environmental characteristics of the known areas surrounding the area to be restored in the remote sensing image to be restored.

[0042] This step is a key step in the automatic multi-category object removal method. The purpose of this step is to provide the necessary information basis for the subsequent restoration process, ensuring that the restoration result is as close to the original image as possible and natural.

[0043] The environmental characteristics here refer to the various physical properties, spectral characteristics, texture characteristics, etc. in the known area around the area to be repaired that can describe the area.

[0044] To accurately capture these environmental characteristics, commonly used technical approaches include, but are not limited to: Image processing algorithms: used to extract texture, edges, and other spatial features from images. Machine learning or deep learning models: trained to identify specific types of objects or features. Geographic Information System (GIS) data: leveraging existing geographic data to assist in determining environmental characteristics. Spectral analysis: analyzing spectral information in different bands to distinguish between different objects.

[0045] Once the environmental characteristics are determined, a corresponding removal strategy can be developed based on this information. For example, if the area to be repaired is located in a vegetated area, other healthy vegetation pixels in the area can be used as samples to reconstruct the occluded or damaged part. Alternatively, in the convolutional neural network (CNN) method, inputting environmental characteristics can help the network better understand the context and produce more realistic repair results.

[0046] In summary, step 102 is an important step in ensuring the quality of the restoration. It provides a solid foundation for subsequent restoration work by carefully analyzing and understanding the environmental characteristics of the area to be restored.

[0047] Step 103: Perform multi-category target removal processing on the area to be repaired according to the removal strategy corresponding to the environmental characteristics.

[0048] During the multi-class object removal process, once the environmental characteristics of the area to be repaired are determined in step 102, the next step is to select an appropriate removal strategy for that area based on these characteristics. This strategy includes at least using images from a known area surrounding the area to be repaired and using a convolutional neural network to inpaint the output image of the area to be repaired based on the input environmental characteristics. This step is important because it directly affects the naturalness and fidelity of the image after the removal.

[0049] Consider a specific application scenario: a radar device is located on a smooth, highly reflective surface. Because the radar's deployment involves military secrets, it needs to be removed from remote sensing images. To achieve this, the environmental characteristics must be carefully considered.

[0050] If the chosen removal strategy ignores the highly reflective nature of the radar's support surface in the original remote sensing image, the resulting image may appear unrealistic, failing to accurately depict the relationship between light and shadow between the radar and its support surface. In this case, the shape of the radar base may leave a dull mark in the image, while surrounding it is a highly reflective area. This inconsistency can easily lead to the detection of the radar's presence.

[0051] Therefore, it's necessary to customize the removal strategy based on the specific characteristics of the radar environment to ensure that the image restoration process preserves environmental details, such as reflected light effects, that were present in the original image without the multi-class target removal. This approach not only successfully removes the multi-class target but also preserves the natural texture and lighting effects of its surroundings, making the restored remote sensing image visually natural and coherent, thus achieving highly natural and effective multi-class target removal.

[0052] Depending on the environmental characteristics, there may be multiple options for removal strategies, or only one most appropriate solution. When faced with multiple removal strategies, historical data and usage results can be used to evaluate which strategy is superior.

[0053] Specifically, when environmental characteristics tend to be fluidity, reflectivity, and other characteristics that are easily changed over time or by external conditions, it is important to note that this does not mean that the environmental characteristics in the remote sensing image are constantly changing in real time, but rather that these characteristics belong to the category of dynamic changes. In this case, reference removal strategies include, but are not limited to, selecting the strongest characteristic among these characteristics, then finding a matching image block that is most similar to the strongest characteristic within the known area surrounding the area to be repaired in the remote sensing image to be repaired, and using this image block to repair the area to be repaired.

[0054] When environmental characteristics tend to be texture-based and not easily changeable over time or with external conditions, it's important to note that these characteristics are static. In this case, possible removal strategies include, but are not limited to: Using texture synthesis techniques to generate camouflage textures based on the surrounding texture features and overlay them on multi-category targets for removal. Using image fusion techniques to fuse images of multi-category targets with images of the surrounding environment, visually blending the multi-category targets into the environment.

[0055] When environmental characteristics are biased, such as transparency, lighting conditions, and temperature variations, optional removal strategies include but are not limited to: For environments with varying lighting conditions, use illumination simulation technology to adjust the illumination effects on the surfaces of multiple target categories to match the ambient lighting. For situations where multiple target categories need to be removed under different spectrum conditions, use multi-spectral camouflage materials or technologies to ensure that multiple target categories are difficult to detect in remote sensing images of different spectrums.

[0056] In summary, step 103 effectively achieves multi-category target removal while maintaining the visual integrity of the remote sensing image and the authenticity of the environmental information, achieving the purpose of removing multi-category targets while maintaining the naturalness of the environment.

[0057] Step 104: If the area to be repaired is segmented, the area to be repaired after the removal process is performed a splicing process.

[0058] In this step, it is described how to perform stitching processing to generate the final high-quality remote sensing image after the area to be repaired is divided into multiple intermediate areas and these areas are removed.

[0059] Specifically, if an object spans multiple segmented regions, at the junction of these regions, the same / different removal strategies are used for this object.

[0060] If different removal strategies are used, it is necessary to eliminate the visual discontinuities caused by different removal strategies, such as: decomposing the image into multiple frequency layers, fusing each layer separately, and then re-synthesizing; using pre-trained neural network models to automatically learn how to optimally fuse different processing results; using content-aware fill or texture synthesis techniques to generate naturally transitioned intermediate images; performing local color correction and contrast adjustment to further improve the overall image quality. If the same removal strategy is used, it is necessary to ensure smooth transitions between areas using the same removal strategy, such as: using feature point detection algorithms such as SIFT and SURF to find common feature points in adjacent areas, and then aligning these feature points through geometric transformations; adjusting the colors of adjacent areas to ensure color consistency; using gradient fusion, feathering effects, and other techniques to reduce hard edge phenomena at the boundaries; and using weighted averaging methods at the junction to smooth transitions, such as Laplacian pyramid blending.

[0061] After generating the stitched image using the above steps, perform a comprehensive quality check on the entire stitched image to identify and correct any unnatural edges or color changes. If slight inconsistencies or stitching artifacts are found, perform local adjustments such as color correction, contrast adjustment, and local smoothing to ensure a smooth transition between stitched images without noticeable stitching artifacts. The processed image is then accurately replaced in its original position.

[0062] The processed image is accurately replaced back to its original location and the relevant metadata record is updated to ensure that future users will know that the part has been specially processed.

[0063] Through the above steps, high-quality image stitching can be effectively achieved, thus generating the final high-quality remote sensing image. This method not only improves the technical level of image processing, but also provides a more reliable data foundation for subsequent applications.

[0064] Based on the above Figure 1As can be seen from the implementation method, the present invention proposes a method and apparatus for automatically removing multi-category objects from remote sensing images. The core of this method lies in flexibly selecting the most appropriate removal method for the area to be removed based on the specific characteristics of the environment in which the multi-category objects are located, thereby effectively concealing the multi-category objects. Specifically, a remote sensing image containing the multi-category objects to be removed is first selected as the "remote sensing image to be repaired" to be processed. Since the locations of the multi-category objects have been identified and marked in advance, it is clear which areas in the remote sensing image to be repaired require repair, namely the "area to be repaired." Next, the environmental characteristics of the known areas surrounding the area to be repaired in the remote sensing image are determined. Finally, based on the identified environmental characteristics, a removal strategy that matches them is selected. The selection of this strategy is crucial, as it guides how to remove the multi-category objects in the area to be repaired, ensuring a natural and effective removal effect that is consistent with the characteristics of the environment. In summary, the method proposed in this invention can flexibly select the most appropriate removal method based on the multi-category object removal requirements under different environmental characteristics, thereby ensuring a natural and effective removal effect. If the area to be repaired is segmented and then the removal strategy is used to remove multiple categories of targets, the segmented area to be repaired needs to be spliced to reduce the splicing inconsistency problem.

[0065] Furthermore, in the above embodiment, a method for removing multiple categories of objects has been outlined, which determines a removal strategy based on the characteristics of the environment in which the multiple categories of objects are located. The core of this method is to select or design the most suitable removal strategy based on the identified environmental characteristics. These strategies can adapt to changes in the environment, making the removed area blend in with the surrounding environment and avoiding obvious visual differences. Now, based on the above discussion, as Figure 2 As shown in the figure, it specifically explains how to select the corresponding removal strategy according to the environmental characteristics.

[0066] Step 201: traverse the area to be repaired and obtain the corresponding four-dimensional range.

[0067] This step is used to define the boundaries of the specific area that needs to be processed. The following is a detailed analysis of this step:

[0068] Traversing the in-progress restoration area actually involves accessing the target boxes for each in-progress restoration area within the remote sensing image. Each target box is a rectangular area that has been marked or identified as requiring restoration. By traversing these target boxes, we can fully understand the location and extent of all in-progress restoration areas, ensuring that no areas are missed.

[0069] The four-bounds range refers to the rectangular bounding box formed by the farthest points in the east, south, west, and north directions of all the areas to be repaired. This bounding box corresponds to the positive X and Y axes, even though the individual target boxes may be tilted rectangles or other shapes. By calculating these bounds, a minimum rectangular box can be determined that encompasses all the areas to be repaired, which is used for subsequent fixed-point cropping.

[0070] Accurately determining the boundaries of the area to be inpainted is crucial for subsequent steps, as it directly influences the selection and analysis of surrounding known areas, which in turn influences the selection and application of removal strategies. This ensures the consistency and authenticity of the inpainted result with the original image, avoiding unnatural transitions or erroneous inpainting results.

[0071] Step 202: Buffer N pixels outward according to the range to obtain an image block to be processed.

[0072] This step is intended to ensure that the environmental characteristics of the surrounding area are utilized during the restoration process, thereby improving the authenticity of the restoration.

[0073] Once the boundaries are determined in step 201, the next step is to expand the original rectangular frame by a certain number of pixels (e.g., N pixels). This means adding N pixels to the top, bottom, left, and right of the original rectangular frame. This ensures that the image is not limited to the original area to be repaired, but also includes a certain width of surrounding environment information.

[0074] To ensure the inpainted image smoothly blends into the surrounding known environment, the rectangular frame obtained in the previous step is expanded outward by N pixels to form a new rectangular frame. This new rectangular frame is used to crop the area to be inpainted, resulting in a processed image patch that includes the inpainted area and its surroundings. The value N should be large enough to capture sufficient contextual information, but not too large to introduce unnecessary complexity or interference.

[0075] It should be noted that the specific number of pixels N to expand outward depends on multiple factors, including but not limited to the size and type of the object to be inpainted, as well as the amount of contextual information required. For example, for larger objects or complex scenes, a larger N value may be required to capture more context; for smaller objects or simple scenes, a smaller N value can be selected to reduce the computational burden.

[0076] After completing the above expansion, a new, larger rectangular area will be obtained. This new area is called the "image block to be processed", which includes the original area to be repaired and a part of the known area around it.

[0077] Step 203: Determine the environmental characteristics of the known area in the image block to be processed.

[0078] In the image block to be processed obtained in the above step 202, since there is a part of known area, the environmental characteristics in this part of known area are obtained to guide the subsequent selection and implementation of the removal strategy.

[0079] Extract various environmental characteristics from a known area. This step can be achieved through the following methods: Spectral analysis: Utilizing information from different wavelengths of remote sensing imagery (such as visible light, near-infrared, and short-wave infrared), the spectral reflectance characteristics of land features are analyzed. Different land feature types typically have unique spectral response patterns, and spectral analysis can effectively distinguish vegetation, water bodies, and buildings. Texture analysis: Using image processing algorithms such as gray-level co-occurrence matrices (GLCMs) and local binary patterns (LBPs) to extract texture features within a known area. Texture features reflect the regularity and complexity of land features' surface structures and are very useful for identifying land features such as forests, grasslands, and urban buildings. Shape and spatial relationships: Analyzing the shape characteristics of land features and their spatial distribution patterns. For example, rivers typically have linear distributions, while lakes tend to be closed planar; buildings may have regular rectangular or square layouts. Geographic Information System (GIS) data integration: Integrating existing geographic information data, such as land cover maps and topographic maps, assists in determining land feature types and other environmental characteristics within a known area. GIS data provides rich contextual information, helping to improve the accuracy of environmental feature identification.

[0080] In summary, this step not only provides the necessary background information and support for subsequent restoration work, but also ensures that the restoration process not only relies on the data of the area to be restored itself, but also makes full use of the information of the surrounding environment, thereby improving the quality and authenticity of the restoration effect.

[0081] Step 204: If the area to be repaired has multiple environmental characteristics, the multiple environmental characteristics are classified according to a preset characteristic classification standard, and the total weight value of all environmental characteristics in each classification is calculated respectively.

[0082] In this step, when dealing with the various environmental characteristics of the area to be restored, they must first be classified according to a preset characteristic classification standard. This classification standard should be scientific and reasonable, and able to effectively distinguish between dynamic and static characteristics. Dynamic characteristics typically include light reflection characteristics, water flow ripple characteristics, reflection characteristics, etc. These characteristics are prone to change over time or external conditions. Although they may not appear as actual movement in the image to be restored, they are crucial to maintaining a natural feel between the restored area and the surrounding environment, avoiding unrealistic conditions after restoration, such as no reflected light in the restored area while reflected light is present in the surrounding area.

[0083] Static features include texture and structure, which are relatively stable. When performing regional restoration for these features, the focus is on restoring the true ground texture and structure.

[0084] The weight calculation method can be based on multiple factors such as the distribution frequency of color in the area, the complexity of the texture, and the lighting and reflection characteristics. When setting the weight, each characteristic should be weighted in advance, and the weight corresponding to different characteristics should also be different.

[0085] Next, the environmental characteristics are categorized according to pre-set classification criteria, such as dynamic classification, static classification, regional classification (dividing the area to be repaired into multiple blocks and grouping the characteristics within each block into a category), and other combinations of these. After the classification is complete, the total weight of all characteristics within each category is calculated.

[0086] If multiple classifications exist, a comparison of these categories is necessary to determine which category plays a leading role in the restoration of the area to be remediated, and to select an effective removal strategy corresponding to that category to ensure a natural restoration effect. Of course, there may be situations where the removal strategies corresponding to multiple classifications produce similar results. In this case, the appropriate removal strategy for the specific environmental characteristics must be determined based on the actual situation to achieve natural removal of multiple categories of targets.

[0087] Through this step, the various environmental characteristics of the area to be repaired can be dealt with clearly and systematically, ensuring the naturalness and authenticity of the repair effect.

[0088] Step 205: Compare the total weight values and select a removal strategy based on the difference between the first two categories with the largest total weight values.

[0089] In this step, since the above step 204 has calculated the total weight value of all environmental characteristics in various categories, it is necessary to set the threshold reasonably when comparing various weights to ensure that the threshold can effectively distinguish the dominance of different categories and handle the situation where the total weight values are similar to avoid misjudgment.

[0090] Specifically, determine a reasonable threshold to distinguish the dominance of different categories. This threshold should be set based on the importance of environmental features, the goal of repairing multiple categories, and the possible error range. The threshold should be sensitive enough to detect significant differences in total weight values, but also lenient enough to avoid false positives when the total weight values are similar.

[0091] Before comparing the total weights of each category calculated in step 204 and determining which category has the highest total weight, the categories are arranged in descending order of total weight to facilitate identification of the dominant category. When the weights of multiple categories are similar, if only a removal strategy corresponding to one category is used to repair the area to be repaired, the repair effect may not be as good as using a combination of removal strategies corresponding to multiple categories. Therefore, a method is used to select the corresponding removal strategy based on the difference between the categories: the difference between the top two categories with the largest total weights is calculated. This difference helps determine whether there is a significant difference between the two categories. If the difference is large, it indicates that the dominant category is very clear, and the removal strategy corresponding to the dominant category can be directly selected. If the difference is small, it indicates that the importance of the top two categories is similar, and further analysis or threshold adjustment is required to determine whether a combination of removal strategies for the two categories is needed, or a more general strategy is selected to balance the two. It is important to note that when the difference between the top two categories with the largest total weights is small, this may mean that both categories are crucial to the selection of the removal strategy. In this case, the importance of other categories should not be ignored. Therefore, continue to judge whether other classifications also have a significant impact on the restoration effect. The specific operation is as follows: calculate the difference between the third largest total weight value and the first two total weight values, and judge whether this difference is greater than or equal to the preset threshold. If this difference is less than the threshold, it means that the classification with the third largest total weight value is equally important, and its removal strategy needs to be considered in the restoration plan. This process can be continued, comparing the difference between the classification with a smaller total weight value and the previous classification to determine whether there are other classifications that need to be considered. In this way, it can be ensured that all important environmental characteristic classifications are taken into account when repairing the area to be repaired, thereby achieving a more natural and coordinated restoration effect.

[0092] When the total weights are similar, consider the following strategies: Combine the removal strategies from two categories to create a new strategy. Choose a more flexible or general removal strategy to accommodate a variety of environmental characteristics. If possible, consult experts or conduct user testing to determine the optimal strategy.

[0093] Through these steps, it is ensured that the selection of removal strategy is both scientific and reasonable, and can maintain the naturalness and consistency of the restored area with the surrounding environment to the greatest extent.

[0094] Since step 205 has described in detail how to select the corresponding removal strategy according to the difference in total weight values, in order to deeply understand its specific implementation process, dynamic and static classification will be used as examples to explain in detail how to select the corresponding removal strategy according to the difference in total weight values.

[0095] Assuming that the total weights of the dynamic and static categories have been calculated and arranged in descending order, the following are specific steps for implementing step 205:

[0096] Step 1: Calculate the total weight values of all environmental characteristics in the dynamic classification and static classification respectively.

[0097] In this step, the total weights of the dynamic and static classifications are compared. If the total weight of the dynamic classification is higher than that of the static classification, then the dynamic characteristics are more important in the area to be repaired; otherwise, the static characteristics are more important.

[0098] Step 2: Calculate the difference between the total weight values corresponding to the dynamic classification and the static classification, and determine whether the difference is greater than or equal to the threshold.

[0099] In this step, the difference between the category with the highest total weight (assuming it is a dynamic category) and the category with the second highest total weight (assuming it is a static category) is calculated. This difference will help determine whether there is a significant difference between the two categories.

[0100] Step 3: Select the removal strategy based on the category corresponding to the maximum total weight value.

[0101] If the difference in step 2 is greater than or equal to the threshold, that is, the difference is large, this indicates that the dynamic classification clearly dominates the area to be repaired. Therefore, you can choose a removal strategy that focuses on dynamic characteristics, for example, using a motion blur effect to simulate the ripples of water flow or the changes in light reflection.

[0102] Step 4: Within the area to be repaired, divide the area to be repaired into multiple sub-areas, determine the category to which each sub-area belongs based on all environmental characteristics within each sub-area, and select the removal strategy corresponding to each sub-area based on the classification.

[0103] If the difference in step 2 is less than the threshold, that is, the difference is small, this indicates that the importance of dynamic and static characteristics in the area to be repaired is almost equal, and there is no obvious advantage classification. In this case, in order to process the area to be repaired more accurately, the following measures can be taken:

[0104] First, since both dynamic and static characteristics have a significant impact on the area, the area to be repaired can be divided into several small areas. This segmentation allows for a more detailed analysis of the environmental characteristics within each small area.

[0105] Next, for each small area, the corresponding removal strategy will be selected according to its specific environmental characteristics classification. For specific operations, please refer to: In small areas where dynamic characteristics are more significant, priority will be given to removal strategies targeting dynamic characteristics, such as simulating changes in light reflection or the effect of water flow. In small areas where static characteristics are more significant, emphasis will be placed on removal strategies targeting static characteristics, for example, adjusting color and texture to match surrounding stable elements. For small areas where both dynamic and static characteristics are relatively balanced, removal strategies of the two characteristics will be combined, or a general strategy that can handle both dynamic and static characteristics will be adopted. In this way, it can be ensured that each small area is properly treated, so that the entire area to be repaired remains natural and coordinated with the surrounding environment after repair.

[0106] Through the above detailed application process, it can be ensured that when repairing the area to be repaired, the removal strategy selected can fully consider the importance of dynamic and static characteristics, as well as the relationship between them, so as to achieve a natural and coordinated repair effect.

[0107] Furthermore, for step 3 in step 205 above, if the category corresponding to the maximum total weight value is a dynamic category, in order to better describe the specific process of how the removal strategy corresponding to the category repairs the area to be repaired, such as Figure 3 As shown, an application process for dynamic classification corresponding removal strategy is provided:

[0108] Step 301: Obtain the repair priority of each boundary pixel in the area to be repaired.

[0109] In this step, the boundary of the area to be repaired is clarified. Boundary pixels usually refer to the pixel points where the area to be repaired touches the surrounding known area. For each boundary pixel, its repair priority is calculated. The calculation of the repair priority can be based on the following factors: the position of the pixel, pixels at certain positions may have a greater impact on the overall visual effect, such as corner or edge pixels; environmental characteristics, considering the environmental characteristics of the area where the pixel is located, such as dynamic characteristics may require a higher priority; visual saliency, the visual saliency of the pixel may affect its priority; contextual information, the contextual information around the pixel, such as the consistency of texture, color, etc. Different algorithms or models can be used to calculate the repair priority, such as saliency detection based on image content, graph-based algorithms, or machine learning models.

[0110] Here is a priority calculation method worthy of reference: the boundary pixels of the area to be repaired are formed into a linked list, and the data item and confidence level of each boundary pixel are obtained. The data item is used to represent the similarity value between the boundary pixel of the area to be repaired and the known pixels in the adjacent known area, and the confidence level is used to represent the degree of trust in the boundary pixel value of the area to be repaired; according to the size relationship between the confidence level and the preset value, a preset formula combining the confidence level and the data item is selected, and the preset formula is used to calculate the repair priority of each boundary pixel in the linked list.

[0111] The following is a preset formula for calculating the restoration priority of the boundary pixels of the area to be restored:

[0112] Let P(i) be the inpainting priority for boundary pixel i, D(i) be the data item for boundary pixel i, and C(i) be the confidence level for boundary pixel i. The preset formula can be expressed as: P(i) = f(D(i), C(i)). Function f can be expressed as: f(D(i), C(i)) = α·C(i) + (1-α)·D(i). Here, α is a weight coefficient between 0 and 1 that balances the influence of the confidence level and the data item on the inpainting priority. The specific value can be adjusted based on actual conditions. Data item D(i): represents the similarity between boundary pixel i and its neighboring known pixels. Similarity can be calculated by comparing pixel features such as color, brightness, and texture. For example, the data item can be calculated using the following formula: D(i) = 1 / (1+d(i, j)). d(i, j) is the distance metric (e.g., Euclidean distance) between boundary pixel i and its nearest known pixel j. Confidence level C(i): represents the confidence level in the value of boundary pixel i in the region to be inpainted. Confidence can be based on pixel visibility, stability, or a reliability score derived from certain image processing algorithms. For example, the confidence level can be calculated using the following formula: C(i) = 1 / (1 + σ(i)), where σ(i) is the uncertainty or noise level of boundary pixel i. Substituting these values into a pre-set formula yields the restoration priority P(i) for each boundary pixel. In practice, the parameters and calculation methods in the formula can be adjusted to achieve optimal restoration results.

[0113] Step 302: After identifying the multi-category target pixel with the maximum restoration priority, analyze the region type where the multi-category target pixel is located.

[0114] After obtaining the restoration priorities of all boundary pixels in step 301, the multi-category target pixel with the highest restoration priority is identified. This pixel will be the starting point for the next restoration process. In this step, the region where the multi-category target pixel resides is classified into flat region, edge region, and corner region.

[0115] Specifically, the purpose of doing this is that if the multi-category target pixels are located in a flat area, it usually means that the texture and color changes in the area are small, and a smaller image block can be selected for repair. If the multi-category target pixels are located in the edge area, the continuity of the edge needs to be considered during repair. It may be necessary to select a larger image block that can cover the edge and use an edge-guided repair algorithm. If the multi-category target pixels are located in the edge area, the continuity of the edge needs to be considered during repair. It may be necessary to select a larger image block that can cover the edge and use an edge-guided repair algorithm.

[0116] Through such analysis, we can ensure that the restoration work is more accurate while maintaining the naturalness and consistency of the image.

[0117] Step 303: According to the preset correspondence between the region type and the size of the image block to be restored, the size of the image block to be restored is determined based on the region type where the multi-category target pixel points are located.

[0118] Since the region categories where the multi-category target pixels are located have been determined in step 302, in this step, the size of the image block to be restored can be determined based on the correspondence between the region categories and the size of the image block to be restored.

[0119] Specifically, first, a correspondence between region types and image block sizes to be restored needs to be preset based on the requirements of the restoration algorithm and image characteristics. This correspondence is typically determined based on experience and experimental results. In step 302, the region types (flat regions, edge regions, or corner regions) where the target pixels of multiple categories are located have already been determined.

[0120] Next, the preset correspondence table is queried based on this information. Based on the query results, the size of the image block to be repaired is determined to be suitable for the type of area where the multi-category target pixels are located. The following are examples of possible correspondences: Flat area: Since flat areas do not change much, a smaller image block size can be selected, such as 3x3 or 5x5 pixels. Edge area: The edge area requires more contextual information to maintain the continuity of the edge, so a medium-sized image block can be selected, such as 7x7 or 9x9 pixels. Corner area: The corner area contains the most complex information and may require a larger image block, such as 11x11 or larger, to ensure that the features of the corner points are properly preserved.

[0121] After determining the image block size, an image block of this size is extracted from the area to be repaired, centered on the multi-category target pixel. This image block will be used in the subsequent search and matching process.

[0122] In this way, step 303 ensures that each area to be repaired can select the most appropriate image block size according to its specific characteristics, so that the structure and details of the image can be more effectively maintained in the subsequent repair process.

[0123] Step 304: Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block with the highest similarity to the image block to be repaired in a known area around the area to be repaired.

[0124] In this step, since the size of the image block to be repaired has been constructed in step 303, it is necessary to search for a matching block that can match the image block to be repaired in the surrounding known area. When determining the search area, that is, the known area around the area to be repaired, this area should be large enough to ensure that a matching block with a high similarity to the image block to be repaired can be found. A local search strategy is adopted, which means that the search will be concentrated on the area around the multi-category target pixel, rather than the entire image. This strategy can reduce the amount of calculation and improve search efficiency. For the image block to be repaired centered on the multi-category target pixel, the similarity between it and the candidate matching block in the known area is calculated. The similarity calculation can be based on a variety of factors, such as color, texture, structure, etc. For the image block to be repaired centered on the multi-category target pixel, the similarity between it and the candidate matching block in the known area is calculated. The similarity calculation can be based on a variety of factors, such as color, texture, structure, etc.

[0125] The specific implementation process can be referred to as follows:

[0126] Step 1: Get the length and width of the remote sensing image to be restored, take the center of the image block to be restored as the base point, set the search radius, and take the smaller value of the length and width of the remote sensing image to be restored, 1 / n, where n is a preset positive integer.

[0127] In this step, the size information of the remote sensing image to be repaired, that is, the length and width of the image, is obtained. Then, with the center of the image block to be repaired as the base point, an initial search radius is set according to the size of the image. This search radius is usually taken as 1 / n of the smaller value of the image length and width, where n is a preset positive integer. The value of n can be adjusted according to actual conditions to balance the search efficiency and matching accuracy. Setting the search radius is the key to ensuring that the search area is large enough to find high-quality matching blocks. By taking 1 / n of the smaller value of the image length and width as the initial radius, it can be ensured that the search area is neither too large to cause a sharp increase in the amount of calculation, nor too small to find enough matching blocks.

[0128] Step 2: Based on the set search radius, check whether there is a matching block in the search area whose similarity with the image block to be repaired meets the matching requirements;

[0129] In this step, all candidate matching blocks are traversed within a set search radius, and the similarity between them and the image block to be repaired is calculated. The similarity calculation can be based on various factors such as color, texture, and structure. In this embodiment, based on the traditional Criminisi algorithm, a spatial feature, namely distance similarity, is introduced to measure the degree of similarity between image blocks. The best matching block is selected for filling according to the matching principle of the following formula:

[0130] ψ q′ =argminSM(ψ p ,ψ q ),ψ q ∈φ

[0131] SM(ψ p ,ψ q )=μ*SSD(ψ p ,ψ q )+ν*Dist(ψ p ,ψ q )

[0132] SSD(ψ p ,ψ q )=∑[(I r -I′ r ) 2 +(I g -I′ g ) 2 +(I b -I′ b ) 2 ]

[0133]

[0134] Where μ and ν are constants, μ + ν = 1. φ represents the intact area. p and q represent the pixel positions of the area to be repaired and the candidate matching block, ψ p ,ψ q Represent the image block and candidate matching block of the area to be repaired, ψ q′ is the best candidate matching block. I r ,I g ,I b is the pixel point ″′ in the multi-category target block

[0135] RGB value, I r ,I g ,I b Is the RGB value of the pixel in the matching sample block. i and B i is a vector of multi-category target image patches and patches to be matched.

[0136] If a matching block is found whose similarity meets the matching requirements, the process proceeds to the next step, step 305. The local search strategy plays a key role in this step. By searching for matching blocks within a smaller search area, the computational effort can be significantly reduced, improving search efficiency. Furthermore, similarity calculation is essential for ensuring the discovery of high-quality matching blocks.

[0137] Step 3: Increase the search radius until at least one matching block with a similarity to the image block to be repaired that meets the matching requirements is found in the search area, and determine the matching block with the highest similarity as the multi-category target matching block.

[0138] In this step, if no matching blocks are found within the initial search radius, the search radius is gradually increased and the search process in step 2 is repeated until at least one matching block is found within the search area. Then, the one with the highest similarity among these matching blocks is selected as the multi-category target matching block. By gradually increasing the search radius, matching blocks are searched over a wider range, increasing the chances of finding high-quality matching blocks. Furthermore, selecting the matching block with the highest similarity as the multi-category target matching block ensures that the restored image remains consistent with the surrounding area in terms of color and texture.

[0139] In summary, step 304, by setting a reasonable search area, employing a local search strategy, and gradually increasing the search radius, efficiently searches for multi-category target matching blocks with the highest similarity to the image block to be restored within the surrounding known area. This provides an important reference and basis for subsequent image restoration work.

[0140] Step 305: Fill the multi-category target matching block into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after repair.

[0141] In this step, the main task is to fill the position of the image block to be repaired with the previously found multi-category target matching blocks (i.e., the known regions with the highest similarity to the image block to be repaired). The purpose of this is to ensure that the repaired image block is consistent with the surrounding known regions in terms of color, texture, structure, etc., thereby achieving visual coherence and naturalness.

[0142] First, the position of the image block to be restored in the image needs to be accurately located, which is usually achieved by determining the coordinates of the image block to be restored in step 303 .

[0143] Next, the previously found multi-category target matching block is extracted from the known region. This matching block is found in step 304 using the local search strategy and has the highest similarity with the image block to be repaired.

[0144] Then, the extracted multi-category target matching block is filled into the position of the image block to be repaired. This step usually involves pixel-level operations, that is, copying each pixel value in the multi-category target matching block to the corresponding position of the image block to be repaired.

[0145] In some cases, to further improve the restoration effect, it may be necessary to perform transition processing on the filled edges. This can be achieved through smoothing filtering, gradient processing, or other image processing techniques to reduce the abruptness between the filled area and the surrounding known areas.

[0146] Finally, the restored image needs to be verified to ensure that the restoration is as expected. This can be achieved by visual inspection, calculating the similarity between the restored area and the surrounding area, or using other quality assessment metrics.

[0147] Through this step, the position of the image block to be repaired can be filled with known area blocks similar to the multi-category target matching blocks, so that the repaired image is more visually coherent and natural. Figure 7 As shown, the application Figure 3 The effect comparison of the dynamic classification corresponding removal strategy proposed in the paper. This strategy performs particularly well in dealing with dynamic environments such as water areas and can effectively remove targets such as ships. It is worth noting that the "dynamic classification corresponding removal strategy" in this embodiment is not limited to removing Figure 7 The example is not about ships, but about target removal for many categories.

[0148] Furthermore, for step 3 in step 204 above, if the category corresponding to the maximum total weight value is a static category, in order to better describe the specific process of how the removal strategy corresponding to the category repairs the area to be repaired, such as Figure 4 As shown, an application process for the static classification corresponding removal strategy is provided:

[0149] Step 401: Obtain the priority of each environmental characteristic on the visual restoration effect in the static classification, and identify the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect.

[0150] In this step, first, all relevant environmental characteristics information in static classification needs to be collected. These environmental characteristics may include color, texture, texture, etc.

[0151] Next, the specific impact of each environmental characteristic on the visual restoration effect needs to be evaluated. This can be achieved through experimental analysis, expert evaluation, or historical data comparison. During the evaluation process, factors such as the importance of the environmental characteristic in the restoration process, its contribution to the restoration effect, and its interaction with other environmental characteristics need to be considered. Based on the evaluation results, a priority can be assigned to each environmental characteristic. The priority is usually a numerical value or a level that indicates the importance of the environmental characteristic in the visual restoration process. The determination of priority can be based on a variety of factors, such as the degree of impact of the environmental characteristic, the necessity of the restoration process, and the degree of improvement in the restoration effect.

[0152] Finally, the environmental feature with the highest priority is identified as the first environmental feature that has the greatest impact on the visual restoration effect. This feature is usually the one that needs to be considered and processed first during the restoration process.

[0153] Through this step, the priority of each environmental characteristic in the static classification on the visual restoration effect can be clarified, and the first environmental characteristic with the greatest impact on the visual restoration effect can be determined, providing an important reference basis for subsequent processing steps.

[0154] Step 402: extracting first environmental characteristics in the remote sensing area to be repaired and the surrounding known areas.

[0155] In this step, the remote sensing area to be repaired and the surrounding known areas must first be accurately located. This is typically achieved through remote sensing image preprocessing steps (such as image registration and segmentation). The area to be repaired is the portion of the image that needs to be repaired or replaced, while the surrounding known areas are intact or less damaged areas that provide a reference for repair.

[0156] Next, we need to select a method suitable for extracting the first environmental characteristic. This method depends on the specific type of the first environmental characteristic, such as color, texture, or structure. For example, if the first environmental characteristic is color, methods such as color space conversion and color histogram statistics can be used to extract color information. If the first environmental characteristic is texture, techniques such as texture analysis and texture synthesis can be used to extract texture features.

[0157] After selecting a suitable extraction method, the first environmental characteristics are extracted from the area to be repaired and the surrounding known areas respectively.

[0158] Finally, it is necessary to verify whether the extracted first environmental features are accurate, complete, and meet the requirements of subsequent processing steps. This can be achieved by comparing the extracted results with the original image, using feature matching algorithms, etc.

[0159] Through this step, the first environmental characteristics can be extracted from the remote sensing area to be repaired and the surrounding known areas, providing important input information for the subsequent step 403 using a convolutional neural network to perform image restoration.

[0160] Step 403: Input the extracted first environmental characteristics into a convolutional neural network, which has a large receptive field characteristic.

[0161] In this step, the large receptive field feature is used to capture the information of the input image and predict and generate the missing image content.

[0162] First, the extracted first environmental characteristics need to be organized into a format suitable for convolutional neural network input. This may include steps such as data resizing, normalization, and data augmentation. The input data typically includes first environmental characteristics of the area to be repaired and its surrounding known areas, which will serve as the input to the convolutional neural network.

[0163] Next, you need to select a convolutional neural network model with a large receptive field. A large receptive field means the network can capture a wider range of input image information, which is crucial for predicting and generating missing image content. Common convolutional neural network models include VGG, ResNet, and Inception. The specific model you choose depends on the complexity of the task, the availability of computing resources, and the performance requirements of the model.

[0164] After selecting an appropriate convolutional neural network model, you need to configure network parameters such as the learning rate, batch size, and number of iterations. These parameters will affect the network's training performance and convergence speed. The prepared input data is fed into the convolutional neural network, and the training process begins. During training, the network learns how to predict and generate the missing image content from the input's initial environmental characteristics. The training process may involve multiple iterations, with the network adjusting its weights and bias parameters in each iteration based on the error calculated by the loss function.

[0165] Step 404: Verify whether the image content is visually consistent with the surrounding known area and meets the restoration standard.

[0166] After the training is completed in step 403, in this step, the performance of the network needs to be verified. This can be achieved by applying the network to a validation dataset and evaluating its restoration effect. Verification indicators may include visual inspection, calculation of image similarity indicators (such as PSNR, SSIM, etc.), comparison of the difference between the repaired area and the surrounding known areas, etc. Once the network training is completed and its performance is verified, it can be used to generate the missing image content. This usually involves inputting the first environmental characteristics of the area to be repaired into the network and outputting the predicted image content.

[0167] Step 405: Gradually add other environmental characteristics according to priority as input and repeat the neural network processing until the repair effect meets the preset requirements.

[0168] If the verification result of step 404 is negative, further measures are required to improve the restoration effect, including: gradually adding other environmental characteristics sorted by priority as input, these characteristics were determined in step 401, but initially only the first environmental characteristic was used. Repeat the neural network processing process of step 403, input the new combination of environmental characteristics into the network, and generate a new restoration image. Repeat the (implicit) verification process of step 404 to evaluate whether the newly generated restoration image meets the preset requirements. This iterative process will continue until the restoration effect meets the preset quality requirements or cannot be improved by adding environmental characteristics.

[0169] Through this iterative process, the restoration effect can be gradually improved until it meets the preset quality requirements. This method fully utilizes the priority information of environmental characteristics and can minimize the consumption of computing resources while ensuring the restoration quality.

[0170] like Figure 8 As shown, the application Figure 4 The effect comparison of the static classification corresponding removal strategy proposed in the paper. This strategy performs particularly well in dealing with environments with static characteristics such as airports, and can effectively remove targets such as Figure 8 The example is not about the aircraft, but is applicable to target removal of various categories.

[0171] Furthermore, with respect to the stitching process in step 104, in order to better describe how to stitch the segmented area to be repaired, a process for selecting an image stitching algorithm based on the removal strategy is provided as follows: Step 1: When the area to be repaired is segmented into multiple intermediate areas, identify the same multi-category target spanning multiple intermediate areas. Step 2: At the region boundary of multiple intermediate areas, determine the removal strategy corresponding to the multiple intermediate areas constituting the region boundary. Step 3: Based on the corresponding removal strategy, select the corresponding image stitching algorithm to stitch the multiple intermediate areas, generate a smooth intermediate image, and update the original image data. Among them, the stitching algorithm is selected based on the removal strategy, including but not limited to the following selection methods: If the removal strategy is to simply remove the boundary or edge part, then it may be necessary to use a stitching technology that can handle obvious boundaries, such as a graph cut-based method. If the removal strategy is to achieve a smooth transition through gradient or other means, then it may be appropriate to adopt a stitching method based on weighted average, multi-scale decomposition and synthesis, or content-awareness.

[0172] In addition to selecting an image stitching algorithm based on the removal strategy, other approaches include: For scenes requiring high texture and color consistency (e.g., natural landscapes), stitching techniques based on multiresolution analysis, such as Laplacian pyramid blending, may be more suitable. When maintaining the consistency of fine structures is crucial, such as in medical imaging, stitching methods based on feature point matching can be considered to ensure that fine structures are not lost or distorted during stitching. Image stitching algorithms can also be selected based on the characteristics of the junctions, including: if adjacent regions are very similar, simple linear blending or color correction-based stitching methods can be considered. For regions with significant differences, more complex stitching methods may be required, such as deep learning-based image fusion models, which can learn how to optimally connect regions with different characteristics. Image stitching algorithms can be selected based on algorithm complexity and performance requirements, including: for applications requiring real-time processing, computationally efficient algorithms, such as fast Fourier transform (FFT)-based stitching methods, may be desirable. If the accuracy requirement is extremely high, you can sacrifice some computational efficiency and choose a more accurate but more computationally intensive algorithm, such as a splicing method based on global optimization.

[0173] This series of steps forms a closed-loop feedback mechanism to ensure that the restoration of remote sensing images meets the expected standards. By continuously evaluating and adjusting the removal strategy, the restoration results can be gradually optimized until specific quality or performance requirements are met.

[0174] Further, based on the same inventive concept, as a Figure 1 In order to realize the embodiment of the method, the embodiment of the present invention also provides a device for automatically removing multiple categories of targets from remote sensing images. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can realize all the contents of the aforementioned method embodiment. Figure 5 As shown, the device includes: an acquisition unit 51, an analysis unit 52, a removal unit 53 and a splicing unit 54.

[0175] An acquisition unit 51 is configured to acquire a remote sensing image to be restored, where the remote sensing image to be restored includes an area to be restored that covers multiple categories of objects to be removed;

[0176] A determining unit 52 is configured to determine environmental characteristics of a known area surrounding the area to be restored in the remote sensing image to be restored in the acquiring unit 51;

[0177] a removal unit 53 for performing multi-category object removal processing on the area to be repaired based on a removal strategy corresponding to the environmental characteristics in the determination unit 52, wherein the removal strategy at least includes repairing the area to be repaired using images within a known area surrounding the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network based on the input environmental characteristics;

[0178] The splicing unit 54 is configured to splice the area to be repaired after the removal process in the removal unit 53 if the area to be repaired is segmented.

[0179] Furthermore, the embodiment of the present invention also provides a remote sensing image multi-category target automatic removal device for the above Figure 1-4 This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not describe the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can implement all the contents of the aforementioned method embodiment. Figure 6 As shown, the device includes: a receiving unit 51, a matching unit 52, a selecting unit 53 and a splicing unit 54.

[0180] In an optional embodiment, the analysis unit 52 includes:

[0181] The traversal module 521 is used to traverse the area to be repaired and obtain the corresponding four-dimensional range;

[0182] An acquisition module 522 is configured to buffer N pixels outward according to the range in the traversal module 521 to obtain an image block to be processed;

[0183] A first determining module 523 is configured to determine environmental characteristics of a known area in the image block to be processed in the acquiring module 522;

[0184] A calculation module 524 is configured to classify the multiple environmental characteristics of the area to be repaired according to a preset characteristic classification standard if the first determination module 523 determines that the area to be repaired has multiple environmental characteristics, and to calculate the total weight value of all environmental characteristics in each classification;

[0185] The comparison module 525 is used to compare the total weight values in the calculation module 524 and select a removal strategy according to the difference between the first two categories with the largest total weight values.

[0186] In an optional embodiment, the first two categories with the largest total weight values in the comparison module 525 are dynamic and static categories, and the comparison module 525 includes:

[0187] A calculation submodule 5251 is used to calculate the total weight values of all environmental characteristics in the dynamic classification and the static classification respectively;

[0188] The judgment submodule 5252 is used to calculate the difference between the total weight values corresponding to the dynamic classification and the static classification in the calculation submodule 5251, and determine whether the difference is greater than or equal to a threshold value. The threshold value is used to determine the dominance of the dynamic classification and the static classification in selecting the removal strategy for the repair area;

[0189] The first selection submodule 5253 is configured to select a removal strategy based on the category corresponding to the maximum total weight value if the judgment result of the judgment submodule 5253 is yes;

[0190] The second selection submodule 5254 is used to divide the area to be repaired into multiple sub-areas within the area to be repaired if the judgment result of the judgment submodule 5253 is no, determine the category to which each sub-area belongs based on all environmental characteristics within each sub-area, and select the removal strategy corresponding to each sub-area based on the classification.

[0191] Among them, when the classification corresponding to the maximum total weight value in the first selection submodule 5253 is a dynamic classification, a multi-category target removal process is performed on the area to be repaired according to the removal strategy corresponding to the environmental characteristics, including: obtaining the repair priority of each boundary pixel of the area to be repaired; after identifying the multi-category target pixel with the maximum repair priority, analyzing the area type where the multi-category target pixel is located, the type includes a flat area, an edge area and a corner area; according to the correspondence between the preset area type and the size of the image block to be repaired, the size of the image block to be repaired is determined based on the area type where the multi-category target pixel is located; based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block with the highest similarity to the image block to be repaired in the known area around the area to be repaired; and filling the multi-category target matching block into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after repair.

[0192] Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for the multi-category target matching block with the highest similarity to the image block to be repaired in the known area around the area to be repaired, including: obtaining the length and width of the remote sensing image to be repaired, taking the center of the image block to be repaired as the base point, setting a search radius, and taking the radius as 1 / n of the smaller value of the length and width of the remote sensing image to be repaired, where n is a preset positive integer; based on the set search radius, checking whether there is a matching block in the search area whose similarity with the image block to be repaired meets the matching requirements; if not, increasing the search radius until at least one matching block whose similarity with the image block to be repaired meets the matching requirements is found in the search area, and determining the matching block with the highest similarity as the multi-category target matching block.

[0193] Among them, when the classification corresponding to the maximum total weight value in the first selection submodule 5253 is a static classification, multi-category target removal processing is performed on the area to be repaired according to the removal strategy corresponding to the environmental characteristics, including: obtaining the priority of each environmental characteristic in the static classification on the visual restoration effect, and confirming the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect; extracting the first environmental characteristic in the remote sensing area to be repaired and the surrounding known area; inputting the extracted first environmental characteristic into a convolutional neural network, which has a large receptive field characteristic and is used to capture the information of the input image and predict and generate missing image content; verifying whether the image content is visually consistent with the surrounding known area and meets the restoration standard; if not, gradually adding other environmental characteristics according to priority as input, and repeating the neural network processing until the restoration effect meets the preset requirements.

[0194] In an optional embodiment, the splicing unit 54 includes:

[0195] Identification module 541, for identifying the same multi-category object spanning multiple intermediate regions when the region to be repaired is divided into multiple intermediate regions;

[0196] The second determining module 542 is configured to determine, at the region junction of the plurality of middle regions in the identifying module 541, a removal strategy corresponding to the plurality of middle regions constituting the region junction;

[0197] The stitching module 543 is used to select a corresponding image stitching algorithm to stitch the multiple intermediate areas according to the corresponding removal strategy in the second determining module 542, generate a smooth intermediate image, and update the original image data.

[0198] Furthermore, an embodiment of the present invention also provides an electronic device, which includes at least one processor, and at least one memory and bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute a method for automatically removing multi-category targets from remote sensing images.

[0199] Furthermore, an embodiment of the present invention also provides a readable storage medium, which is used to store a computer program, wherein when the computer program is running, it controls the device where the storage medium is located to execute the method for automatically removing multiple categories of targets in remote sensing images.

[0200] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for automatically removing multi-category targets from remote sensing images, characterized in that: The method comprises: Acquiring a remote sensing image to be restored, wherein the remote sensing image to be restored includes an area to be restored that covers multiple categories of objects to be removed; Determining environmental characteristics of a known area surrounding the area to be restored in the remote sensing image to be restored, wherein the environmental characteristics are various physical properties, spectral characteristics, and texture characteristics within the known area surrounding the area to be restored; Performing multi-category target removal processing on the area to be repaired according to a removal strategy corresponding to the environmental characteristics, the removal strategy at least comprising repairing the area to be repaired using images within a known area surrounding the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network based on the input environmental characteristics; If the area to be repaired is segmented, the area to be repaired after the removal process is performed a splicing process; Determining environmental characteristics of known areas surrounding the area to be restored in the remote sensing image to be restored, including: Traverse the area to be repaired and obtain the corresponding four-dimensional range, which refers to the rectangular bounding box formed by the farthest points in the east, south, west, and north directions of all the areas to be repaired; Buffer N pixels outward according to the four-to-range to obtain an image block to be processed; Determining environmental characteristics of a known area in the image block to be processed; If the known area has multiple environmental characteristics, the multiple environmental characteristics are classified according to a preset characteristic classification standard, and the total weight value of all the environmental characteristics in each classification is calculated respectively; The total weight values are compared, and the removal strategy is selected according to the difference between the first two categories with the largest total weight value.

2. The method according to claim 1, characterized in that If the first two categories with the largest total weight are dynamic and static, dynamic categories refer to characteristics that are easily changed with time or external conditions, and static categories refer to relatively stable characteristics; The comparing of the total weight values and selecting the removal strategy according to the difference between the first two categories with the largest total weight values includes: Calculate the total weight values of all environmental characteristics in the dynamic classification and the static classification respectively; Calculating a difference between the total weight values corresponding to the dynamic classification and the static classification, and determining whether the difference is greater than or equal to a threshold, the threshold being used to determine the dominance of the dynamic classification and the static classification in selecting a removal strategy for the area to be repaired; If so, selecting the removal strategy according to the category corresponding to the maximum total weight value; If not, then within the area to be repaired, the area to be repaired is divided into multiple sub-areas, and the category to which each sub-area belongs is determined based on all the environmental characteristics within each sub-area, and the removal strategy corresponding to each sub-area is selected according to the classification.

3. The method according to claim 2, characterized in that When the classification corresponding to the maximum total weight value is a dynamic classification; performing multi-category target removal processing on the area to be repaired according to the removal strategy corresponding to the environmental characteristics includes: Obtaining a repair priority for each boundary pixel of the area to be repaired; After identifying the multi-category target pixels with the maximum restoration priority, analyzing the region types where the multi-category target pixels are located, the region types including flat regions, edge regions, and corner regions; According to the correspondence between the preset region type and the size of the image block to be restored, the size of the image block to be restored is determined based on the region type where the multi-category target pixel points are located; Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block having the highest similarity to the image block to be repaired in a known area around the area to be repaired; The multi-category target matching block is filled into the position of the image block to be repaired in the area to be repaired, so that the position of the image block to be repaired is consistent with the surrounding known area after repair.

4. The method according to claim 3, characterized in that Based on the image block to be repaired centered on the multi-category target pixel, a local search strategy is adopted to search for a multi-category target matching block having the highest similarity to the image block to be repaired in a known area around the area to be repaired, including: Obtain the length and width of the remote sensing image to be restored, and set a search radius based on the center of the image block to be restored, where the radius is 1 / n of the smaller value of the length and width of the remote sensing image to be restored, where n is a preset positive integer; Based on the set search radius, checking whether there is a matching block in the search area whose similarity with the image block to be repaired meets the matching requirement; If not, the search radius is increased until at least one matching block whose similarity with the image block to be repaired meets the matching requirements is found in the search area, and the matching block with the highest similarity is determined as the multi-category target matching block.

5. The method according to claim 2, characterized in that When the classification corresponding to the maximum total weight value is a static classification; performing multi-category target removal processing on the area to be repaired according to the removal strategy corresponding to the environmental characteristics includes: Obtaining the priority of each environmental characteristic in the static classification on the visual restoration effect, and identifying the environmental characteristic with the highest priority as the first environmental characteristic with the greatest impact on the visual restoration effect; Extracting first environmental characteristics from the remote sensing area to be repaired and the surrounding known areas; Inputting the extracted first environmental characteristics into a convolutional neural network, wherein the convolutional neural network has a large receptive field characteristic and is used to capture information prediction of the input image and generate missing image content; Verify that the image content is visually consistent with the surrounding known area and meets the restoration standards; If not, other environmental characteristics according to priority are gradually added as inputs, and the neural network processing is repeated until the repair effect meets the preset requirements.

6. The method according to claim 1, characterized in that If the area to be repaired is segmented, the area to be repaired after the removal process is performed a splicing process, including: When the area to be repaired is divided into multiple intermediate areas, identifying that the same multi-category target spans multiple intermediate areas; At the region junction of the plurality of the middle regions, determining removal strategies corresponding to the plurality of the middle regions constituting the region junction; According to the corresponding removal strategy, a corresponding image stitching algorithm is selected to stitch the plurality of intermediate areas to generate a smooth intermediate image, and the original image data is updated.

7. A device for automatically removing multi-category targets from remote sensing images, characterized in that: The device comprises: An acquisition unit is used to acquire a remote sensing image to be restored, wherein the remote sensing image to be restored includes an area to be restored that covers multiple categories of objects to be removed; a determining unit configured to determine environmental characteristics of a known area surrounding the area to be restored in the remote sensing image to be restored in the acquiring unit, comprising: traversing the area to be restored and acquiring a corresponding four-dimensional range, wherein the four-dimensional range refers to a rectangular bounding box formed by the farthest points in the four directions of east, south, west, and north of all the areas to be restored; Buffer N pixels outward according to the four-to-range to obtain an image block to be processed; Determining environmental characteristics of a known area in the image block to be processed; If the known area has multiple environmental characteristics, the multiple environmental characteristics are classified according to a preset characteristic classification standard, and the total weight value of all the environmental characteristics in each classification is calculated respectively; Comparing the total weight values and selecting a removal strategy based on the difference between the first two categories with the largest total weight values, wherein the environmental characteristics are various physical properties, spectral characteristics, and texture characteristics in a known area around the area to be repaired; a removal unit, configured to perform multi-category target removal processing on the area to be repaired according to a removal strategy corresponding to the environmental characteristics in the determination unit, wherein the removal strategy at least includes repairing the area to be repaired using images within a known area surrounding the area to be repaired and repairing the output image of the area to be repaired using a convolutional neural network based on the input environmental characteristics; The splicing unit is used to splice the area to be repaired after the removal process in the removal unit if the area to be repaired is divided.

8. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the method for automatic removal of multi-category targets in remote sensing images as described in any one of claims 1-6.

9. A readable storage medium, characterized in that The storage medium is used to store a computer program, wherein when the computer program is running, it controls the device where the storage medium is located to execute the method for automatically removing multi-category targets from remote sensing images according to any one of claims 1 to 6.

Images