Remote sensing image automatic cloud removal mosaic method and device, equipment and storage medium

Abstract

The invention relates to a remote sensing image automatic cloud removal mosaic method and device, equipment and a storage medium. The method comprises the following steps: extracting cloud and shadow regions in a remote sensing orthoimage through a pre-trained extraction model, and generating a vector file; based on the vector files of the cloud region and the target range, generating a cloud removal mosaic strategy in the target range, and generating a vector combination table; based on a cloud removal strategy in the vector combination table, selecting an image patch and filling clouds and shadows in the to-be-processed image to generate a cloudless remote sensing image; and on the basis of an embedding strategy in the vector combination table, embedding the cloud-removed remote sensing images in sequence, and eliminating boundary lines by adopting a feathering strategy to generate a cloudless graph of the target area. According to the method, automatic production from a remote sensing image to a cloudless image can be realized, and the processing efficiency of an image production task is remarkably improved through an automatic process, so that the image production quality is improved, the image production speed is increased, and manual operation is greatly reduced.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a remote sensing image automatic cloud removal and mosaicking method, device, equipment and storage medium. Background Art

[0002] Remote sensing images are widely used in environmental monitoring, agricultural surveys, urban planning and other fields. High-quality remote sensing image data can provide rich surface information and help accurately interpret surface information. In order to improve the efficiency of image interpretation and utilization, related technologies usually need to mosaic multiple remote sensing orthophotos into a large area of ​​cloud-free image to cover a wider surface area. The process usually includes steps such as cloud extraction, cloud and shadow removal, color grading and mosaicking, accompanied by a series of technical difficulties: relying on manual operation for image production tasks such as cloud removal, color grading and mosaicking. Although manual operation can check the image quality at any time, the complexity and technical requirements of manual operation limit production efficiency, and subjective differences affect the consistency and quality of the image. This means that in application scenarios where a large amount of remote sensing data needs to be quickly acquired and processed, the production cycle is long and the quality is difficult to unify, which affects the timeliness and practicality of the data. Automation and intelligence of image production are urgently needed.

[0003] The information disclosed in this background technology section is only intended to increase the understanding of the overall background of the application and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the purpose of the present application is to provide a method, device, equipment and storage medium for automatic cloud removal and mosaicking of remote sensing images that can avoid at least some of the above-mentioned technical defects.

[0005] In order to achieve the above-mentioned invention object, the technical solution provided by this application is as follows:

[0006] A remote sensing image automatic cloud removal mosaic method, comprising:

[0007] Extract cloud and shadow areas from remote sensing orthophotos using a pre-trained extraction model and generate vector files;

[0008] Based on the vector files of the cloud area and the target range, a cloud removal mosaic strategy within the target range is generated, and a vector combination table is generated;

[0009] Based on the cloud removal strategy in the vector combination table, image patches are selected and filled with clouds and shadows in the image to be processed to generate cloud-free remote sensing images;

[0010] Based on the mosaicking strategy in the vector combination table, the cloud-free remote sensing images are mosaicked in sequence, and the feathering strategy is used to eliminate the boundary lines to generate a cloud-free image of the target area.

[0011] Further: extracting clouds and shadow areas in the remote sensing orthophoto through the pre-trained extraction model and generating a vector file includes:

[0012] Input the target range vector and multiple DOM images to be processed, and only retain the DOM images within the target range according to the vector range;

[0013] Input the filtered DOM images and obtain the effective area vector files of each image;

[0014] Write image-related information into a vector file. The record content includes the image file corresponding to the vector, the image acquisition date, the image resolution, and the sensor type.

[0015] Further: based on the vector files of the cloud area and the target range, a cloud removal mosaic strategy within the target range is generated, and a vector combination table is generated, including:

[0016] Based on user input, prioritize images and their corresponding vectors;

[0017] Input the image and its corresponding vector, and reproject the image and vector to a consistent target reference coordinate system;

[0018] The target range vector is used as the reference vector, and the effective area vectors are erased in sequence. In each erasure, the overlap between the reference vector and the current effective area vector is checked, and the overlapping elements are removed from the reference vector. After the erasure is completed, the remaining vectors will be updated to the new reference vector. Repeat the process and use the next effective area vector for erasure until the reference vector no longer contains any elements, or the effective area vector has been traversed to the end.

[0019] Generates a mosaic table of prioritized polygon patches within the target extent, along with attribute information for each polygon.

[0020] Further: the cloud removal strategy in the vector combination table is used to select image patches and fill clouds and shadows in the image to be processed to generate a cloud-free remote sensing image, including:

[0021] Step 3.1: According to the order of the face elements in the combined table vector, count the non-repeated image names in the feature attributes, extract the unique identifier of each image, and the recorded image names and corresponding unique identifiers will be organized into a list of images to be declouded;

[0022] Step 3.2: Image traversal: traverse the image to be de-clouded and perform the following operations:

[0023] Step 3.3: Traverse the filled area features in order and perform the following steps for each filled area feature:

[0024] Step 3.3.1: Get the processing range; get the four boundaries of the filling area element as the processing range;

[0025] Step 3.3.2: Raster clipping: within the scope to be processed, clip the target raster data of the raster image corresponding to the target area element, and clip the filling raster data of the raster image corresponding to the filling area element;

[0026] Step 3.3.3: Convert vector to raster: rasterize the filled area elements and convert them into a binary raster image f, where the pixel values ​​in the valid area are marked as 1 and the pixel values ​​outside the valid area are marked as 0;

[0027] Step 3.3.4: Edge distance map generation: Input the binary raster image f, first generate the edge distance map, which is used to represent the distance from each pixel to the nearest edge, and use mean filtering;

[0028] Step 3.3.5: Weight map optimization: normalize the pixel values ​​of the edge distance map to the range of 0-1, and perform further smoothing to generate a feathering weight map;

[0029] Step 3.3.6: Generate declouding patches; according to the weight map, fuse the target raster data and the filling raster data to generate declouding patches;

[0030] Step 3.3.7: Cloud removal is completed; the cloud removal patch is weighted to replace the filled surface feature annotation area in the target image, thereby completing the cloud removal in the area;

[0031] Step 3.3.8 Repeat steps 3.3.1-3.3.7 until all the filled surface elements are traversed, and the cloud removal of the single-view image is completed;

[0032] Step 3.4: Repeat steps 3.2 and 3.3 until all the images to be declouded are traversed, that is, all images are declouded.

[0033] Further: the traversing of the to-be-declouded images includes:

[0034] According to the valid area vector corresponding to the image, determine whether it has clouds: if there are no clouds, skip to the next one; if there are no clouds, proceed to the subsequent steps;

[0035] After marking the raster image as the target image, obtain its four-dimensional range to determine the target range;

[0036] Within the target range, according to the order of the surface elements in the combined table vector, the surface elements that do not contain the target image in the feature attributes are selected as the filling surface elements, and their corresponding raster images are the filling images.

[0037] Further: the mosaicking strategy in the vector combination table is used to sequentially mosaic the cloud-free remote sensing images, and a feathering strategy is used to eliminate the boundary lines to generate a cloud-free image of the target area, including:

[0038] Step 4.1: Count the four-dimensional ranges of all images after cloud removal and convert them into the final four-dimensional range of the mosaicked image;

[0039] Step 4.2: Create a blank mosaic image based on the final four-boundary range. The size of the image will be calculated based on the determined four-boundary range and the required output resolution. Give the blank mosaic image appropriate coordinate system information, including: projection type, coordinate system and related spatial reference parameters;

[0040] Step 4.3: Traverse the images after cloud removal and perform the following steps for each image:

[0041] Step 4.3.1: After resampling the image to the resolution of the mosaic image, the resulting image will be marked as the image to be processed, and the valid pixel area will be identified and defined as the area to be processed;

[0042] Step 4.3.2: Extract the image to be processed and the mosaicked image in the area to be processed, mark them as raster data to be processed and mosaicked data, and check the pixel values ​​in the mosaicked data to determine whether the mosaicked data are all unfilled pixels: if all pixels are invalid values, it means that the area lacks valid information. If all are unfilled data, directly mark the raster data to be processed as output data and jump to step 4.3.7; otherwise, continue;

[0043] Step 4.3.3: Input the raster data to be processed and the mosaicked data, and take a single band and normalize it according to the pixel length of each valid pixel from the nearest image edge or invalid pixel to generate distance map dis1 and distance map dis2;

[0044] Step 4.3.4: Input distance map dis1 and distance map dis2 to generate mosaic weight map w mos ; The mosaic weight map generation rules are as follows:

[0045]

[0046] Step 4.3.5: Input the raster data to be processed, the mosaicked data, the uniform color template, and uniform color to the uniform color template;

[0047] Step 4.3.6: Feathering analysis: Use the mosaic weight map to perform weighted synthesis of the raster data to be processed and the mosaicked data, apply weights to different data sources, and perform linear interpolation for each pixel in the area to be processed according to its weight value in the weight map. This can achieve a feathering effect of smooth transition, thereby reducing the obvious switching at the boundary and generating the output data I of the area to be processed. mos ;

[0048] Step 4.3.7: Write the output data into the processing area corresponding to the mosaic image, update the mosaic image, and ensure that the data after each processing is correctly stored;

[0049] Step 4.3.8: Loop through 4.3.1-4.3.7 until the de-clouded images are traversed, and the mosaicking task is completed. After the traversal is completed, all the de-clouded images are successfully mosaicked, and the entire mosaicking task is completed.

[0050] Further:

[0051] The rule of feathering is I mos =w mos I trans1 +(1-w mos )I trans2 ;

[0052] The steps of color matching include:

[0053] Convert the image to be processed and the uniform color template into LAB color space for color correction;

[0054] Calculate the mean and variance of each component of the image to be processed and the uniform color template in the LAB space;

[0055] Perform component migration and adjust the LAB component of the image to be processed to align it with the mean of the uniform color template. The formula is as follows:

[0056]

[0057] Among them, L, A, B represent three spatial components, u and v represent the mean and standard deviation, and T represents the uniform color template image;

[0058] Merge the adjusted L, A, and B components and convert them to RGB color space to generate the uniformly colored raster data to be processed and the mosaicked data I trans1 ,I trans2 .

[0059] A remote sensing image automatic cloud removal and mosaicking device, comprising:

[0060] A vector file generation module is used to extract cloud and shadow areas in remote sensing orthophotos through a pre-trained extraction model and generate vector files;

[0061] A vector combination table generation module is used to generate a cloud removal mosaic strategy within the target range based on the vector files of the cloud area and the target range, and to generate a vector combination table;

[0062] A remote sensing image generation module is used to select image patches and fill in clouds and shadows in the image to be processed based on the cloud removal strategy in the vector combination table to generate a cloud-free remote sensing image;

[0063] The target area map generation module is used to mosaic the cloud-free remote sensing images in sequence based on the mosaic strategy in the vector combination table, and adopt the feathering strategy to eliminate the boundary lines to generate a cloud-free map of the target area.

[0064] An electronic device comprises at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the automatic de-clouding and mosaicking method for remote sensing images described in any embodiment of the present application.

[0065] A computer-readable storage medium, wherein the computer instructions are used to enable the computer to execute the remote sensing image automatic cloud removal and mosaicking method described in any embodiment of the present application.

[0066] The automatic de-clouding and mosaicking method for remote sensing images provided in this application can automatically plan the de-clouding and mosaicking strategy according to the requirements of different tasks, and realize the automated production of remote sensing images into a cloud-free image. The automated process significantly improves the processing efficiency of image production tasks, thereby improving the quality and speed of image production, greatly reducing manual operations, and can well meet the needs of practical applications.

[0067] Other features and advantages of the present application will be described in the subsequent description, and part of them will become apparent from the description, or some of them can be inferred or determined without doubt from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1This is a flow chart of a method for automatic cloud removal and mosaicking of remote sensing images according to an embodiment of the present application.

[0070] Figure 2 A schematic diagram of the process flow of a method for automatic cloud removal and mosaicking of remote sensing images according to an embodiment of the present application.

[0071] Figure 3 A schematic diagram of a planning combination table generation process according to an embodiment of the present application.

[0072] Figure 4 A schematic diagram of a cloud area and shadow filling strategy according to an embodiment of the present application.

[0073] Figure 5 This is a structural block diagram of a remote sensing image automatic cloud removal and mosaicking device according to an embodiment of the present application.

[0074] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0076] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0077] In view of the defects existing in the related art, the present application embodiment provides a remote sensing image automatic cloud removal mosaic method, such as Figure 1 As shown, the method may include steps S10 to S40:

[0078] S10, extracting cloud and shadow areas from remote sensing orthophotos through a pre-trained extraction model and generating vector files;

[0079] S20, based on the vector files of the cloud area and the target range, generating a cloud removal mosaic strategy within the target range, and generating a vector combination table;

[0080] S30, based on the cloud removal strategy in the vector combination table, selecting an image patch and filling clouds and shadows in the image to be processed to generate a cloud-free remote sensing image;

[0081] S40, based on the mosaicking strategy in the vector combination table, mosaicking the cloud-free remote sensing images in sequence, and using a gradual transition feathering strategy to eliminate boundary lines, thereby generating a cloud-free image of the target area.

[0082] The automatic cloud removal and mosaicking method for remote sensing images provided in the embodiment of the present application adopts a remote sensing image automatic cloud removal and mosaicking technology, which includes four steps: intelligent cloud detection, planning combination table generation, cloud area and shadow filling, and seamless mosaicking, to realize the automated production of remote sensing images into a cloud-free image, reduce manual participation in the production process, and achieve efficient image production.

[0083] refer to Figure 2 As shown, in another embodiment, the remote sensing image automatic cloud removal mosaic method may include the following steps:

[0084] Step 1: Intelligent cloud detection: extract cloud and shadow areas from remote sensing orthophotos through a pre-trained extraction model and generate vector files.

[0085] By automatically identifying cloud and shadow areas through pre-trained extraction models, clouds and shadow parts in remote sensing images can be extracted efficiently and accurately, reducing human intervention, improving the accuracy and consistency of cloud removal effects, and improving the automation and precision of cloud removal.

[0086] The pre-trained extraction model can be obtained in advance by performing model training on remote sensing image data based on a deep learning-based cloud detection algorithm.

[0087] Step 1 may include steps 1.1 to 1.3:

[0088] Step 1.1: Image screening: Input the target range vector and multiple DOM images to be processed, and only retain the DOM images within the target range according to the vector range;

[0089] Step 1.2: Obtain effective area vectors; input the filtered DOM images, and use the remote sensing image cloud detection method based on feature multi-scale perception and adaptive aggregation to obtain the effective area vector files of each image;

[0090] Step 1.3: Data information writing: Write image-related information into the vector file to ensure the traceability and systematicness of the data. The record content includes: the image file corresponding to the vector, the image acquisition date, the image resolution, the sensor type, etc.

[0091] Step 2: Plan the generation of the combination table, input the vector files of the cloud area and the target range, make adjustments based on specific mission requirements, formulate a cloud removal mosaic strategy within the target range, and generate a vector combination table;

[0092] like Figure 3 As shown in the figure, the vector files of the cloud area and target range are input, and based on specific mission requirements, they are adjusted according to different standards such as phase priority, resolution priority or specific order, a cloud removal mosaic strategy within the target range is formulated, and a vector combination table is generated, so that it can be adjusted according to different mission requirements and a targeted cloud removal mosaic strategy can be formulated. This flexibility enables the method to adapt to a variety of application scenarios, such as different image resolutions, different geographical areas or different time periods.

[0093] The specific implementation is as follows:

[0094] Step 2.1: Based on user input, determine the priority of images and their corresponding vectors, including phase priority, resolution priority, or user-defined order. In the case of phase priority, the system will sort the images according to the time of image acquisition, giving priority to newer images to ensure that the data used reflects the latest environmental conditions; in the strategy of resolution priority, the system will give priority to images with higher resolution to ensure the clarity and details of the images during the analysis process; in addition, the user can also enter a customized sorting order based on specific needs;

[0095] Step 2.2: Coordinate reprojection: Input the image and its corresponding vector, and reproject the image and vector to a consistent target reference coordinate system;

[0096] Step 2.3: Vector erasing; Using the target range vector as the reference vector, erase the valid area vectors in order. In each erasure, check the overlap between the reference vector and the current valid area vector, and remove the overlapping elements from the reference vector. After the erasure is completed, the remaining vectors will be updated to the new reference vector. Then, loop this process and repeatedly use the next valid area vector for erasing until the reference vector no longer contains any elements, or the valid area vector has been traversed to the end, finally achieving effective extraction and analysis of the target range;

[0097] Step 2.4: Generate a mosaic table of surface feature patches sorted by priority within the target range, as well as the attribute information of each surface feature. The attribute information is consistent with the description in step 1.3.

[0098] Step 3: Fill in the cloud area and shadows, such as Figure 4As shown in the figure, based on the cloud removal strategy in the vector combination table, the best image patch is automatically selected and the clouds and shadows in the image to be processed are filled, so as to generate a remote sensing image with no clouds and natural color transition, and realize automatic cloud area repair. Through the cloud removal strategy and image patch selection, the cloud and shadow areas can be automatically filled and the integrity of the image can be restored. This process greatly reduces the workload and time cost of repair and improves the processing efficiency.

[0099] The specific implementation is as follows:

[0100] Step 3.1: Image statistics: According to the order of the face elements in the combined table vector, count the non-repeated image names in the feature attributes, extract the unique identifier of each image, and the recorded image names and corresponding unique identifiers will be organized into a list of images to be de-clouded to ensure the integrity and accuracy of the data; by counting the unique identifiers in the images, ensure that the name and identifier of each image can accurately correspond. This provides a clear and systematic image list for subsequent image processing, avoids data redundancy or omission, and ensures the integrity and accuracy of data processing.

[0101] Step 3.2: Image traversal: traverse the image to be de-clouded and perform the following operations:

[0102] Step 3.2.1: Determine whether there are clouds; Based on the valid area vector corresponding to the image, determine whether it has clouds: If there are no clouds, skip to the next image; If there are no clouds, proceed to the subsequent steps; By determining whether the image contains clouds, invalid processing of cloud-covered areas is avoided. Skipping the processing of cloud-free images reduces unnecessary calculations, improves data processing efficiency, and avoids inaccurate results affecting analysis results.

[0103] Step 3.2.2: Image range acquisition: After marking the raster image as the target image, its four-dimensional range can be obtained to determine the target range. Specifically, by analyzing the geographic information of the raster image, its boundary coordinates can be extracted, including the northernmost, southernmost, easternmost, and westernmost coordinate values. These coordinate values ​​will be used to define the target range; by obtaining the four-dimensional range of the image through the geographic information of the raster image, the spatial position and size of the image can be clarified, thereby providing accurate geographic coordinate references for subsequent processing operations (such as image stitching, filling, etc.). This operation avoids the problem of image dislocation and inaccurate spatial position, and ensures the spatial consistency of data processing.

[0104] Step 3.2.3: Feature screening; within the target range, according to the order of the surface elements in the combined table vector, the surface elements that do not contain the target image in the feature attributes are screened as the filling surface elements, and their corresponding raster images are the filling images. Through this screening process, the complementary relationship between the filling surface elements and the target image can be ensured, thereby achieving effective filling processing; by combining the surface element order in the combined table vector, the surface elements that are complementary to the target image within the image range are screened. This process ensures the consistency in space and attributes between the filling surface elements and the target image. Reasonable screening of the filling image can improve the accuracy of the filling effect, avoid erroneous areas or inconsistent areas in the filling process, and thus ensure the quality of data processing.

[0105] Step 3.3: Traverse the filled area features in order and perform the following steps for each filled area feature:

[0106] Step 3.3.1: Get the processing range; get the four boundaries of the filling area element as the processing range;

[0107] Step 3.3.2: Raster clipping: within the scope to be processed, clip the target raster data of the raster image corresponding to the target area element, and clip the filling raster data of the raster image corresponding to the filling area element;

[0108] In steps 3.3.1 and 3.3.2, by accurately obtaining the four boundaries of the filled surface elements and clipping the image data based on the range, it is ensured that the cloud removal operation only occurs in the valid area of ​​the image, avoiding the processing of irrelevant areas. This not only improves the processing efficiency, but also reduces the errors or redundant calculations caused by unnecessary regional processing, and realizes accurate image clipping and processing.

[0109] Step 3.3.3: Convert vector to raster: rasterize the filled area elements and convert them into a binary raster image f, where the pixel values ​​in the valid area are marked as 1 and the pixel values ​​outside the valid area are marked as 0;

[0110] Step 3.3.3: By converting the filled surface elements into binary raster images, the valid and invalid areas (1 and 0) can be clearly marked, providing clear basic data for subsequent edge processing and weight calculation, and achieving accurate conversion of the filled surface elements. The binarization process further simplifies the subsequent calculation process and improves the processing efficiency and accuracy of the data.

[0111] Step 3.3.4: Edge distance map generation: Input binary raster image f, first generate edge distance map, which is used to represent the distance from each pixel to the nearest edge, and use mean filtering, with the filter kernel size set to N, to preliminarily smooth the image and reduce noise:

[0112]

[0114] In the formula, d(x,y) represents the output pixel value at the edge distance map (x,y).

[0115] Step 3.3.4 generates an edge distance map and performs a mean filter on it, which can help capture the changes in details in the image, especially when processing the transition area between cloud and non-cloud areas. The filtering operation effectively smooths the noise and avoids the rough edges of the cloud layer from affecting the declouding effect. This operation optimizes the processing of image edges, reduces the occurrence of artifacts or misjudgments, improves the smoothness and naturalness of the declouding results, and achieves accurate edge processing and smoothing.

[0116] Step 3.3.5: Weight map optimization; normalize the pixel values ​​of the edge distance map to the range of 0-1, and perform smoothing to generate the feathering weight map w rem , the normalization formula is as follows:

[0117]

[0118] The weight smoothing formula is as follows:

[0119]

[0120] Step 3.3.5 generates a feathering weight map by normalizing and weight smoothing the edge distance map, which can effectively control the transition effect between the de-clouded area and the surrounding non-cloudy area. The smoothed weight map helps to avoid hard boundaries, thereby achieving a natural fusion of cloud and non-cloud areas. This operation makes the de-clouding process more refined, reduces the abruptness of the transition area, improves the visual quality of the image, optimizes the weight map, and enhances the de-clouding effect.

[0121] Step 3.3.6: Generate declouding patches; according to the weight map, fuse the target raster data p cloud and fill grid data p source , generate the cloud removal patch p rem , the fusion rules are as follows:

[0122] p rem =w rem p cloud +(1-w rem ) source (4)

[0123] Step 3.3.6 combines the target raster data and the filling raster data and fuses them according to the weight map to generate cloud removal patches. This step can adjust the weights according to the needs of different regions to achieve a more accurate cloud removal effect. The formulation of fusion rules ensures that the target area is consistent with the original image after cloud removal, thereby improving the integrity and continuity of the image content.

[0124] Step 3.3.7: Cloud removal is completed; the cloud removal patch is used to replace the filled surface feature annotation area in the target image to complete the cloud removal in the area;

[0125] Step 3.3.8: Repeat steps 3.3.1-3.3.7 until all the filled surface elements are traversed, thus completing the cloud removal of the single-view image.

[0126] Step 3.4: Repeat steps 3.2 and 3.3 until all the images to be declouded are traversed, that is, all images are declouded.

[0127] The steps described above mainly involve declouding operations in image processing, especially the processing of filled surface features and the generation of declouding patches. This series of steps can effectively improve the accuracy and efficiency of image processing. Through the fine processing of image data, from image cropping to cloud removal and detail optimization, it can effectively remove clouds while maintaining the original features of the image, and finally generate high-quality declouded images. The automated process greatly improves the processing efficiency. The precise edge and weight processing ensures the natural transition and high-quality output of the declouding results, which is suitable for batch processing and analysis of large-scale image data. This method has important application value in the fields of remote sensing images, satellite images and environmental monitoring.

[0128] Step 4: Seamless mosaicking. Based on the mosaicking strategy in the vector combination table, mosaic the de-clouded remote sensing images in sequence, even the colors, and use a gradual transition feathering strategy to eliminate the boundary lines to generate a cloud-free image of the target area.

[0129] The use of a gradual transition feathering strategy to eliminate the boundary lines of the mosaicked image can effectively avoid obvious seams or unnatural transitions during the stitching process, ensuring that the visual effect of the final image is more natural and seamless. The specific implementation is as follows:

[0130] Step 4.1: Range estimation: Count the four-dimensional ranges of all images after cloud removal and convert them into the final four-dimensional range of the mosaicked image;

[0131] Step 4.2: Create mosaicked image; Based on the final four-boundary range, first create a blank mosaicked image. The size of the image will be calculated based on the determined four-boundary range and the required output resolution. Then, assign appropriate coordinate system information to the blank mosaicked image, including: projection type, coordinate system and related spatial reference parameters;

[0132] Step 4.3: Traverse the images after cloud removal and perform the following steps for each image:

[0133] Step 4.3.1: Image resampling: After the image is resampled to the resolution of the mosaic image, the generated image will be marked as the image to be processed, and the valid pixel area will be identified and defined as the area to be processed;

[0134] Step 4.3.2: Extract the image to be processed and the mosaicked image in the area to be processed, mark them as raster data to be processed and mosaicked data, and check the pixel values ​​in the mosaicked data to determine whether the mosaicked data are all unfilled pixels: if all pixels are invalid values, it means that the area lacks valid information. If all are unfilled data, directly mark the raster data to be processed as output data and jump to step 4.3.7; otherwise, continue with step 4.3.3;

[0135] By checking the pixel values ​​in the mosaicked data, it is possible to automatically identify which areas have no valid data, and then decide whether to directly mark the raster data to be processed as output data. This can automatically filter out areas that do not need further processing and improve data processing efficiency. In addition, it can also reduce unnecessary calculations. If the pixel values ​​in the mosaicked data area are all invalid values ​​(that is, all unfilled data), it means that there is no valid image data in the area. In this case, it can automatically determine that the area is invalid, thereby avoiding additional and meaningless calculations and processing, reducing resource waste, and significantly improving processing efficiency, especially when processing large-scale data. Further, by detecting and skipping invalid areas, it can ensure that only areas containing valid data are processed, thereby improving the quality of the final output data and avoiding erroneous results or inaccurate output caused by interference from invalid data. By detecting valid pixels in the mosaicked image, it can ensure that the raster data to be processed is directly marked as output data when it meets the conditions, which helps more accurate image synthesis or image stitching process. The integrity and consistency of the data are guaranteed, especially in the scene of stitching or fusion of multiple images.

[0136] In addition, the ability to effectively respond to data areas in different situations makes the processing process more flexible. For example, if there are missing or invalid values ​​in the data of certain areas, the system can automatically adapt to this situation and skip or mark these areas, thus avoiding the need for manual intervention and making the processing process more intelligent and adaptive.

[0137] Step 4.3.3: Distance map generation: Input the raster data to be processed and the mosaicked data, take a single band and normalize it according to the pixel length of each valid pixel from the nearest image edge or invalid pixel, and generate distance map dis1 and distance map dis2;

[0138] Step 4.3.4: Mosaic weight map generation; input distance map dis1 and distance map dis2 to generate mosaic weight map w mos ; The mosaic weight map generation rules are as follows:

[0139]

[0140] Step 4.3.5: Color grading: input the raster data to be processed, the mosaicked data, and the color grading template, and grading to the color grading template; the color grading method is as follows:

[0141] (1) Convert the image to be processed and the uniform color template into the LAB color space for color correction;

[0142] (2) Calculate the mean and variance of each component of the image to be processed and the uniform color template in the LAB space;

[0143] (3) Perform component migration and adjust the LAB component of the image to be processed so that it is aligned with the mean value of the uniform color template. The formula is as follows:

[0144]

[0145]

[0146] In the formula, L, A, and B represent three spatial components, u and v represent the mean and standard deviation, and T represents the uniform color template image.

[0147] (4) Merge the adjusted L, A, and B components and convert them to RGB color space to generate the uniformly colored raster data to be processed and the mosaicked data I trans1 ,I trans2 ;

[0148] Step 4.3.6: Feathering analysis: Use the mosaic weight map to perform weighted synthesis of the raster data to be processed and the mosaicked data, apply weights to different data sources, and perform linear interpolation for each pixel in the area to be processed according to its weight value in the weight map. This can achieve a feathering effect of smooth transition, thereby reducing the obvious switching at the boundary and generating the output data I of the area to be processed. mos ; The feathering rules are as follows:

[0149] I mos =w mos I trans1 +(1-w mos )I trans2 (9)

[0150] Step 4.3.7: Mosaic data writing: writing the output data into the to-be-processed area corresponding to the mosaic image;

[0151] Step 4.3.8: Repeat 4.3.1-4.3.6 until all the images after de-clouding are traversed, and the mosaicking task is completed.

[0152] This implementation adopts a remote sensing image automatic cloud removal and mosaicking technology, which includes four steps: intelligent cloud detection, planning combination table generation, cloud area and shadow filling, and seamless mosaicking. It realizes the automated production of remote sensing images into a cloud-free picture, reduces manual participation in the production process, and achieves efficient image production.

[0153] The automatic declouding and mosaicking method for remote sensing images in the embodiment of the present application adopts a flexible declouding and mosaicking strategy. Through the declouding and mosaicking strategy, a complete and high-quality remote sensing image can be generated to ensure that the image data of the target area is cloudless and shadow-free, and the application effect of the image in subsequent analysis, modeling, monitoring, etc. is improved. The automated declouding and mosaicking process can reduce manual operations and human intervention, and improve the speed and efficiency of remote sensing image processing. At the same time, due to the flexibility and adaptability of the method, it can be widely used in different types of remote sensing data and task requirements. In addition, this method supports large-scale remote sensing image processing. The automation characteristics of the method enable it to be used for large-scale remote sensing image processing tasks, suitable for large-scale geographical areas or high-frequency remote sensing image data processing, and has good scalability. Therefore, this method can not only improve the accuracy and automation of declouding and mosaicking, but also optimize the visual effects of remote sensing images and the effectiveness of subsequent applications, and has great advantages in the field of remote sensing image processing.

[0154] refer to Figure 5 As shown, another embodiment of the present application provides a remote sensing image automatic cloud removal and mosaicking device, comprising:

[0155] A vector file generation module is used to extract cloud and shadow areas in remote sensing orthophotos through a pre-trained extraction model and generate vector files;

[0156] A vector combination table generation module is used to generate a cloud removal mosaic strategy within the target range based on the vector files of the cloud area and the target range, and to generate a vector combination table;

[0157] A remote sensing image generation module is used to select image patches and fill in clouds and shadows in the image to be processed based on the cloud removal strategy in the vector combination table to generate a cloud-free remote sensing image;

[0158] The target area map generation module is used to mosaic the cloud-free remote sensing images in sequence based on the mosaic strategy in the vector combination table, and adopt the feathering strategy to eliminate the boundary lines to generate a cloud-free map of the target area.

[0159] Exemplarily, the method of extracting cloud and shadow areas in a remote sensing orthophoto using a pre-trained extraction model and generating a vector file includes:

[0160] Input the target range vector and multiple DOM images to be processed, and only retain the DOM images within the target range according to the vector range;

[0161] Input the filtered DOM images and obtain the effective area vector files of each image;

[0162] Write image-related information into a vector file. The record content includes the image file corresponding to the vector, the image acquisition date, the image resolution, and the sensor type.

[0163] Exemplarily, based on the vector files of the cloud area and the target range, a cloud removal mosaic strategy within the target range is generated, and a vector combination table is generated, including:

[0164] Based on user input, prioritize images and their corresponding vectors;

[0165] Input the image and its corresponding vector, and reproject the image and vector to a consistent target reference coordinate system;

[0166] The target range vector is used as the reference vector, and the effective area vectors are erased in sequence. In each erasure, the overlap between the reference vector and the current effective area vector is checked, and the overlapping elements are removed from the reference vector. After the erasure is completed, the remaining vectors will be updated to the new reference vector. Repeat the process and use the next effective area vector for erasure until the reference vector no longer contains any elements, or the effective area vector has been traversed to the end.

[0167] Generates a mosaic table of prioritized polygon patches within the target extent, along with attribute information for each polygon.

[0168] Exemplarily, the traversing the to-be-declouded images includes:

[0169] According to the valid area vector corresponding to the image, determine whether it has clouds: if there are no clouds, skip to the next one; if there are no clouds, proceed to the subsequent steps;

[0170] After marking the raster image as the target image, obtain its four-dimensional range to determine the target range;

[0171] Within the target range, according to the order of the surface elements in the combined table vector, the surface elements that do not contain the target image in the feature attributes are selected as the filling surface elements, and their corresponding raster images are the filling images.

[0172] Another embodiment of the present application provides an electronic device, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the automatic de-clouding and mosaicking method for remote sensing images described in any embodiment of the present application.

[0173] Another embodiment of the present application provides a computer-readable storage medium, characterized in that the computer instructions are used to enable the computer to execute the automatic de-clouding and mosaicking method for remote sensing images described in any embodiment of the present application.

[0174] refer to Figure 6 , which is a block diagram of an electronic device according to an embodiment of the present application. The electronic device may represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframe computers, and other types of computing devices.

[0175] In addition, the electronic device may also be various mobile devices, such as smart phones, wearable devices, cellular phones, personal digital assistants, and other similar devices. The components, their connection methods, relationships, and functions mentioned herein are only examples, and are intended to assist understanding, but are not intended to limit the implementation methods described or required in this application.

[0176] like Figure 6 As shown, the electronic device may include one or more processors 101, a memory 102, and is equipped with interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected through different types of buses, and can also be installed on the same motherboard as needed, or assembled in other ways. The processor is responsible for executing instructions in the electronic device, which are stored in the memory, or loaded and executed by the memory, and are used to present graphical information of a graphical user interface (GUI) on an external input / output device (such as a display device connected to the interface). If necessary, multiple processors, multiple buses, and multiple memories can work together to provide higher computing power. Similarly, multiple electronic devices can also be interconnected to achieve their respective partial functions through collaboration (such as a server array, a blade server cluster, or a multi-processor system). Figure 6 The illustration in FIG. 1 shows a case of one processor 101 .

[0177] In the example of the present application, the memory 102 is a non-transient computer-readable storage medium that stores instructions executable by at least one processor, which enable the processor to execute the method provided in the present application. The memory 102 includes a program storage area and a data storage area, the former is used to store an operating system and related applications, and the latter is used to store data generated during the use of the electronic device. In addition, the memory 102 may also include a high-speed random access memory (RAM) and a non-transient memory, such as a disk storage, a flash memory device, or other solid-state memory.

[0178] In some embodiments, the memory 102 may also include a storage device remotely configured relative to the processor 101, and these remote memories may be connected to the electronic device via a network. Common networks include the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The electronic device may also be equipped with an input device 103 and an output device 104, and the processor 101, the memory 102, the input device 103, and the output device 104 may be connected via a bus or other means. Figure 6 The method adopted is through bus connection.

[0179] It should be noted that, in the description of the present application, if there are reference terms such as "one embodiment", "some embodiments", "examples", "specific examples" or "some examples", the description of such reference terms means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification. In the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0180] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the request processing method according to an embodiment of the present invention. The present invention may also be implemented as a device or system program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A remote sensing image automatic cloud removal mosaic method, characterized in that: include: Extract cloud and shadow areas from remote sensing orthophotos using a pre-trained extraction model and generate vector files; Based on the vector files of the cloud area and the target range, a cloud removal mosaic strategy within the target range is generated, and a vector combination table is generated; Based on the cloud removal strategy in the vector combination table, image patches are selected and filled with clouds and shadows in the image to be processed to generate cloud-free remote sensing images; Based on the mosaicking strategy in the vector combination table, the cloud-free remote sensing images are mosaicked in sequence, and the feathering strategy is used to eliminate the boundary lines to generate a cloud-free image of the target area.

2. The method according to claim 1, characterized in that: The pre-trained extraction model is used to extract cloud and shadow areas in remote sensing orthophotos and generate vector files, including: Input the target range vector and multiple DOM images to be processed, and only retain the DOM images within the target range according to the vector range; Input the filtered DOM images and obtain the effective area vector files of each image; Write image-related information into a vector file. The record content includes the image file corresponding to the vector, the image acquisition date, the image resolution, and the sensor type.

3. The method according to claim 1, characterized in that: Based on the vector files of the cloud area and the target range, a cloud removal mosaic strategy within the target range is generated, and a vector combination table is generated, including: Based on user input, prioritize images and their corresponding vectors; Input the image and its corresponding vector, and reproject the image and vector to a consistent target reference coordinate system; The target range vector is used as the reference vector, and the effective area vectors are erased in sequence. In each erasure, the overlap between the reference vector and the current effective area vector is checked, and the overlapping elements are removed from the reference vector. After the erasure is completed, the remaining vectors will be updated to the new reference vector. Repeat the process and use the next effective area vector for erasure until the reference vector no longer contains any elements, or the effective area vector has been traversed to the end. Generates a mosaic table of prioritized polygon feature patches within the target extent, along with attribute information for each polygon feature.

4. The method according to claim 1, characterized in that: The cloud removal strategy based on the vector combination table, selecting image patches and filling clouds and shadows in the image to be processed to generate a cloud-free remote sensing image, includes: Step 3.1: According to the order of the face elements in the combined table vector, count the non-repeated image names in the feature attributes, extract the unique identifier of each image, and the recorded image names and corresponding unique identifiers will be organized into a list of images to be declouded; Step 3.2: Image traversal: traverse the image to be de-clouded and perform the following operations: Step 3.3: Traverse the filled area features in order and perform the following steps for each filled area feature: Step 3.3.1: Get the processing range; get the four boundaries of the filling area element as the processing range; Step 3.3.2: Raster clipping: within the scope to be processed, clip the target raster data of the raster image corresponding to the target area element, and clip the filling raster data of the raster image corresponding to the filling area element; Step 3.3.3: Convert vector to raster: rasterize the filled area elements and convert them into a binary raster image f, where the pixel values ​​in the valid area are marked as 1 and the pixel values ​​outside the valid area are marked as 0; Step 3.3.4: Edge distance map generation: Input the binary raster image f, first generate the edge distance map, which is used to represent the distance from each pixel to the nearest edge, and use mean filtering; Step 3.3.5: Weight map optimization: normalize the pixel values ​​of the edge distance map to the range of 0-1, and perform further smoothing to generate a feathering weight map; Step 3.3.6: Generate declouding patches; according to the weight map, fuse the target raster data and the filling raster data to generate declouding patches; Step 3.3.7: Cloud removal is completed; the cloud removal patch is weighted to replace the filled surface feature annotation area in the target image, thereby completing the cloud removal in the area; Step 3.3.8 Repeat steps 3.3.1-3.3.7 until all the filled surface elements are traversed, and the cloud removal of the single-view image is completed; Step 3.4: Repeat steps 3.2 and 3.3 until all the images to be declouded are traversed, that is, all images are declouded.

5. The method according to claim 1, characterized in that: The traversing of the to-be-declouded images includes: According to the valid area vector corresponding to the image, determine whether it has clouds: if there are no clouds, skip to the next one; if there are no clouds, proceed to the subsequent steps; After marking the raster image as the target image, obtain its four-dimensional range to determine the target range; Within the target range, according to the order of the surface elements in the combined table vector, the surface elements that do not contain the target image in the feature attributes are selected as the filling surface elements, and their corresponding raster images are the filling images.

6. The method according to claim 1, characterized in that: The mosaicking strategy based on the vector combination table is used to mosaic the cloud-free remote sensing images in sequence, and the feathering strategy is used to eliminate the boundary lines to generate a cloud-free image of the target area, including: Step 4.1: Count the four-dimensional ranges of all images after cloud removal and convert them into the final four-dimensional range of the mosaicked image; Step 4.2: Create a blank mosaic image based on the final four-boundary range. The size of the image will be calculated based on the determined four-boundary range and the required output resolution. Give the blank mosaic image appropriate coordinate system information, including: projection type, coordinate system and related spatial reference parameters; Step 4.3: Traverse the images after cloud removal and perform the following steps for each image: Step 4.3.1: After resampling the image to the resolution of the mosaic image, the resulting image will be marked as the image to be processed, and the valid pixel area will be identified and defined as the area to be processed; Step 4.3.2: Extract the image to be processed and the mosaicked image in the area to be processed, mark them as raster data to be processed and mosaicked data, and check the pixel values ​​in the mosaicked data to determine whether the mosaicked data are all unfilled pixels: if all pixels are invalid values, it means that the area lacks valid information. If all are unfilled data, directly mark the raster data to be processed as output data and jump to step 4.3.7; otherwise, continue; Step 4.3.3: Input the raster data to be processed and the mosaicked data, and take a single band and normalize it according to the pixel length of each valid pixel from the nearest image edge or invalid pixel to generate distance map dis1 and distance map dis2; Step 4.3.4: Input distance map dis1 and distance map dis2 to generate mosaic weight map w mos ; The mosaic weight map generation rules are as follows: Step 4.3.5: Input the raster data to be processed, the mosaicked data, the uniform color template, and uniform color to the uniform color template; Step 4.3.6: Feathering analysis: Use the mosaic weight map to perform weighted synthesis of the raster data to be processed and the mosaicked data, apply weights to different data sources, and perform linear interpolation for each pixel in the area to be processed according to its weight value in the weight map. This can achieve a feathering effect of smooth transition, thereby reducing the obvious switching at the boundary and generating the output data I of the area to be processed. mos ; Step 4.3.7: Write the output data into the processing area corresponding to the mosaic image, update the mosaic image, and ensure that the data after each processing is correctly stored; Step 4.3.8: Loop through 4.3.1-4.3.7 until the de-clouded images are traversed, and the mosaicking task is completed. After the traversal is completed, all the de-clouded images are successfully mosaicked, and the entire mosaicking task is completed.

7. The method according to claim 6, characterized in that: The rule of feathering is I mos =w mos I trans1 +(1-w mos )I trans2 ; The steps of color matching include: Convert the image to be processed and the uniform color template into LAB color space for color correction; Calculate the mean and variance of each component of the image to be processed and the uniform color template in the LAB space; Perform component migration and adjust the LAB component of the image to be processed to align it with the mean of the uniform color template. The formula is as follows: Among them, L, A, B represent three spatial components, u and v represent the mean and standard deviation, and T represents the uniform color template image; Merge the adjusted L, A, and B components and convert them to RGB color space to generate the uniformly colored raster data to be processed and the mosaicked data I trans1 ,I trans2 .

8. A remote sensing image automatic cloud removal and mosaicking device, characterized in that: include: A vector file generation module is used to extract cloud and shadow areas in remote sensing orthophotos through a pre-trained extraction model and generate vector files; A vector combination table generation module is used to generate a cloud removal mosaic strategy within the target range based on the vector files of the cloud area and the target range, and to generate a vector combination table; A remote sensing image generation module is used to select image patches and fill in clouds and shadows in the image to be processed based on the cloud removal strategy in the vector combination table to generate a cloud-free remote sensing image; The target area map generation module is used to mosaic the cloud-free remote sensing images in sequence based on the mosaic strategy in the vector combination table, and adopt the feathering strategy to eliminate the boundary lines to generate a cloud-free map of the target area.

9. An electronic device, characterized in that: It includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the remote sensing image automatic de-clouding and mosaicking method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer instructions are used to enable the computer to execute the remote sensing image automatic cloud removal and mosaicking method described in any one of claims 1 to 7.

Images