A remote sensing image reconstruction method and device based on SAR images and similarity analysis, equipment, medium and product

Abstract

The application discloses a remote sensing image reconstruction method and device based on SAR images and similarity analysis, equipment, medium and product, and relates to the technical field of remote sensing data processing. The method is to obtain target area and the same size of the to-be-repaired remote sensing image, reference remote sensing image and SAR image, retrieve a plurality of similar reference image elements from the reference remote sensing image, then perform secondary screening on the similar image element set according to the similarity of the overlapping auxiliary image elements in the SAR image and the target overlapping auxiliary image elements, then perform spatial and temporal based image element value reconstruction on the missing image elements based on the similar image element set, and finally fuse the remote sensing image values of the missing image elements according to the spatial and temporal based reconstruction results, so that the robustness, effectiveness, detail performance and visual consistency of the reconstructed image can be effectively improved in the area where the ground surface changes frequently, and the method has an advantage in the same type of reconstruction method.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing data processing technology, specifically relating to a method, apparatus, equipment, medium, and product for remote sensing image reconstruction based on SAR imagery and similarity analysis. Background Technology

[0002] Optical remote sensing imagery plays a crucial role in numerous fields, including environmental monitoring, agriculture, urban planning, and disaster management. However, the presence of thick clouds significantly impacts the visual quality and data performance of these images. A study based on a medium-resolution imaging spectrometer indicates that the long-term average global cloud cover is approximately 60%. Other research suggests that Landsat imagery (i.e., remote sensing imagery provided by the Landsat imagery data service system) can have cloud cover exceeding 70% in some tropical regions. Extensive thick clouds lead to a loss of remote sensing data integrity, limiting the usability of these already valuable images. For example, time-series analysis tasks require earlier years' images, which are scarce, thus greatly interfering with subsequent applications. Thick cloud removal can increase the effective information in a single image, reduce the number of revisits and costs associated with acquiring cloud-free images, and enhance image quality, thereby ensuring the effectiveness of remote sensing imagery in subsequent applications across multiple fields. Therefore, thick cloud removal of remote sensing images is highly significant.

[0003] Unlike thin clouds, thick clouds are opaque to all optical bands in remote sensing imagery. Therefore, thick cloud removal and processing techniques are a complex but crucial part of remote sensing data processing. Because pixel loss is so severe as to leave no trace, thick cloud removal generally relies on a certain amount of reference data. Numerous researchers have explored various reference data, including cloudless optical image data and synthetic aperture radar (SAR) data. Based on the differences in the data used in the processing, existing thick cloud removal methods can be broadly categorized into: spatial interpolation-based methods, multi-temporal-based methods, and spatiotemporal fusion-based methods.

[0004] Spatial interpolation-based thick cloud removal is a common and effective technique for filling in areas missing in remote sensing images due to thick cloud cover. Its development has undergone various technological evolutions and optimizations. Early methods primarily stemmed from improvements to traditional image processing interpolation methods, such as nearest-neighbor interpolation, inverse distance weighted interpolation, and bilinear interpolation. These improvements enhanced computational efficiency and interpolation accuracy. Interpolation methods geared towards geographic information science, such as spline interpolation and Kriging interpolation, incorporated statistical and mathematical models. These methods can handle more complex spatial correlations and other geographic features, becoming mainstream methods for a certain period. Machine learning-based methods, based on large amounts of data and complex models, perform better in handling complex and nonlinear problems, further improving thick cloud removal. Overall, spatially based thick cloud removal methods typically rely on local information in the image, effectively handling details and features of local areas. For static or relatively unchanging scenes, spatially based methods can provide high-quality reconstruction results with lower time and computational costs. However, when there are large gaps and significant spatial heterogeneity in the image, spatial-based methods may not provide accurate results.

[0005] Multi-temporal thick cloud removal methods use spectral, shape, and category information from revisited images with similar reference times as a reference for thick cloud image reconstruction. The earliest multi-temporal thick cloud removal method was multi-temporal synthesis, which constructs a complete cloud-free image by synthesizing cloud-free areas at different time points. This method is simple and intuitive, and can achieve good reconstruction results for areas with high observation frequency. However, this is also a drawback, as obtaining high-revisit satellite images is often difficult. Methods based on time series analysis process multi-temporal data by establishing time series models or spatiotemporal statistical methods to repair pixels in cloud-occluded areas. Examples include multi-temporal image reconstruction using coupled tensor analysis, and cloud detection and removal based on low-rank and group sparsity regularization. In recent years, many researchers have used machine learning methods to construct nonlinear relationships between time series images. In general, multi-temporal thick cloud removal methods use revisited images from similar times as reference images, comprehensively considering the mapping relationship between the reference and target images. This allows them to capture changes within a certain temporal range and, compared to methods based solely on a single image space, can reconstruct larger cloud-covered areas, thus improving thick cloud removal performance. However, this method assumes that images across multiple temporal phases have a consistent land cover type, and pixels identified as similar in the reference image are assumed to be applicable to the target image. Because this type of method cannot capture the changes and trends within the region, it is not suitable for frequently changing dynamic scenes.

[0006] The method for removing thick clouds from remote sensing images based on temporal and spatial fusion combines spatial and temporal information. It utilizes not only the spatial relationships between individual image pixels but also the changing patterns of long-term image sequences to complete the image of areas covered by thick clouds. This approach has been used in the development of methods from neighbor-similar pixel interpolation for Landsat 7 strip removal to improved neighbor-similar pixel interpolation for thick cloud removal. Similarly, combining the spatiotemporal fusion approach with deep learning, such as spatiotemporal attention networks, three-input three-output networks, and hierarchical and structure-preserving fusion networks, has achieved further success in the task of removing thick clouds from remote sensing images. Overall, based on recent work, the spatiotemporal fusion-based method is the most robust and performs best compared to spatial interpolation-based and multi-temporal-based methods.

[0007] Currently, determining whether and how pixels have changed under thick clouds is a key issue that needs to be addressed when applying thick cloud removal methods to remote sensing image restoration in frequently changing areas. Among the aforementioned methods based on spatial interpolation, multi-temporal phases, and spatiotemporal fusion, there is an assumption that the differences between the areas to be restored in images from adjacent temporal phases are small. This is because these methods only use optical images, while thick clouds are opaque to all optical images. Although many scholars have made various attempts from an algorithmic perspective, there is still room for improvement in their effectiveness. Synthetic Aperture Radar (SAR), due to its active microwave imaging principle, can penetrate clouds, smoke, and rain, and is unaffected by diurnal variations, enabling it to perform surface monitoring tasks under various weather conditions. In recent years, SAR has achieved numerous research results in thick cloud removal from optical images. For example, by using a transformation network, the model can gradually learn the correlation between SAR (Synthetic Aperture Radar) imagery and optical imagery, thereby applying the model to cloud-covered areas to generate results similar to cloudless images; based on the layered fusion of optical and SAR remote sensing images, cloud and shadow areas in optical remote sensing images can be restored; using synthetic aperture radar images and low-resolution heterogeneous images as reference images, information can be restored from high-resolution images with cloud contamination; and so on.

[0008] Currently, a key challenge in using SAR data for thick cloud removal lies in the difference in imaging mechanisms between SAR and optical images. This difference leads to significant variations in the appearance of ground features within the same area in the two types of images. Directly using SAR images as the data source for image reconstruction leaves room for improvement in detail and visual consistency. However, the valuable information about ground features beneath thick clouds contained in SAR images remains significant. Using this information as supplementary data to assist in the restoration of thick cloud areas from optical images may be a worthwhile approach to explore. Summary of the Invention

[0009] The purpose of this invention is to provide a remote sensing image reconstruction method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on SAR imagery and similarity analysis, in order to solve the problem that existing thick cloud removal techniques, which directly use SAR imagery as a data source in image reconstruction, still have room for improvement in terms of detail and visual consistency of the reconstructed images.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a remote sensing image reconstruction method based on SAR imagery and similarity analysis is provided, including: Acquire remote sensing images, reference remote sensing images, and SAR images of the target area that are of the same size. The remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image that has no missing pixels at the position corresponding to the missing pixels. Multiple similar reference pixels that are similar to the target overlapping reference pixel are retrieved from the reference remote sensing image, wherein the target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel; For each missing overlapping reference pixel in the set of similar pixels composed of the plurality of similar reference pixels, firstly, the missing overlapping auxiliary pixel in the SAR image and corresponding to it is determined. Then, the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel is calculated. Finally, if the similarity does not meet the pixel retention condition, the corresponding pixel is removed from the set of similar pixels. Here, the missing overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to other missing pixels in the remote sensing image to be repaired. The missing overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the other missing pixel. The target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel. Based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the set of similar pixels, the initial remote sensing image value of the missing pixel is reconstructed using an interpolation method, and the reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel, wherein the non-missing overlapping reference pixel refers to other pixels in the set of similar pixels besides all the missing overlapping reference pixels. Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, a remote sensing image value prediction model for the missing pixel is obtained by calibration and verification modeling based on a time series model. The remote sensing image value of the target overlapping reference pixel is then imported into the remote sensing image value prediction model, and the reconstructed pixel value of the missing pixel based on time is output. The remote sensing image value of the missing pixel is obtained by fusing the reconstructed pixel values ​​based on the missing pixel and the spatial and temporal values ​​respectively.

[0011] Secondly, a remote sensing image reconstruction device based on SAR imagery and similarity analysis is provided, including an image data acquisition unit, a similar pixel initial selection unit, a similar pixel re-selection unit, a spatial dimension reconstruction unit, a temporal dimension reconstruction unit, and a reconstruction result fusion unit. The image data acquisition unit is used to acquire remote sensing images to be repaired, reference remote sensing images, and SAR images of the target area with the same size. The remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image without missing pixels at the position corresponding to the missing pixels. The similar pixel initial selection unit is communicatively connected to the image data acquisition unit and is used to retrieve multiple similar reference pixels that are similar to the target overlapping reference pixel from the reference remote sensing image. The target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel. The similar pixel reselection unit is communicatively connected to the image data acquisition unit and the similar pixel initial selection unit, respectively. It is used to, for each missing overlapping reference pixel in the similar pixel set composed of the plurality of similar reference pixels, first determine the corresponding missing overlapping auxiliary pixel in the SAR image, then calculate the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, and finally, if the similarity does not meet the pixel retention condition, remove the corresponding pixel from the similar pixel set. Here, the missing overlapping reference pixel refers to the pixel in the reference remote sensing image located at a position corresponding to other missing pixels in the remote sensing image to be repaired; the missing overlapping auxiliary pixel refers to the pixel in the SAR image located at a position corresponding to the other missing pixel; and the target overlapping auxiliary pixel refers to the pixel in the SAR image located at a position corresponding to the missing pixel. The spatial dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to reconstruct the initial remote sensing image value of the missing pixel using an interpolation method based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the similar pixel set. The reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel. The non-missing overlapping reference pixel refers to other pixels in the similar pixel set other than all the missing overlapping reference pixels. The time-dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to obtain a remote sensing image value prediction model for the missing pixel based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, and to import the remote sensing image value of the target overlapping reference pixel into the remote sensing image value prediction model, and output the time-based reconstructed pixel value of the missing pixel. The reconstruction result fusion unit is communicatively connected to the spatial dimension reconstruction unit and the temporal dimension reconstruction unit, respectively, and is used to fuse the remote sensing image value of the missing pixel based on the reconstructed pixel value based on space and time respectively.

[0012] Thirdly, the present invention provides a computer device comprising a storage module, a processing module, and a transceiver module connected in sequence for communication, wherein the storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the remote sensing image reconstruction method as described in the first aspect or any possible design in the first aspect.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the remote sensing image reconstruction method as described in the first aspect or any possible design in the first aspect.

[0014] Fifthly, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the remote sensing image reconstruction method as described in the first aspect or any possible design in the first aspect.

[0015] The beneficial effects of the above scheme are: (1) This invention creatively provides a new scheme for spatiotemporal integrated reconstruction of remote sensing images to be repaired based on SAR data and similarity analysis results. That is, after acquiring remote sensing images to be repaired, reference remote sensing images and SAR images of the same size in the target area, multiple similar reference pixels similar to the overlapping reference pixels of the target are retrieved from the reference remote sensing image. Then, the similar pixel set is screened a second time according to the similarity between the overlapping auxiliary pixels in the SAR image and the overlapping auxiliary pixels of the target. Then, the missing pixels are reconstructed based on space and time according to the similar pixel set. Finally, the remote sensing image value of the missing pixels is obtained by fusing the reconstruction results based on space and time. Through ablation experiments and comparative experiments, it is proved that it can effectively improve robustness, effectiveness and detail performance and visual consistency of the reconstructed image in areas where the surface changes frequently. Thus, it has advantages among similar reconstruction methods. By combining experimental results in real scenarios and analysis of running time costs, it is proved that it has the potential to be applied to large-scale actual production scenarios, which is convenient for practical application and promotion. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the remote sensing image reconstruction method based on SAR imagery and similarity analysis provided in the embodiments of this application.

[0018] Figure 2 Examples of the similarities and differences between traditional single-pixel and surface-based similarity comparison methods provided in embodiments of this application are shown in the following figures. Figure 2 Image (a) shows an example of the results using the traditional single-pixel similarity comparison method. Figure 2 (b) shows a classification case diagram. Figure 2 Figure (c) shows an example of the results using a face-based similarity comparison method.

[0019] Figure 3 Example diagrams illustrating the differences in the spectral dimension between traditional single-pixel and surface-based similarity comparison methods provided in embodiments of this application, wherein... Figure 3 Figure (a) shows an example diagram illustrating the positional relationship between similar pixels and their surrounding clustered pixels. Figure 3 (b) shows a close-up example of four similar pixels. Figure 3 (c) shows the spectral curves of four similar pixels.

[0020] Figure 4 Example diagrams illustrating the spatial scene features and relationships between multi-level surfaces provided in embodiments of this application, wherein, Figure 4 Figure (a) shows an example diagram illustrating the positional relationship between the target surface element and its multi-level adjacent surface elements. Figure 4 Figure (b) shows a statistical representation of the number of adjacent facets and the mean of the region for multi-order adjacent facets. Figure 4 (c) shows a histogram of the regional mean of multi-order adjacent surface elements.

[0021] Figure 5 An example diagram illustrating the calculation of shape features provided in this application embodiment, wherein, Figure 5 Figure (a) shows an example diagram of shape feature calculation for a long strip road surface element. Figure 5 Figure (b) shows an example of the location of elongated road surface elements and planar bare sand surface elements in a remote sensing image. Figure 5 (c) shows an example diagram of the shape feature calculation of a planar bare sand surface element.

[0022] Figure 6 An example diagram of a two-step reference pixel filtering process provided for embodiments of this application.

[0023] Figure 7 This is a design example diagram of the Transformer structure provided in an embodiment of this application, wherein, Figure 7 The residual connection is omitted.

[0024] Figure 8 The experimental results and average error diagrams for different algorithm structures in the US experimental region provided in this application embodiment are shown.

[0025] Figure 9 Example diagram of strip repair simulation experiment results provided for embodiments of this application in the Shanghai experimental area, wherein, Figure 9 The upper figure of (d) to (i) is the reconstruction result figure, and the lower figure is the error figure.

[0026] Figure 10 Example figures of thick cloud removal simulation experiment results provided for embodiments of this application in the experimental area of ​​Brazil, wherein, Figure 10 The upper figure (d) to (h) in the diagram is the reconstruction result, and the lower figure is the error diagram.

[0027] Figure 11 Example diagram of the simulation results of thick cloud removal in the experimental area of ​​Saudi Arabia provided for embodiments of this application, wherein, Figure 11 The upper figure (d) to (h) in the diagram is the reconstruction result, and the lower figure is the error diagram.

[0028] Figure 12Example graph showing the comparison results of spectral reconstruction degree of different reconstruction methods provided in the embodiments of this application.

[0029] Figure 13 Example image of thick cloud removal experimental results in a real-world scenario in the Ruoergai experimental area, provided for embodiments of this application.

[0030] Figure 14 Example diagrams showing parameter settings and experimental results for two screenings provided in this application embodiment, wherein, Figure 14 Figures (a) and (b) show the relationship between the number of similar pixels and the evaluation index during the two screening processes, respectively. Figure 14 (c) and (d) in the figure show the relationship between the number of similar pixels and the computation time during the two screenings.

[0031] Figure 15 A schematic diagram of the structure of the remote sensing image reconstruction device based on SAR imagery and similarity analysis provided in the embodiments of this application.

[0032] Figure 16 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0034] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0035] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0036] Example like Figure 1 As shown, the remote sensing image reconstruction method based on SAR imagery and similarity analysis provided in the first aspect of this embodiment can be executed, but is not limited to, by a computer device with certain computing resources, such as a server, a personal computer (PC, referring to a multi-purpose computer of a size, price, and performance suitable for personal use; desktop computers, laptops, mini-laptops, tablets, and ultrabooks are all personal computers), a smartphone, a personal digital assistant (PDA), or a wearable device. Figure 1 As shown, the remote sensing image reconstruction method includes, but is not limited to, the following steps S1 to S6.

[0037] S1. Acquire remote sensing images, reference remote sensing images, and SAR images of the target area that are of the same size, wherein the remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image that has no missing pixels at the position corresponding to the missing pixels.

[0038] In step S1, the remote sensing image to be repaired may have missing pixels due to thick cloud cover or other reasons. The acquisition time span between the reference remote sensing image and the remote sensing image to be repaired can be quite long, but the acquisition time span between the SAR image and the remote sensing image to be repaired needs to be as short as possible. The remote sensing image to be repaired generally contains multiple missing pixels in clusters. Since this embodiment reconstructs the missing parts pixel by pixel, the reference remote sensing image is also allowed to have missing pixels, as long as there are no missing pixels at the multiple locations that correspond one-to-one with the multiple missing pixels. In addition, the remote sensing image to be repaired, the reference remote sensing image, and the SAR image may be uploaded by users, but are not limited to these. The purpose of keeping them at the same size is to ensure that all pixels in any two images can correspond one-to-one, thereby ensuring the smooth operation of this embodiment.

[0039] S2. Retrieve multiple similar reference pixels from the reference remote sensing image that are similar to the target overlapping reference pixel, wherein the target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel.

[0040] In step S2, when the remote sensing image to be repaired contains multiple missing pixels in clusters, the reference remote sensing image contains multiple target overlapping reference pixels that correspond one-to-one with the multiple missing pixels. For each target overlapping reference pixel, multiple similar reference pixels similar to the corresponding pixel are retrieved from the reference remote sensing image. The set of similar pixels composed of these multiple similar reference pixels provides precise guidance for accurately restoring the missing pixels in the target image (i.e., the remote sensing image to be repaired). In this embodiment, it is preferable to use image clustering facets (also known as multi-scale segmentation objects of images, i.e., generating independent units of different sizes by aggregating pixels of similar land features) as the basic unit for similarity comparison; the rationale for this similarity comparison method stems from the fact that objects in nature usually exist at different spatial scales in images. Therefore, compared with traditional single-pixel similarity comparison methods, facet-based similarity comparison methods can achieve more accurate and semantically rich analysis. According to... Figure 2 The differences between traditional single-pixel and surface-based similarity comparison methods shown can be seen as follows: Figure 2 Of the six similar pixels shown in (a), only pixels a, b, and c in the figure have the same category as the target pixel. By comparing the similarity of clustered pixels, four similar pixels can be selected that are all in the same category as the target pixel. Figure 3 The differences between traditional single-pixel similarity comparison methods and surface-based methods are shown from a spectral perspective: Figure 3(a) shows that pixels O, A, B, and C have very similar spectral responses because these pixels have similar land cover materials. However, in reality, pixel O belongs to an exposed area, pixel A belongs to a man-made building, pixel B belongs to a road, and only pixel C represents the same land cover as pixel O. This illustrates that traditional single-pixel similarity comparison methods are more prone to making incorrect judgments because they only use the spectral response characteristics of a single pixel to determine whether they belong to the same land cover type; while area-based similarity comparison methods can overcome this deficiency.

[0041] In step S2, in order to make the set of similar pixels composed of the multiple similar reference pixels more in line with reality, this embodiment adopts a multi-feature similarity comparison method to retrieve similar pixels. That is, preferably, multiple similar reference pixels similar to the target overlapping reference pixels are retrieved from the reference remote sensing image, including but not limited to the following steps S21 to S26.

[0042] S21. Perform multi-scale segmentation processing on the reference remote sensing image to obtain multiple image clustering elements.

[0043] In step S21, the specific process of the multi-scale segmentation can be implemented conventionally using existing image multi-scale segmentation algorithms, but is not limited to this.

[0044] S22. For each image cluster element in the plurality of image cluster elements, obtain the corresponding multi-feature vector.

[0045] In step S22, the multiple feature vectors include, but are not limited to, spectral feature vectors, spatial scene feature vectors, and shape feature vectors. That is, this embodiment uses multiple aspects such as spectral features, spatial scene features, and shape features to determine whether a pixel cluster belongs to the same category.

[0046] In step S22, the spectral features are represented by a sequence of average values ​​of multiple bands for all pixels within a surface element. Specifically, for each image cluster surface element in the plurality of image cluster surface elements, a corresponding multi-feature vector is obtained. This includes: for a specific image cluster surface element in the plurality of image cluster surface elements, obtaining the corresponding spectral feature vector according to the following formula. :

[0047] In the formula, This indicates the total number of bands in the reference remote sensing image. Indicates less than or equal to positive integers, Indicated in the spectral eigenvector The first in The value of each feature term, This represents the total number of pixels in a given image cluster. Indicates less than or equal to positive integers, In a certain image cluster element, the first... The pixel in the first Values ​​for each band.

[0048] In step S22, considering the first law of geography, which states that everything distributed in geographic space is related to its neighbors, and the closer the distance, the closer the relationship; remote sensing imagery, as a carrier of geographic features, must follow this law because adjacent features inevitably interact; therefore, this embodiment proposes using spatial scene features to attempt to capture this pattern between pixels, in order to enhance the reliability of similarity comparison results. Similarly, this embodiment still uses clustered facets as the basic unit of analysis, and the remaining facets closely adjacent to the target facet are called first-order adjacent facets, with the facets expanding outwards sequentially called second-order, third-order, and so on. C Adjacent face elements of order, Figure 4 (a) illustrates the relationship between them. Because the image to be processed has multiple bands, this embodiment first uses principal component analysis and selects the first principal component to create a single-band image, denoted as [image name missing]. I pca Then, multi-scale segmentation of the surface elements is used. I pca Statistical analysis of the mean region is performed to obtain the feature value of each cluster element (this is done to reduce computation while maintaining maximum differentiation between land cover types). In this embodiment, the spatial scene features can be represented by the statistical values ​​of the features of the adjacent cluster elements. That is, for each image cluster element in the plurality of image cluster elements, the corresponding multi-feature vector is obtained. Specifically, for a certain image cluster element in the plurality of image cluster elements, the corresponding spatial scene feature vector is obtained according to the following formula. :

[0049] In the formula, This represents the total order obtained by expanding outwards from a certain image cluster element as the central element by a series of adjacent elements of multiple orders, wherein the multiple adjacent elements belong to the plurality of image cluster elements. Indicates less than or equal to positive integers, Represents the feature vector of the spatial scene The first in The value of each feature term, This represents the preset distance factor. This indicates that the number of elements expanding outwards from a given image cluster element is... The total number of adjacent face elements of order 1. Indicates less than or equal to positive integers, Indicated in the first The first adjacency element in the order The result value is obtained by performing mean region statistical analysis on the principal component image for each surface element. The principal component image refers to a single-band image created by selecting the first principal component of the surface elements using principal component analysis. Furthermore, the distance factor... For example, it can be equal to the ground sampling distance of the reference remote sensing image.

[0050] In step S22, considering the shape features of clustered surface elements in the image, these features are important supplementary information for analyzing ground feature attributes. Shape features describe the geometric attributes of the surface elements, typically including area, perimeter, compactness, aspect ratio, and many other representation methods. Since remote sensing images are composed of discrete pixels, this differs from vector-based planar graphics. This embodiment preferably adopts a new method for representing the shape of clustered surface elements. This method treats the clustered pixel clusters within the surface element as discrete points, and then uses statistical analysis to represent the shape characteristics of the surface elements composed of these scattered points. Specifically, for each image clustered surface element in the multiple image clustered surface elements, the corresponding multi-feature vector is obtained. This includes obtaining the corresponding shape feature vector for a certain image clustered surface element in the multiple image clustered surface elements according to the following steps S221 to S224.

[0051] S221. Establish a rectangular coordinate system with the center pixel of a certain image cluster as the origin.

[0052] In step S221, as Figure 5 As shown, a rectangular coordinate system can be established with the center pixel of the long road surface element on the left (i.e., the image cluster surface element) as the origin, or another rectangular coordinate system can be established with the center pixel of the bare sand surface element on the right (i.e., the image cluster surface element) as the origin.

[0053] S222. Make the center of a square frame with a fixed size coincide with the origin.

[0054] In step S222, the square frame is also known as Figure 5 The blue box in the middle.

[0055] S223. Based on the Cartesian coordinate system, vectorize each pixel of a certain image cluster that is located inside the square frame into a scatter point corresponding to each pixel.

[0056] In step S223, as Figure 5 As shown, this step transforms the problem of evaluating the shape characteristics of surface elements into the problem of evaluating the distribution shape of these scattered points.

[0057] S224. The explanatory variance ratio of the first principal component in principal component analysis is used to measure the distribution shape of the scatter set, and the measurement result is used as the shape feature vector of a certain image cluster element.

[0058] In step S224, the scatter set is the set consisting of all the scatter points. Furthermore, there are many mathematical methods available for evaluating the distribution shape of these scatter points; in addition to the explained variance ratio, other methods can be used for calculation.

[0059] S23. Select multiple neighboring pixels in the reference remote sensing image that are located within a preset neighborhood of the target overlapping reference pixel, wherein the target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel.

[0060] In step S23, the neighboring cell can be a cell in the reference remote sensing image that is located at a position corresponding to other missing cells in the remote sensing image to be repaired, or it can be other existing cells (i.e., it cannot be a missing cell in the reference remote sensing image).

[0061] S24. For each neighbor pixel among the plurality of neighbor pixels, based on the multi-feature vector of the first image cluster surface to which the corresponding pixel belongs and the multi-feature vector of the second image cluster surface to which the target overlapping reference pixel belongs, the similarity between the corresponding pixel and the target overlapping reference pixel is calculated according to the following formula. :

[0062] In the formula, This represents the total number of feature terms in the multiple feature vectors. Indicates less than or equal to positive integers, This represents the first feature vector in the multi-feature vector of the first image cluster element. The value of each feature term, This represents the first feature vector in the multi-feature vector of the second image cluster element. The values ​​of each feature term, and in From time to time .

[0063] In step S24, since the multiple feature vectors specifically include, but are not limited to, spectral feature vectors, spatial scene feature vectors, and shape feature vectors, the spectral feature vectors, spatial scene feature vectors, and shape feature vectors need to be flattened into a one-dimensional shape of the following form before calculation:

[0064] In the formula, The shape feature vector is actually a vector containing one element, where... .

[0065] S25. Arrange the multiple neighboring pixels in descending order of similarity to obtain a neighboring pixel queue.

[0066] S26. Select the previous one from the neighbor cell queue. One pixel serves as a plurality of similar reference pixels that overlap with the target reference pixel. Represents a preset positive integer.

[0067] In step S26, This can be analyzed in detail in the subsequent discussion. Generally, the value is taken in the range of 100 to 1000.

[0068] S3. For each missing overlapping reference pixel in the similar pixel set composed of the plurality of similar reference pixels, first determine the missing overlapping auxiliary pixel in the SAR image and the corresponding missing overlapping auxiliary pixel, then calculate the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, and finally, if the similarity does not meet the pixel retention condition, remove the corresponding pixel from the similar pixel set. Here, the missing overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to other missing pixels in the remote sensing image to be repaired, the missing overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the other missing pixel, and the target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel.

[0069] In step S3, considering that the similarity retrieval method in step S2 only retrieves similar reference pixels on the reference remote sensing image that are similar to the target overlapping reference pixels, it cannot detect the changes that occur between the corresponding pixel pairs of the two remote sensing images in two time phases; in other words, the final set of similar pixels should ideally only include pixel pairs where the ground features in the reference image and the target image have undergone the same changes, so that the final determined model is more targeted. Fortunately, SAR images can capture areas under thick clouds in the target image, so this embodiment uses SAR images as auxiliary data to further filter the initially screened set of similar pixels. The specific process of calculating the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel can be implemented with reference to the aforementioned steps S21-S22 and S24, that is, calculating the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, specifically including but not limited to the following steps S31-S33.

[0070] S31. Perform multi-scale segmentation processing on the SAR image to obtain multiple image segmentation elements.

[0071] S32. Obtain the multi-feature vector of the first image segmentation surface to which the missing overlapping auxiliary pixel belongs, and also obtain the multi-feature vector of the second image segmentation surface to which the target overlapping auxiliary pixel belongs, wherein the target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel.

[0072] S33. Based on the multi-feature vectors of the first image segmentation pixel and the second image segmentation pixel, the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel is calculated according to the following formula. :

[0073] In the formula, This represents the total number of feature terms in the multiple feature vectors. Indicates less than or equal to positive integers, In the multi-feature vector of the first image segmentation pixel, the first... The value of each feature term, In the multi-feature vector of the second image segmentation pixel, the first... The values ​​of each feature term, and in From time to time .

[0074] In step S3, the pixel retention condition can be that the similarity reaches a preset similarity threshold, or it can be based on the similarity first. All missing and overlapping reference pixels in the set of similar pixels are arranged in descending order of height, and then the top-ranked pixels are selected from the arrangement results. Among the pixels that satisfy the retention criteria, , It represents another preset positive integer (which will be analyzed in detail in the subsequent discussion, and generally takes a value in the range of 10 to 100). Figure 6 The two-step reference pixel filtering process based on the aforementioned steps S2 to S3 is demonstrated, so that the final set of similar pixels includes the following two parts of similar reference pixels: missing overlapping reference pixels and non-missing overlapping reference pixels (i.e., other reference pixels besides the target overlapping reference pixel and all the missing overlapping reference pixels). The former represents pixels with the same type of land cover as the missing pixel (because they have similar spectral responses in the band carried by the SAR image), while the latter represents pixels with the same type of land cover as the target overlapping reference pixel in the reference remote sensing image. Therefore, the set of similar pixels finally obtained through the aforementioned steps S2 to S3 can reflect the same land cover changes between two adjacent images.

[0075] In step S3, since the subsequent spatial and temporal remote sensing image reconstruction is based on all non-missing overlapping reference pixels in the similar pixel set, it is necessary to perform a secondary screening on all non-missing overlapping reference pixels in the similar pixel set. Preferably, for each non-missing overlapping reference pixel in the similar pixel set composed of the multiple similar reference pixels, the corresponding non-missing overlapping auxiliary pixel in the SAR image is first determined, and then the similarity between the non-missing overlapping auxiliary pixel and the target overlapping auxiliary pixel is calculated (the specific calculation process can be found in the aforementioned steps S31 to S33, which will not be repeated here). Finally, if the similarity does not meet the pixel retention condition, the corresponding pixel is removed from the similar pixel set. At this point, the following can be combined: For each missing overlapping reference pixel in the set of similar pixels composed of the multiple similar reference pixels, first determine the corresponding missing overlapping auxiliary pixel in the SAR image, then calculate the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, and finally, if the similarity does not meet the pixel retention condition, remove the corresponding pixel from the set of similar pixels. Here, the missing overlapping reference pixel refers to the pixel in the reference remote sensing image located at a position corresponding to other missing pixels in the remote sensing image to be repaired; the missing overlapping auxiliary pixel refers to the pixel in the SAR image located at a position corresponding to those other missing pixels; and the target overlapping auxiliary pixel is... The term "the pixel in the SAR image located at the position corresponding to the missing pixel" and the aforementioned technical means allow the entire step S3 to be replaced as follows: For each overlapping reference pixel in the similar pixel set composed of the plurality of similar reference pixels, first determine the corresponding overlapping auxiliary pixel in the SAR image, then calculate the similarity between the overlapping auxiliary pixel and the target overlapping auxiliary pixel (the specific calculation process can be found in the aforementioned steps S31 to S33, and will not be repeated here), and finally, if the similarity does not meet the pixel retention condition, remove the corresponding pixel from the similar pixel set. The target overlapping auxiliary pixel refers to the pixel in the SAR image located at the position corresponding to the missing pixel. Furthermore, the missing overlapping reference pixel in the similar pixel set can also be removed before or after step S3.

[0076] S4. Based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the set of similar pixels, the initial remote sensing image value of the missing pixel is reconstructed using an interpolation method, and the reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel, wherein the non-missing overlapping reference pixel refers to other pixels in the set of similar pixels besides all the missing overlapping reference pixels.

[0077] In step S4, considering the first law of geography which states that features closer together have a stronger connection, the relationship between adjacent pixels in the target image provides another reasonable source for restoring the missing target pixel. After the secondary screening in steps S2-S3, all the missing overlapping reference pixels and all the non-missing overlapping reference pixels in the similar pixel set represent the same feature as the missing pixel. Therefore, the distribution and pixel values ​​of all non-missing pixels corresponding one-to-one with all the non-missing overlapping reference pixels in the similar pixel set can be used to jointly predict the possible value of the missing pixel. Based on this idea, for a certain missing pixel, the spatial missing pixel restoration problem can be transformed into the problem of predicting points with unknown attributes based on many points with known attributes distributed in space, which is a classic geographical interpolation problem. Specifically, but not limited to, Kriging interpolation can be used to reconstruct missing pixels (i.e., the interpolation method used is Kriging interpolation). Kriging interpolation is a classic interpolation method based on geostatistical thinking, and its principle is well known and will not be elaborated here. Considering that reasonably measuring the distance between nodes is one of the core contents of the Kriging interpolation method, in order to make Kriging interpolation more reasonable for predicting the value of unknown pixels, this embodiment specifically uses the remote sensing image value of the pixel as the third dimension of the pixel coordinate (i.e., the dimension represented by the Z-axis coordinate). Then the coordinates of the node can be represented as (x, y, z), where z is the remote sensing image value. In this way, the node in the initial two-dimensional space is placed in three-dimensional space (the rationale for this is that the land features that act as known nodes and are represented by the remote sensing image values ​​of the various non-missing pixels are highly similar to the unknown nodes themselves, and the degree of similarity is linearly related to the remote sensing image values).

[0078] S5. Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, a remote sensing image value prediction model for the missing pixel is obtained by calibration and verification modeling based on a time series model. The remote sensing image value of the target overlapping reference pixel is then imported into the remote sensing image value prediction model, and the time-based reconstructed pixel value of the missing pixel is output.

[0079] In step S5, the time-based thick cloud removal method is considered as a process of recovering unknown missing pixels from corresponding pixels in adjacent temporal reference images. Therefore, the set of similar pixels finally obtained through the aforementioned steps S2 to S3 is the data source for missing pixel reconstruction using the time-based thick cloud removal method. This data source can reflect the changes that occur between pixels in adjacent temporal phases. The missing pixel value is denoted as... The time-based thick cloud removal method can be expressed as:

[0080] In the formula, This indicates the target overlapping reference pixel. This is a model for time-based thick cloud removal. In short, it's a model that uses remote sensing data of all non-missing overlapping reference pixels in the set of similar pixels as labeled samples to determine the model. The problem is that, specifically, based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, a prediction model for the remote sensing image values ​​of the missing pixel is obtained by calibration and verification modeling based on a time-series model, including but not limited to the following steps S51 to S52.

[0081] S51. Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, sample data is obtained in the following manner: the remote sensing image value of a certain non-missing overlapping reference pixel in the set of similar pixels is used as the model input item, and the remote sensing image value of a certain non-missing pixel in the remote sensing image to be repaired and corresponding to the certain non-missing overlapping reference pixel is used as the model output item, and then the model input item and the model output item are used as a sample data.

[0082] In step S52, considering the task perspective, the model The process is straightforward: it involves deriving values ​​for another B bands from the values ​​of the first B bands, and all the non-missing overlapping reference pixels constituting the set of similar pixels possess a high degree of similarity. In other words, band information can be viewed as sequential data, even if its horizontal axis is not time.

[0083] S52. Apply all the sample data to calibrate and validate the time series model using the Transformer structure to obtain the remote sensing image value prediction model for the missing pixels.

[0084] In step S52, considering the learning task of deriving the output sequence from the input sequence, the most popular and reliable method is the Transformer structure. Therefore, in this embodiment, the Transformer structure is specifically used to determine the model. Furthermore, considering the complexity of the task, the embedding dimension in the model is set to 9, and a two-layer nested multi-head attention mechanism is employed, with the number of heads in the multi-head attention mechanism set to 3. Figure 7 As shown.

[0085] In step S5, considering that the pixel-based reconstruction algorithm theory requires training a remote sensing image value prediction model for each unknown pixel (i.e., each missing pixel), in order to speed up the algorithm's operation, this embodiment preferably uses the following training strategy of pre-training plus fine-tuning: During pre-training, the remote sensing image values ​​of all non-missing overlapping reference pixels in all similar pixel sets corresponding one-to-one with all missing pixels, and the remote sensing image values ​​of all non-missing pixels in the remote sensing image to be repaired that correspond one-to-one with all non-missing overlapping reference pixels, are used to obtain the initial remote sensing image based on time-series model calibration and verification modeling. Remote sensing image value prediction model; after the remote sensing image value prediction model is finalized, the remote sensing image values ​​of all non-missing overlapping reference pixels in the set of similar pixels corresponding to each missing pixel, and the remote sensing image values ​​of all non-missing pixels in the remote sensing image to be repaired that correspond one-to-one with all the non-missing overlapping reference pixels are used to train the remote sensing image value prediction model for the current missing pixel; during the model training process, the verification accuracy can be used as the model training termination condition, that is, when the verification accuracy is greater than or equal to the set threshold, the current model is automatically saved and the prediction step is executed. After completion, the reconstruction calculation of the next missing pixel continues.

[0086] S6. Based on the missing pixels and the reconstructed pixel values ​​based on space and time respectively, the remote sensing image values ​​of the missing pixels are fused to obtain the remote sensing image values ​​of the missing pixels.

[0087] In step S6, the weighted average parallel connection method, commonly used in existing spatiotemporal fusion methods, can be used to determine the final missing pixel value. Specifically, based on the reconstructed pixel values ​​of the missing pixel, both spatially and temporally, the remote sensing image value of the missing pixel is fused. This includes, but is not limited to, calculating the remote sensing image value of the missing pixel according to the following formula based on the reconstructed pixel values ​​of the missing pixel, both spatially and temporally. :

[0088] In the formula, This represents the missing pixel and the spatially reconstructed pixel value. This represents the time-based reconstructed pixel value of the missing pixel. and These represent the weighting coefficients, and have Considering the appropriate weight settings is crucial for obtaining accurate missing pixel values. However, in previous weighting methods, the balance between the two weights was usually adjusted manually using empirical methods, which can easily lead to human-induced uncertainties. This results in limitations in accurately recovering ground features. To avoid introducing such uncertainties, in this embodiment, these two weight parameters can be obtained automatically as follows: during the prediction of a missing pixel, and The calculation is based on the results of a secondary screening (SAR similarity), specifically, This is equal to the mean similarity between all missing overlapping auxiliary pixels and the target overlapping auxiliary pixel, wherein all missing overlapping auxiliary pixels correspond one-to-one with all the missing overlapping reference pixels in the set of similar pixels. One explanation for this is that the higher the SAR similarity result on the target image, the more reliable the image reconstruction result using spatial methods will be; conversely, more weight will be allocated to predicting missing pixels using temporal methods. Furthermore, after fusing the remote sensing image values ​​of all the missing pixels, the restoration result of the remote sensing image to be restored can be obtained, i.e., a newly generated remote sensing image of the target area.

[0089] To verify the technical effectiveness of the remote sensing image reconstruction method based on the above steps S1 to S6, this embodiment also provides the following experimental steps (A) to (G).

[0090] (A) Experimental data and preprocessing To verify the algorithm's generalization ability, this embodiment selected five experimental areas worldwide, with land cover types including urban areas, farmland, wetlands, rainforests, and bare sandy areas. These five areas included satellite imagery from three sensor types: Landsat 9 OLI-2, Landsat 8 OLI, and Sentinel-2 multispectral sensors. Landsat 9 and Landsat 8 surface reflectance data were extracted from seven optical bands, while Sentinel-2 data was extracted from ten optical bands. Regardless of the experimental area, all SAR images used as auxiliary data utilized Sentinel-1 C-band synthetic aperture radar data. When used for Landsat image restoration, these images were resampled to a resolution matching the Landsat imagery. Imagery from the same season of the previous year was used as reference images. These images were generated by median composite of multiple images within a certain time period. The platform used was Google Earth Engine, and the data sources were the U.S. Geological Survey and the European Space Agency. Table 1 shows the locations of the experimental areas and their detailed information. In all experimental areas, the time interval between the reference image and the target image was one year. This is because, in real-world scenarios, it is not easy to obtain reference images that are close in time. In fact, within a one-year interval, the terrain features in the five selected areas of the experiment underwent significant changes. Capturing these changes is the key problem that the algorithm must solve.

[0091] Table 1. Experimental area and its information

[0092] As mentioned earlier, the multi-feature similarity retrieval method is based on clustering facets. In the experiment, these facets can be obtained by multi-scale segmentation of images using eCognition software. Furthermore, in the time-based reconstruction, this embodiment divides all sample data into training and testing sets in an 8:2 ratio (a validation set is not included here to balance reconstruction effectiveness and efficiency). The machine learning model used in this embodiment has a very simple structure, requiring minimal hyperparameter adjustments.

[0093] (B) Parameter settings and evaluation indicators (B1) Parameter settings: In the multi-feature similarity retrieval stage of the method proposed in this embodiment, the order of spatial scene feature calculation is... In the spatial missing region reconstruction stage, the semivariance function of Kriging interpolation was fitted linearly. In the temporal missing region reconstruction stage, the dimension of the linear embedding layer was set to 9, the number of nested multi-head attention layers was set to 2, the number of multi-head self-attention heads was 3, the network optimizer was the commonly used Adaptive Moment Estimation (Adam), the initial learning rate was set to 0.00005, the batch size during training was set to 64, the machine learning module was implemented using the PyTorch framework, and the experiment used an NVIDIA GeForce RTX 3080 GPU with a processor cache of 10GB.

[0094] (B2) Evaluation metrics: Four widely used image reconstruction evaluation metrics were selected to verify the performance of various reconstruction methods for reconstructing missing image regions: mean peak signal-to-noise ratio (MPSNR), mean structural similarity (MSSIM), spectral angle similarity (SAM), and cross correlation (CC). MPSNR and MSSIM calculate the similarity and structural consistency between the real image and the generated image, respectively. They are used to measure spatial visual quality by averaging across all spectral bands. A higher MPSNR value means better visual quality, and the best value for MSSIM is 1. SAM and CC reflect the spectral fidelity of the image. The best value for SAM is 0, and the best value for CC is 1.

[0095] (C) Ablation test To verify the effectiveness of different reconstruction methods, this embodiment conducted ablation experiments using data from the US experimental area. Specifically, the reconstruction model in the image reconstruction method (MSSM) provided in this embodiment was decomposed into space-base-MSSM (corresponding to the reconstruction model in step S4) reconstructed using only spatial methods, time-base-MSSM (corresponding to the reconstruction model in step S5) reconstructed using only temporal methods, and integration-MSSM (corresponding to the reconstruction model in step S6) reconstructed using spatiotemporal fusion methods. All of them used the same data source, and all underwent multi-feature similarity pixel retrieval and secondary retrieval based on target SAR data. In addition, the effectiveness of area-based similarity comparison was verified. Specifically, experiments were conducted using pixel-based rather than area-based similarity comparison, and the prefixes "Pixel" and "Area" were added to the names of the corresponding methods for distinction. Table 2 shows the ablation experiment results of different reconstruction methods. It can be seen that among the three reconstruction methods, the reconstruction effect of the temporal method, spatial method, and integration method improves in that order. In addition, various indicators show that the surface-based method is generally better than the pixel-based method, and the surface-based integrated spatiotemporal model reconstruction method has achieved the best experimental results.

[0096] Table 2. Quantitative evaluation results of different algorithm structures in the US experimental area

[0097] Figure 9 The results of various reconstruction methods in the ablation experiment are shown in the experimental area in the United States. The simulated cloud-covered areas are circled in yellow. The green boxes in Figure (a) indicate the locations of the average error maps in sub-figures (d)-(i). The reference image and the target image in the experimental area of ​​the United States are separated by one year. As can be seen from the figure, the difference between them is very large, which is quite common in reality because obtaining reference images is always difficult. Consistent with the indicators in Table 3, the reconstruction method based on the ensemble spatiotemporal model of the surface element is closest to the result of sub-figure (c), and the same is reflected in their average error maps.

[0098] (D) Comparative Experiment To verify the effectiveness of the reconstruction method, this embodiment selected three experimental areas—Shanghai, Brazil, and Saudi Arabia—for comparative experiments. Various reconstruction methods were used to reconstruct simulated gaps, allowing the reconstruction results to be used in conjunction with the original images for index calculation. As is well known, due to sensor damage, the quality of some Landsat 7 ETM+ data is severely affected by banding; therefore, image reconstruction of missing band areas is one of the algorithm's objectives. To address this issue, this embodiment simulated missing bands in Landsat 9OLI-2 data from the Shanghai experimental area and used various reconstruction methods to repair them, verifying the methods' capabilities in this regard. Regarding reconstruction methods, this embodiment selects four excellent reconstruction methods proposed in recent years for comparative experiments, including the SAR structure data-based reconstruction method (SARSD), the convolutional-deconvolutional network-based reconstruction method (CMD), the weighted linear regression algorithm-based reconstruction method (WLR), and the virtual image-based cloud removal method (VICR). These reconstruction methods adopt different concepts for image inpainting. Among them, SARSD only uses SAR images as reference data without using additional reference images for inpainting; CMD inputs SAR images and reference images into the network together for learning, training, and inpainting. Both of these reconstruction methods directly input SAR data into the deep network for reconstruction; WLR and VICR are two methods based on pixel similarity comparison. WLR reconstructs images using linear regression of similar pixels, while VICR uses a design that integrates multiple reference images into a concept graph for image reconstruction. Neither of them uses SAR images.

[0099] Table 3. Evaluation of comparative experimental results in the three experimental areas

[0100] (D1) Strip repair simulation experiment in Shanghai experimental zone Because the various indicators of the comparative experiment require the calculation of both real and reconstructed images, the experiment was not conducted directly on Landsat 7 images with defects. Instead, Landsat 9 data was used directly in the Shanghai experimental area to simulate the stripes of Landsat 7 through a masking method. Figure 9 As shown in (c), this is reasonable. Additionally, the spatially based Gapfill algorithm was added to this comparative experiment because it is a well-known plugin in the remote sensing image processing software ENVI, making it a highly representative comparison.

[0101] As can be seen from the evaluation indicators of the various experimental results in the Shanghai area in Table 3, MSSM achieved the best results, while SARSD performed poorly. This may be because the ground feature information contained in the SAR images of this area is too similar. Figure 10 The reconstruction results of these reconstruction methods are shown. The reconstruction result of Gapfill appears to be too distorted, the result of SARSD has obvious bands, and the reconstruction results of the other reconstruction methods are more ideal. As can be seen from the error plot in the third row, the error of MSSM is very small among these methods.

[0102] (D2) Simulation experiment on thick cloud removal in the Brazilian experimental area The land cover type in the Brazilian experimental area is mainly tropical rainforest, interspersed with some artificial structures. This area primarily validates the effectiveness of the algorithm in forest-covered regions. As shown in Table 3, MSSM achieved the best results on most metrics. Figure 10 The image reconstruction results of various methods are shown. As can be seen from the figures, the MSSM reconstruction is more natural, closer to the real image. However, in terms of ground feature changes, SARSD is more sensitive to capturing the differences between the target image and the reference image, which is clearly reflected in the simulated cloud patch in the upper right corner of the image. Although SARSD performs well in change capture, its overall reconstruction effect is not satisfactory, as shown in the error map. In summary, the MSSM algorithm performs well in rainforest reconstruction, but its performance needs improvement for drastically changing man-made structures.

[0103] (D3) Simulation experiment on thick cloud removal in the experimental area of ​​Saudi Arabia The land cover in the Saudi Arabian region is primarily bare sand and artificial structures. These artificial structures underwent dramatic changes during the two years selected for the experiment, allowing for a good comparison of the advantages and disadvantages of various reconstruction methods. Figure 11Comparing the reconstruction results of various methods, except for SARSD, the reconstruction results of the other methods are relatively smooth. Among them, the reconstruction result of the CMD method is closer to the reference image, which may be because the weight of the SAR image in this method is relatively small, and it does not capture the changes in the surface that occurred during the year very well. Salt-and-pepper noise appears to varying degrees in the reconstructed images of SARSD, WLR, and VICR. As can be seen from the error map, SARSD has the largest error, while MSSM has the smallest error but is very close to CMD, which is consistent with the conclusions of the evaluation indicators in Table 3.

[0104] (D4) Spectral refraction of different reconstruction methods As a remote sensing image reconstruction method, good reconstruction results in the spatial domain alone are insufficient; good fidelity in the spectral domain is also essential. The SAM index in Table 3 quantitatively represents the spectral fidelity of different reconstruction methods; specifically, in the three experimental regions, MSSM achieved the optimal SAM value (the smaller the value, the better). In this embodiment, the spectral curves of each comparison method were obtained by randomly selecting a pixel location in the reconstructed images of the three regions. The differences between these curves and the absolute values ​​of the differences at that pixel location were calculated to obtain the... Figure 12 This figure reflects the difference between the reconstructed and true values ​​for each band. Consistent with the data in Table 3, MSSM exhibits the best high-spectral fidelity in the comparative experiments across the three implementation regions. In the Shanghai experimental region, Gapfill and SARSD performed the worst in strip restoration simulation. In the Brazil and Saudi Arabia experimental regions, CMD performed better, while SARSD showed relatively poor spectral fidelity.

[0105] (E) Experiment on thick cloud removal in real-world scenarios To verify the practical effect of the MSSM method in real-world scenarios, this embodiment also selected four square areas within the Ruoergai Wetland National Nature Reserve to conduct a real-world thick cloud removal experiment. Each area was approximately 1300×1300 pixels in size. The land cover in the missing areas of areas 1-4 mainly included meadows, crops, towns, rivers, and valleys. Figure 13 The target image shown in (a) is a composite image of quality bands from the summer of 2023 (June to August). The composite image is still covered by thick clouds and can be de-clouded using the pixel quality attribute of the Landsat series image, which is generated by the CFMASK algorithm (and any missing thick cloud edges can be manually removed). Figure 13 The multispectral image shown in (b) and used as a reference is... Figure 13 The SAR image shown in (c) is data from the same season in 2024. Figure 13Figure (d) shows the results of image restoration using MSSM. As can be seen from the figure, MSSM performs admirably even when there are significant differences in land features between the reference and target images. For example, within the yellow boxes of Area 1 and Area 2, the farmland in the reference image was brownish-red at the time, while the restored image shows the crops in vibrant green. The same is true for the yellow box area of ​​Area 3, where the riverbanks are restored to their proper colors. Overall, MSSM is effective in real-world scenarios.

[0106] (F) Discuss the relationship between parameter settings and experimental results in the reconstruction method proposed in this embodiment. In MSSM, two key parameters can affect the image reconstruction effect: the threshold number of similar pixels in each feature screening. To quantify the impact of these two parameters on the image reconstruction effect, this embodiment conducted an experiment in a 400×400 area in Brazil. Considering the acceptable computational time cost of the method, the pixel count range for the first screening was set between 100 and 1000, in increments of 100, and the pixel count range for the second screening was set between 10 and 100, in increments of 10. For evaluation metrics, MPSNR and SAM, which reflect both spatial and spectral reconstruction effects, were selected. Figure 14 The results of the experiment are presented: From Figure 14 As shown in (a), with the increase of the number of similar pixels in the first screening, the performance of both MPSNR and SAM indicators first improves and then deteriorates, with the inflection point appearing around 500 pixels; the number of pixels in the second screening also shows a similar trend, from... Figure 14 As can be seen from (b) in the figure, the value is optimal when it is around 50. In fact, it is intuitive that the more similar pixels there are, the better the effect. However, as the number increases, the quality of similar pixels may decrease, that is, they will become less and less similar to the target pixels, which may be the reason for the decrease in reconstruction quality. Figure 15 (c) and Figure 15 (d) in the figure reflects the change in time spent as the threshold parameter changes during the two screening processes. Obviously, the more similar pixels there are, the longer it takes.

[0107] (G) Discuss the time cost of the reconstruction method proposed in this embodiment. The time cost of an algorithm's execution is a crucial indicator of its practicality. This embodiment also compares the computation time of several reconstruction methods in the aforementioned comparative experiment. To maintain a consistent comparison scale, the same hardware was used: an Intel(R) Xeon(R) W-2255 CPU with a frequency of 3.70GHz, an NVIDIA GeForce RTX 3080 GPU, and 64GB of memory. Except for the Gapfill method, which was run directly using ENVI software, all other reconstruction methods were implemented using Python. The experimental area was Shanghai, and the reconstructed image shape was 1000×1000×7. Table 4 shows the runtime costs of various reconstruction methods.

[0108] Table 4. Comparison of time costs of various reconstruction algorithms in the Shanghai experimental zone

[0109] As can be seen from the table, Gapfill exhibits a short computation time for strip inpainting tasks because it only uses interpolation methods for image inpainting. CMD, due to the large amount of input data (SAR image and reference image) for model training and reconstruction, has a longer computation time. SARSD and WLR have relatively short running times because they involve less data in their calculations. Similarly, VICR, which uses pixel similarity comparison, also takes a relatively long time due to its complex algorithm calculation process. MSSM's running time is relatively short, but not the shortest. Considering the reconstruction results of the aforementioned comparative experiments, this time is acceptable, and MSSM can be effective in real-world scenarios.

[0110] Therefore, based on the remote sensing image reconstruction method described in steps S1 to S6 above, a new scheme for spatiotemporal comprehensive reconstruction of remote sensing images to be repaired based on SAR data and similarity analysis results is provided. Specifically, after acquiring remote sensing images to be repaired, reference remote sensing images, and SAR images of the same size for the target area, multiple similar reference pixels with overlapping reference pixels are retrieved from the reference remote sensing image. Then, the set of similar pixels is further filtered based on the similarity between overlapping auxiliary pixels in the SAR image and overlapping auxiliary pixels in the target. Next, based on the set of similar pixels, spatial and temporal pixel value reconstruction is performed on the missing pixels. Finally, the spatial and temporal reconstruction results are fused to obtain the remote sensing image value of the missing pixels. Ablation experiments and comparative experiments have demonstrated that this method can effectively improve robustness, effectiveness, and the detail and visual consistency of the reconstructed images even in areas where the surface changes frequently. Therefore, it has advantages over similar reconstruction methods. Furthermore, by combining experimental results in real-world scenarios and analyzing the operating time cost, it has been proven to have the potential for application in large-scale real-world production scenarios, facilitating practical application and promotion.

[0111] like Figure 15 As shown, the second aspect of this embodiment provides a virtual device for implementing the remote sensing image reconstruction method described in the first aspect, including an image data acquisition unit, a similar pixel initial selection unit, a similar pixel re-selection unit, a spatial dimension reconstruction unit, a temporal dimension reconstruction unit, and a reconstruction result fusion unit. The image data acquisition unit is used to acquire remote sensing images to be repaired, reference remote sensing images, and SAR images of the target area with the same size. The remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image without missing pixels at the position corresponding to the missing pixels. The similar pixel initial selection unit is communicatively connected to the image data acquisition unit and is used to retrieve multiple similar reference pixels that are similar to the target overlapping reference pixel from the reference remote sensing image. The target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel. The similar pixel reselection unit is communicatively connected to the image data acquisition unit and the similar pixel initial selection unit, respectively. It is used to, for each missing overlapping reference pixel in the similar pixel set composed of the plurality of similar reference pixels, first determine the corresponding missing overlapping auxiliary pixel in the SAR image, then calculate the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, and finally, if the similarity does not meet the pixel retention condition, remove the corresponding pixel from the similar pixel set. Here, the missing overlapping reference pixel refers to the pixel in the reference remote sensing image located at a position corresponding to other missing pixels in the remote sensing image to be repaired; the missing overlapping auxiliary pixel refers to the pixel in the SAR image located at a position corresponding to the other missing pixel; and the target overlapping auxiliary pixel refers to the pixel in the SAR image located at a position corresponding to the missing pixel. The spatial dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to reconstruct the initial remote sensing image value of the missing pixel using an interpolation method based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the similar pixel set. The reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel. The non-missing overlapping reference pixel refers to other pixels in the similar pixel set other than all the missing overlapping reference pixels. The time-dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to obtain a remote sensing image value prediction model for the missing pixel based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, and to import the remote sensing image value of the target overlapping reference pixel into the remote sensing image value prediction model, and output the time-based reconstructed pixel value of the missing pixel. The reconstruction result fusion unit is communicatively connected to the spatial dimension reconstruction unit and the temporal dimension reconstruction unit, respectively, and is used to fuse the remote sensing image value of the missing pixel based on the reconstructed pixel value based on space and time respectively.

[0112] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the remote sensing image reconstruction method described in the first aspect, and will not be repeated here.

[0113] like Figure 16As shown, the third aspect of this embodiment provides a computer device for executing the remote sensing image reconstruction method as described in the first aspect, including a storage module, a processing module, and a transceiver module connected in sequence. The storage module stores a computer program, the transceiver module sends and receives messages, and the processing module reads the computer program and executes the remote sensing image reconstruction method as described in the first aspect. Specifically, the storage module may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processing module may, but is not limited to, use a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power supply module, a display screen, and other necessary components.

[0114] The working process, working details and technical effects of the aforementioned computer equipment provided in the third aspect of this embodiment can be found in the remote sensing image reconstruction method described in the first aspect, and will not be repeated here.

[0115] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions comprising the remote sensing image reconstruction method as described in the first aspect. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the remote sensing image reconstruction method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0116] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be found in the remote sensing image reconstruction method as described in the first aspect, and will not be repeated here.

[0117] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the remote sensing image reconstruction method as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0118] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote sensing image reconstruction method based on SAR imagery and similarity analysis, characterized in that, include: Acquire remote sensing images, reference remote sensing images, and SAR images of the target area that are of the same size. The remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image that has no missing pixels at the position corresponding to the missing pixels. Based on the multi-feature similarity of surface pixels, multiple similar reference pixels that are similar to the target overlapping reference pixel are retrieved from the reference remote sensing image to form a preliminary set of similar pixels. The target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel. The SAR image is used to perform a secondary screening on the initially selected set of similar pixels to obtain a second set of similar pixels. The secondary screening specifically includes: for each missing overlapping reference pixel in the initially selected set of similar pixels, firstly, determining the corresponding missing overlapping auxiliary pixel in the SAR image; then, calculating the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel; finally, if the similarity does not meet the pixel retention condition, removing the corresponding pixel from the initially selected set of similar pixels. The missing overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to other missing pixels in the remote sensing image to be repaired; the missing overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the other missing pixel; and the target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel. Based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the set of similar pixels after secondary screening, the initial remote sensing image value of the missing pixel is reconstructed using an interpolation method, and the reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel. The non-missing overlapping reference pixel refers to other pixels in the set of similar pixels after secondary screening, excluding all the missing overlapping reference pixels. Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, a remote sensing image value prediction model for the missing pixel is obtained by calibration and verification modeling based on a time series model. The remote sensing image value of the target overlapping reference pixel is then imported into the remote sensing image value prediction model, and the reconstructed pixel value of the missing pixel based on time is output. The remote sensing image value of the missing pixel is obtained by fusing the reconstructed pixel values ​​based on the missing pixel and the spatial and temporal values ​​respectively.

2. The remote sensing image reconstruction method according to claim 1, characterized in that, Multiple similar reference pixels that are similar to the overlapping reference pixels of the target are retrieved from the reference remote sensing image, including: The reference remote sensing image is segmented at multiple scales to obtain multiple image clustering elements; For each image cluster element in the plurality of image cluster elements, obtain the corresponding multi-feature vector; In the reference remote sensing image, select multiple neighboring pixels located within a preset neighborhood range of the target overlapping reference pixel, wherein the target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel. For each neighboring pixel among the plurality of neighboring pixels, the similarity between the corresponding pixel and the target overlapping reference pixel is calculated according to the following formula, based on the multi-feature vector of the first image cluster element to which the corresponding pixel belongs and the multi-feature vector of the second image cluster element to which the target overlapping reference pixel belongs. : In the formula, This represents the total number of feature terms in the multiple feature vectors. Indicates less than or equal to positive integers, This represents the first feature vector in the multi-feature vector of the first image cluster element. The value of each feature term, This represents the first feature vector in the multi-feature vector of the second image cluster element. The values ​​of each feature term, and in From time to time ; The neighboring pixels are arranged in descending order of similarity to obtain a neighboring pixel queue. Select the first from the neighbor cell queue One pixel serves as a plurality of similar reference pixels that overlap with the target reference pixel. Represents a preset positive integer.

3. The remote sensing image reconstruction method according to claim 2, characterized in that, The multiple feature vectors include spectral feature vectors, spatial scene feature vectors, and shape feature vectors. For each image cluster element in the multiple image clustering elements, a corresponding multiple feature vector is obtained, including: For a specific image cluster element among the plurality of image cluster elements, the corresponding spectral feature vector is obtained according to the following formula. : In the formula, This indicates the total number of bands in the reference remote sensing image. Indicates less than or equal to positive integers, Indicated in the spectral eigenvector The first in The value of each feature term, This represents the total number of pixels in a given image cluster. Indicates less than or equal to positive integers, In a certain image cluster element, the first... The pixel in the first Values ​​for each band; And / or, for a specific image cluster element among the plurality of image cluster elements, obtain the corresponding spatial scene feature vector according to the following formula. : In the formula, This represents the total order obtained by expanding outwards from a certain image cluster element as the central element by a series of adjacent elements of multiple orders, wherein the multiple adjacent elements belong to the plurality of image cluster elements. Indicates less than or equal to positive integers, Represents the feature vector of the spatial scene The first in The value of each feature term, This represents the preset distance factor. This indicates the number of times the cluster expands outward from a given image cluster element. The total number of adjacent face elements of order 1. Indicates less than or equal to positive integers, Indicated in the first The first adjacency element in the order The result value is obtained by performing mean region statistical analysis on the principal component image of each element. The principal component image refers to a single-band image made by selecting the first principal component of the element using the principal component analysis method. And / or, for a specific image cluster element among the plurality of image cluster elements, obtain the corresponding shape feature vector according to the following steps: Establish a rectangular coordinate system with the center pixel of a certain image cluster as the origin; Make the center of a square frame of a fixed size coincide with the origin; Based on the Cartesian coordinate system, each pixel in a certain image cluster that is located inside the square frame is vectorized into a scatter point that corresponds one-to-one with each pixel; The distribution shape of the scatter set is measured by the ratio of the explained variance of the first principal component in principal component analysis, and the measurement result is used as the shape feature vector of a certain image cluster element.

4. The remote sensing image reconstruction method according to claim 1, characterized in that, Calculate the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, including: The SAR image is subjected to multi-scale segmentation processing to obtain multiple image segmentation elements; The multi-feature vector of the first image segmentation surface to which the missing overlapping auxiliary pixel belongs is obtained, and the multi-feature vector of the second image segmentation surface to which the target overlapping auxiliary pixel belongs is also obtained, wherein the target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel. Based on the multi-feature vectors of the first image segmentation pixel and the second image segmentation pixel, the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel is calculated according to the following formula. : In the formula, This represents the total number of feature terms in the multiple feature vectors. Indicates less than or equal to positive integers, In the multi-feature vector of the first image segmentation pixel, the first... The value of each feature term, In the multi-feature vector of the second image segmentation pixel, the first... The values ​​of each feature term, and in From time to time .

5. The remote sensing image reconstruction method according to claim 1, characterized in that, Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, a prediction model for the remote sensing image values ​​of the missing pixel is obtained through time-series model calibration and verification modeling, including: Based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, sample data is obtained in the following manner: the remote sensing image value of a non-missing overlapping reference pixel in the set of similar pixels is used as the model input, and the remote sensing image value of a non-missing pixel in the remote sensing image to be repaired and corresponding to the non-missing overlapping reference pixel is used as the model output. Then, the model input and the model output are used as a sample data. Using all the sample data, the time series model with the Transformer structure is calibrated and validated to obtain the remote sensing image value prediction model for the missing pixels.

6. The remote sensing image reconstruction method according to claim 1, characterized in that, Based on the missing pixels and the reconstructed pixel values ​​based on space and time respectively, the remote sensing image values ​​of the missing pixels are fused to obtain the following: Based on the missing pixels and the reconstructed pixel values ​​based on space and time respectively, the remote sensing image value of the missing pixels is calculated according to the following formula. : In the formula, This represents the missing pixel and the spatially reconstructed pixel value. This represents the time-based reconstructed pixel value of the missing pixel. and These represent the weighting coefficients, and have as well as It is equal to the mean of the similarity between all missing overlapping auxiliary pixels and the target overlapping auxiliary pixel, wherein all missing overlapping auxiliary pixels correspond one-to-one with all the missing overlapping reference pixels in the set of similar pixels.

7. A remote sensing image reconstruction device based on SAR imagery and similarity analysis, characterized in that, It includes an image data acquisition unit, a similar pixel initial selection unit, a similar pixel re-selection unit, a spatial dimension reconstruction unit, a temporal dimension reconstruction unit, and a reconstruction result fusion unit; The image data acquisition unit is used to acquire remote sensing images to be repaired, reference remote sensing images, and SAR images of the target area with the same size. The remote sensing image to be repaired refers to a remote sensing image with missing pixels, and the reference remote sensing image refers to a remote sensing image without missing pixels at the position corresponding to the missing pixels. The similar pixel initial selection unit is communicatively connected to the image data acquisition unit. It is used to retrieve multiple similar reference pixels that are similar to the target overlapping reference pixel from the reference remote sensing image based on the multi-feature similarity of the surface pixels to form an initial selection set of similar pixels. The target overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to the missing pixel. The similar pixel reselection unit is communicatively connected to the image data acquisition unit and the similar pixel initial selection unit, respectively. It is used to perform a secondary screening on the initial similar pixel set using the SAR image to obtain a secondary screening similar pixel set. The secondary screening specifically includes: for each missing overlapping reference pixel in the initial similar pixel set, firstly, determining the corresponding missing overlapping auxiliary pixel in the SAR image, then calculating the similarity between the missing overlapping auxiliary pixel and the target overlapping auxiliary pixel, and finally, if the similarity does not meet the pixel retention condition, removing the corresponding pixel from the initial similar pixel set. The missing overlapping reference pixel refers to the pixel in the reference remote sensing image that is located at the position corresponding to other missing pixels in the remote sensing image to be repaired. The missing overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the other missing pixel. The target overlapping auxiliary pixel refers to the pixel in the SAR image that is located at the position corresponding to the missing pixel. The spatial dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to reconstruct the initial remote sensing image value of the missing pixel using an interpolation method based on the remote sensing image values ​​of each non-missing pixel in the remote sensing image to be repaired and corresponding one-to-one with each non-missing overlapping reference pixel in the similar pixel set after secondary screening. The reconstructed image value is used as the spatially reconstructed pixel value of the missing pixel. The non-missing overlapping reference pixel refers to other pixels in the similar pixel set after secondary screening other than all the missing overlapping reference pixels. The time-dimension reconstruction unit is communicatively connected to the image data acquisition unit and the similar pixel reselection unit, respectively. It is used to obtain a remote sensing image value prediction model for the missing pixel based on the remote sensing image values ​​of each non-missing overlapping reference pixel and the remote sensing image values ​​of each non-missing pixel, and to import the remote sensing image value of the target overlapping reference pixel into the remote sensing image value prediction model, and output the time-based reconstructed pixel value of the missing pixel. The reconstruction result fusion unit is communicatively connected to the spatial dimension reconstruction unit and the temporal dimension reconstruction unit, respectively, and is used to fuse the remote sensing image value of the missing pixel based on the reconstructed pixel value based on space and time respectively.

8. A computer device, characterized in that, The system includes a storage module, a processing module, and a transceiver module that are sequentially connected in communication. The storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the remote sensing image reconstruction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores instructions that, when executed on a computer, perform the remote sensing image reconstruction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the remote sensing image reconstruction method as described in any one of claims 1 to 6.

Images