Remote sensing image restoration method, device, electronic equipment and medium

Abstract

The present invention provides a remote sensing image restoration method, device, electronic device and medium. The specific implementation scheme includes: obtaining an image to be processed and a candidate image set of a target area, wherein the image to be processed is a remote sensing image containing a problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times; performing information recognition on non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed; determining a candidate remote sensing image that matches the image to be processed from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set to obtain a reference image; and using the reference image to restore the image to be processed to obtain a target image.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, image processing, and the like, and in particular to a remote sensing image restoration method, device, electronic equipment, and storage medium. Background Art

[0002] Due to the defects of sensor structure or the influence of imaging methods, some valid data in remote sensing images are missing, which reduces the usability of remote sensing images. Therefore, it is necessary to restore or complete the missing data in remote sensing images. Summary of the invention

[0003] The present invention provides a remote sensing image restoration method, device, electronic equipment, storage medium and computer program product.

[0004] According to one aspect of the present invention, a remote sensing image restoration method is provided, comprising: obtaining an image to be processed and a candidate image set of a target area, wherein the image to be processed is a remote sensing image containing a problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times; identifying information of non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed; determining a candidate remote sensing image that matches the image to be processed from the candidate image set based on the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set to obtain a reference image; and restoring the image to be processed using the reference image to obtain a target image.

[0005] According to another aspect of the present invention, a remote sensing image restoration device is provided, comprising: an acquisition module, used to acquire an image to be processed and a candidate image set of a target area, wherein the image to be processed is a remote sensing image containing a problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times; an identification module, used to perform information identification on non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed; a determination module, used to determine a candidate remote sensing image that matches the image to be processed from the candidate image set based on the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, to obtain a reference image; and a restoration module, used to restore the image to be processed using the reference image, to obtain a target image.

[0006] According to another aspect of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the remote sensing image restoration method as described above.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the remote sensing image restoration method as described above.

[0008] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, which implements the remote sensing image restoration method as described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 is a flow chart of a remote sensing image restoration method according to an embodiment of the present invention;

[0011] Figure 2 is a schematic diagram of a process of training an initial recognition model according to an embodiment of the present invention;

[0012] Figure 3 is a block diagram of a remote sensing image restoration device according to an embodiment of the present invention;

[0013] Figure 4 It is a block diagram of an electronic device suitable for implementing a remote sensing image restoration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention and the drawings in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0015] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0016] In addition, in the description of the present invention, the terms “first”, “second”, etc. (if any) are used only to distinguish the description and cannot be understood as indicating or implying relative importance.

[0017] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (for example, including but not limited to user personal information) shall comply with the relevant laws and regulations of the relevant countries and regions and shall not violate public order and good morals.

[0018] In addition, in the technical solution of the present invention, before obtaining or collecting relevant data, the authorization or consent of the data owner is obtained.

[0019] In the process of remote sensing image acquisition, due to the defects of sensor structure or the influence of imaging methods, there will be data missing in remote sensing images. These data missing areas are difficult to reflect the real situation on the ground, which is not conducive to the information extraction of remote sensing images and reduces the usability of remote sensing images. Therefore, it is necessary to restore or complete the missing data in remote sensing images.

[0020] Related technologies usually use time-based methods to perform remote sensing image restoration. Time-based methods for image restoration usually refer to using image sequences taken at different times in the same area to repair damaged or missing data in the image to be restored. However, the above-mentioned remote sensing image restoration method needs to rely on the imaging time of the image to be restored to select a suitable image sequence to ensure that the selected images are close enough in time so that they can reflect similar surface features, while reducing the impact of surface cover changes caused by time differences on the image restoration effect. If the imaging time of the image to be restored is missing or distorted, it will be difficult to ensure the rationality and accuracy of the selected phase, which will affect the accuracy and reliability of the restoration effect.

[0021] Figure 1 is a flow chart of a remote sensing image restoration method according to an embodiment of the present invention.

[0022] like Figure 1 As shown, the remote sensing image restoration method 100 includes operations S110 to S140.

[0023] In operation S110 , an image to be processed and a candidate image set of a target area are obtained, the image to be processed is a remote sensing image containing the problem area, and the candidate image set includes candidate remote sensing images obtained at multiple imaging times.

[0024] In operation S120, information recognition is performed on non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed.

[0025] In operation S130, according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, a candidate remote sensing image matching the image to be processed is determined from the candidate image set to obtain a reference image.

[0026] In operation S140 , the image to be processed is restored using the reference image to obtain a target image.

[0027] According to an embodiment of the present invention, the image to be processed may be, for example, any optical remote sensing image of the target area that includes the problem area. The target area refers to, for example, a certain geographical space or geographic range, and the target area has its own location information (or longitude and latitude). For example, the target area may be a province, a city, a county, a district, etc., and may also be other specific geographic ranges, which are not limited by the present invention.

[0028] According to an embodiment of the present invention. The problem area may, for example, refer to an area where data is missing in the image to be processed. The problem area may, for example, include one of a noise area, a null value area, a thick cloud area, a thin cloud area, a fog area, a shadow area, and a fuzzy area. Among them, the noise area may include a Gaussian noise area, a salt and pepper noise area, a Poisson noise area, an impulse noise area, a stripe noise area, and a mixed noise area. A mixed noise area refers to a noise area in which two or more types of noise exist.

[0029] According to an embodiment of the present invention, the candidate image set includes candidate remote sensing images corresponding to the target area acquired at multiple imaging times. Each candidate remote sensing image in the candidate image set may be an optical remote sensing image acquired by sensors carried by different satellites, or may be an optical remote sensing image acquired by different sensors carried by the same satellite. In an embodiment of the present invention, the imaging time may, for example, refer to the time point at which the carried sensor samples the ground objects in the target area.

[0030] Each candidate remote sensing image can be, for example, a high-quality optical remote sensing image that does not contain a problem area, or each candidate remote sensing image can be an optical remote sensing image that contains a problem area, but the area corresponding to the problem area of ​​the image to be processed in each candidate remote sensing image is a non-problem area. A reference image that matches the image to be processed can be selected from the candidate image set so that the image to be processed can be repaired using the selected reference image.

[0031] According to an embodiment of the present invention, the actual imaging time of the image to be processed may be missing or distorted. In this case, information recognition can be performed on the non-problem area of ​​the image to be processed except the problem area, so as to determine the time information corresponding to at least one object in the non-problem area of ​​the image to be processed, and then determine the imaging time of the image to be processed according to the time information corresponding to at least one object in the non-problem area. In this way, the time range corresponding to the image to be processed can be clarified by identifying the time information of the objects in the non-problem area of ​​the image to be processed.

[0032] According to an embodiment of the present invention, each object in the non-problem area refers to an independent feature located in the non-problem area of ​​the image to be processed, such as a house, rice, etc. The time information corresponding to each object in the non-problem area is used to indicate the time range in which each object is located when the image to be processed is imaged. The time information can be information indicating the phenological period or growth cycle of the object in the image to be processed, or information indicating the imaging time of the image to be processed.

[0033] It can be understood that since the time information corresponding to each object in the non-problem area of ​​the image to be processed can reflect the imaging time of the image to be processed, when the actual imaging time of the image to be processed is missing or distorted, the imaging time of the image to be processed can be represented by integrating these time information, so that a reference image with a reasonable phase can be screened according to the imaging time of the above-mentioned image to be processed, and image restoration can be performed accordingly, which is beneficial to improving the accuracy and reliability of the image restoration effect.

[0034] According to an embodiment of the present invention, after determining the imaging time of the image to be processed, a reference image can be matched from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, so that the time interval between the reference image and the image to be processed is as short as possible, so as to minimize the impact of surface cover changes caused by time differences on the image restoration effect.

[0035] According to an embodiment of the present invention, the area in the reference image corresponding to the problem area of ​​the image to be processed is a non-problem area. Based on the mapping relationship between the reference image and the image to be processed, the problem area of ​​the image to be processed can be repaired according to the spectral characteristics and spatial texture of the pixel points in the corresponding area in the reference image to obtain a target image.

[0036] In the technical solution of the present invention, by identifying information of non-problem areas other than the problem area in the image to be processed, the time information corresponding to at least one object in the non-problem area of ​​the image to be processed can be determined. Since the time information corresponding to each object in the non-problem area of ​​the image to be processed can reflect the imaging time of the image to be processed, the imaging time of the image to be processed can be represented based on the time information corresponding to at least one object in the non-problem area, and then according to the determined imaging time of the image to be processed, a reference image with a reasonable phase can be screened, so that the reference image can express the information contained in the problem area of ​​the image to be processed to a certain extent, which is conducive to improving the image restoration effect. According to the technical solution of the present invention, even if the actual imaging time of the image to be processed is missing or distorted, the accuracy and reliability of the image restoration effect are guaranteed to a certain extent.

[0037] According to an embodiment of the present invention, the above-mentioned process of performing information recognition on the non-problem area other than the problem area in the image to be processed to determine the imaging time of the image to be processed may include the following process.

[0038] Firstly, the recognition model is used to identify the non-problem areas of the image to be processed, and the time information and ground object categories corresponding to multiple objects in the non-problem areas are obtained.

[0039] According to an embodiment of the present invention, the recognition model is obtained by training the initial recognition model using sample remote sensing images carrying time information labels and ground object category labels. Figure 2 The training process of the initial recognition model is illustrated.

[0040] like Figure 2 As shown, the initial recognition model may include, for example, a feature extraction module 210 , a feature fusion module 220 , a ground object recognition module 230 and a time recognition module 240 .

[0041] The feature extraction module 210 may be, for example, a network structure composed of multiple convolutional layers and pooling layers, and is used to extract features from an input image to obtain multiple scale feature maps.

[0042] The feature fusion module 220 can, for example, be composed of a network structure for multi-scale fusion (such as a feature pyramid network or a U-Net network), which is used to perform multi-scale fusion processing on multiple scale feature maps to obtain multiple multi-scale fused feature maps, and perform fusion processing on multiple multi-scale fused feature maps to obtain a target fused feature map.

[0043] The object recognition module 230 may be, for example, a network structure composed of one or more fully connected layers, and is used to predict the object category according to the target fusion feature map.

[0044] The time identification module 240 may be, for example, a network structure composed of one or more fully connected layers, and is used to predict time information according to a target fusion feature map.

[0045] It should be noted that the structure of the initial recognition model is not limited to Figure 2 In some embodiments, the above-mentioned time recognition module can be added to the pre-trained classification model or semantic segmentation model (for example, including but not limited to convolutional neural network, residual neural network, Transormer-based neural network, etc.) to obtain an initial recognition model. Then, the initial recognition model can be fine-tuned using sample remote sensing images carrying time information labels and object category labels to obtain a recognition module.

[0046] Below Figure 2Taking the structure of the initial recognition model shown in FIG. 1 as an example, the training process of the initial recognition model is exemplarily described.

[0047] First, a sample remote sensing image is obtained, where the sample remote sensing image includes a sample object and a time information label and a ground object category label of the sample object.

[0048] According to an embodiment of the present invention, the sample remote sensing image may be, for example, an optical remote sensing image including at least one sample object. The sample object may include, for example, but is not limited to, sample objects of land object types such as woodland, rice, wheat, grassland, water body, and building. The time information tag is used to mark the time information corresponding to each sample object in the sample remote sensing image. The time information may be information indicating the phenological period or growth cycle of the sample object in the sample remote sensing image, or information indicating the imaging time of the sample remote sensing image. The land object category tag is used to mark the land object category corresponding to each sample object in the sample remote sensing image.

[0049] Next, the acquired sample remote sensing image may be input into the feature extraction module 210 for feature extraction to obtain multiple scale feature maps of the sample remote sensing image.

[0050] Next, the feature fusion module 220 is used to perform multi-scale fusion processing on the multiple scale feature maps to obtain multiple multi-scale fusion feature maps, and the multiple multi-scale fusion feature maps are fused to obtain a target fusion feature map.

[0051] Next, the object recognition module 230 is used to predict the object category based on the target fusion feature map, and the sample object category corresponding to each sample object in the sample remote sensing image is obtained. The time recognition module 240 is used to predict the time information based on the target fusion feature map, and the sample time information corresponding to each sample object in the sample remote sensing image is obtained.

[0052] Next, the sample loss is determined based on the sample time information, the time information label, the sample object category and the object category label.

[0053] According to an embodiment of the present invention, the sample loss is determined based on a first loss and a second loss. The first loss is obtained based on the difference between the sample time information and the time information label. The second loss is obtained based on the difference between the sample object category and the object category label. In an example, the first loss can be calculated using a mean square error loss function, and the second loss can be calculated using a cross entropy loss function.

[0054] Next, the parameters of the initial recognition model are adjusted according to the sample loss to obtain the recognition model.

[0055] According to an embodiment of the present invention, sample loss can be used to adjust the parameters of the feature extraction module, feature fusion module, object recognition module and time recognition module in the initial recognition model until the training end condition is met. The training end condition refers to, for example, that the cumulative number of model trainings reaches a preset number threshold, or that the sample loss meets a preset convergence condition. The above recognition model obtained through training can be used to perform information recognition on the input image to obtain the time information and object category corresponding to each object in the input image.

[0056] After using the recognition model to identify information in the non-problem area of ​​the image to be processed and obtaining the time information and object categories corresponding to each of the multiple objects in the non-problem area of ​​the image to be processed, it can be determined whether the object categories corresponding to the multiple objects include the target object category.

[0057] According to an embodiment of the present invention, the target object category may refer to, for example, an object category such as rice, wheat, etc. that will change significantly over time. Compared with other object categories (such as buildings), the target object category (such as rice) usually has a certain growth cycle law, and at different growth stages, it has corresponding feature information (such as but not limited to spectrum, color, texture, etc.). Based on this feature information, the time information corresponding to the target object category can be better determined, which is conducive to accurately estimating the imaging time of the remote sensing image. In view of this, the time information of the target object corresponding to the target object category can be used to characterize the imaging time of the image to be processed, so as to improve the accuracy and reliability of the imaging time of the image to be processed.

[0058] If it is determined that the ground object categories corresponding to the plurality of objects respectively include the target ground object category, the time information of the target object corresponding to the target ground object category is determined as the imaging time of the image to be processed.

[0059] If it is determined that the object categories corresponding to the multiple objects do not include the target object category, the number of time information that falls into the same time range among the time information corresponding to the multiple objects is determined, and the time information with the largest number is used as the imaging time of the image to be processed. In the embodiment of the present invention, the "same time range" mentioned here can be in units of days, months, or quarters, etc., which can be set according to actual needs.

[0060] For example, assuming that the object categories corresponding to multiple objects do not include the target object category, and their corresponding time information includes "August 1, 2023", "August 5, 2023", "August 4, 2023", and "August 4, 2023", then the time information "August 4, 2023" can be used to represent the imaging time of the image to be processed.

[0061] It should be noted that when it is determined that the object categories corresponding to multiple objects do not include the target object category, the average or median of the time information of all objects can be taken as the imaging time of the image to be processed, or other appropriate methods can be used to determine the imaging time of the image to be processed.

[0062] In addition, the target object category may include at least one of the target object categories described above. If the time information of multiple target object categories is inconsistent, the average, median or mode of the time information of all target object categories may be taken as the imaging time of the image to be processed, or other appropriate methods may be used to determine the imaging time of the image to be processed.

[0063] According to an embodiment of the present invention, the reference image may be determined in the following manner.

[0064] In one example, the imaging time of the image to be processed can be used to query the candidate image set. If at least one candidate remote sensing image whose imaging time is consistent with the imaging time of the image to be processed can be screened out from the candidate image set, that is, at least one candidate remote sensing image whose imaging time is the same as the imaging time of the image to be processed, then a candidate remote sensing image with higher image quality is selected from the at least one screened candidate remote sensing image as a reference image. If a candidate remote sensing image whose imaging time is consistent with the imaging time of the image to be processed is not screened out from the candidate image set, a candidate remote sensing image whose imaging time is the shortest between the imaging time and the imaging time of the image to be processed is selected from the candidate image set as a reference image.

[0065] According to an embodiment of the present invention, using a reference image to repair the image to be processed to obtain a target image may include: performing multiple iterative repair processes on the problem area in the image to be processed according to the reference image to obtain an initial repair image including an initial repair area, wherein the initial repair area corresponds to the spatial position of the problem area. Then, performing multiple iterative optimization processes on the initial repair area in the initial repair image to obtain the target image.

[0066] According to an embodiment of the present invention, the process of performing multiple iterative repair processes on the problem area can be, for example, a process of starting from the edge of the problem area and gradually repairing the internal space. The pixel points restored in each iterative repair process are regarded as known pixel points and used as potential similar pixel points in the next iterative repair process, which makes the distance between the target pixel points and the potential similar pixel points in each iterative repair process as close as possible to reduce spectral changes, thereby improving the effectiveness of similar pixel points, and further facilitating the improvement of the accuracy and reliability of the repair effect.

[0067] The following is a detailed introduction to the repair process of each iteration.

[0068] According to an embodiment of the present invention, each iterative repair process includes the following process.

[0069] First, an n-1th iteration region is obtained after performing an n-1th iteration repair process on the problem region.

[0070] According to an embodiment of the present invention, the n-1th iteration region corresponds to the spatial position of the problem region in the image to be processed. The n-1th iteration region includes the n-1th problem region and the n-1th repair region, wherein the n-1th problem region is located in the internal space of the n-1th iteration region. The n-1th repair region refers to the pixel repair region in the n-1th iteration region, which includes all pixels restored in the first n-1 iteration repair processes, and these restored pixels can be used as known information for subsequent filling. The n-1th problem region refers to the region in the n-1th iteration region where there are still missing pixels, that is, the region in the n-1th iteration region that has not been repaired, and n is an integer greater than 1.

[0071] Next, the boundary pixel points of the n-1th problem area are determined to obtain a plurality of first target pixel points.

[0072] According to an embodiment of the present invention, the boundary pixel point may refer to, for example, a pixel point located at the edge of the n-1th problem area. The plurality of first target pixel points are pixel points that need to be restored in the nth iteration repair process.

[0073] In one example, for example, the boundary pixel points of the n-1th problem area can be determined by an erosion operation to obtain multiple first target pixel points. For example, a 3*3 structure element can be applied to detect the boundary pixel points of the n-1th problem area by an erosion operation to obtain multiple first target pixel points. Of course, the embodiments of the present invention can also use other suitable methods to determine the boundary pixel points of the n-1th problem area, which is not limited here.

[0074] Next, pixel points corresponding to the plurality of first target pixel points in the reference image are determined to obtain a plurality of first reference pixel points.

[0075] According to an embodiment of the present invention, for example, a first reference pixel corresponding to each first target pixel may be determined in the reference image according to the coordinates of the first target pixel. The first reference pixel corresponds to the coordinates of the corresponding first target pixel.

[0076] Next, for each first reference pixel point, a first similar pixel point associated with the first reference pixel point is searched within a first neighborhood range corresponding to the first reference pixel point.

[0077] According to an embodiment of the present invention, the first neighborhood is a neighborhood corresponding to the first reference pixel determined in the reference image. The area corresponding to the first neighborhood in the image to be processed after the n-1th iterative repair process is at least partially located in the n-1th repair area. The first neighborhood includes the first reference pixel and a plurality of first neighboring pixels associated with the first reference pixel. The size, shape, etc. of the first neighborhood can be set according to actual needs and are not limited here.

[0078] According to an embodiment of the present invention, searching for a first similar pixel point associated with the first reference pixel point within a first neighborhood range corresponding to the first reference pixel point may include the following process.

[0079] First, a plurality of first neighboring pixel points associated with the first reference pixel point within the first neighborhood are determined. For example, in a first neighborhood of size 3*3, there are 8 first neighboring pixel points associated with the first reference pixel point.

[0080] Next, a first spectral deviation value between the first reference pixel and each first adjacent pixel is determined.

[0081] According to an embodiment of the present invention, the reference image and the image to be processed may include data of multiple bands, for example. The first spectral deviation value between each first neighboring pixel and the first reference pixel may include, for example, the spectral deviation value of the first neighboring pixel and the first reference pixel in all bands.

[0082] According to an embodiment of the present invention, assuming that the total number of bands of the reference image is p, the number of first neighboring pixels in the first neighborhood is q, and p and q are both positive integers. The spectral deviation value between each first neighboring pixel and the first reference pixel in each band can be determined using the following formula (1).

[0083] (1)

[0084] In formula (1), (x i ,y i ) represents the coordinates of the i-th first neighboring pixel in the first neighborhood, (x, y) represents the coordinates of the first reference pixel, b represents the b-th band in the p bands, Indicates the (x i , y i ) is the spectral value of the first neighboring pixel point at the i-th location, Represents the spectral value of the first reference pixel at (x, y) in the bth band in the reference image. Indicates the (x i ,y i) and the first reference pixel at (x, y). i=1,2,…,q,b=1,2,…, p.

[0085] Next, each first spectral deviation value is compared with the first spectral deviation threshold. If a first spectral deviation value is less than the first spectral deviation threshold, the first neighboring pixel point corresponding to the first spectral deviation value is determined as the first similar pixel point; otherwise, the first neighboring pixel point corresponding to the first spectral deviation value is determined not to be the first similar pixel point.

[0086] According to an embodiment of the present invention, the first spectral deviation threshold may include, for example, a first sub-spectral deviation threshold corresponding to each band. The first sub-spectral deviation threshold corresponding to each band may be determined according to the spectral values ​​of all pixels in the first neighborhood in the band.

[0087] In one example, for example, a first local deviation value can be determined based on the spectral values ​​of all pixels (including the first reference pixel and multiple first neighboring pixels) in each band within the first neighborhood range, and the first local deviation value can be used as the first sub-spectrum deviation threshold corresponding to the band.

[0088] In one example, the following formula (2) may be used to determine the first sub-spectrum deviation threshold corresponding to each band.

[0089] (2)

[0090] In formula (2), (x k ,y k ) represents the coordinates of the kth pixel point in the first neighborhood, b represents the bth band among the p bands, Indicates the (x k ,y k ) is the spectral value of the kth pixel point, represents the average spectral value of all pixels in the first neighborhood of the bth band in the reference image, q represents the number of first neighboring pixels in the first neighborhood, and th ref_b Indicates the first sub-spectrum deviation threshold corresponding to the b-th band. k=1,2,…,q+1,b=1,2,…,p.

[0091] According to an embodiment of the present invention, comparing the first spectral deviation value with the first spectral deviation threshold comprises comparing the spectral deviation value of the first neighboring pixel and the first reference pixel in each band with the first sub-spectral deviation threshold corresponding to the band.

[0092] If the spectral deviation values ​​of the first neighboring pixel point and the first reference pixel point in each band are both smaller than the first sub-spectral deviation threshold corresponding to the band, it is determined that the first spectral deviation value is smaller than the first spectral deviation threshold, and the first neighboring pixel point corresponding to the first spectral deviation value can be determined as the first similar pixel point.

[0093] If the spectral deviation value between the first neighboring pixel and the first reference pixel in a certain band is not less than the first sub-spectral deviation threshold corresponding to the band, it is determined that the corresponding first neighboring pixel is not the first similar pixel.

[0094] In some embodiments, if not enough first similar pixel points are found within the first neighborhood, for example, when the number of first similar pixel points does not reach a preset number threshold, the first neighborhood can be updated to another first neighborhood to expand the pixel search range. Then, based on the other first neighborhood, the operation of searching for first similar pixel points associated with the first reference pixel point within the first neighborhood corresponding to the first reference pixel point is repeated until a sufficient number of first similar pixel points are found, or until the size of the first neighborhood reaches a preset maximum value.

[0095] Through the above process, a plurality of first similar pixel points associated with the first reference pixel point may be determined.

[0096] In some embodiments, if the first similar pixel associated with the first reference pixel still cannot be searched based on the above process, the repair process of the first target pixel corresponding to the first reference pixel is abandoned during the nth iterative repair process. After the repair process of all pixels in the problem area of ​​the image to be processed is completed, the initial repaired image is obtained, and the initial repair area in the initial repaired image is optimized for multiple iterations, the first target pixels can be repaired during the multiple iterations. This process will be described later.

[0097] Next, based on the spectral values ​​of the multiple first similar pixels respectively associated with the multiple first reference pixels in the processed image and the reference image after the n-1th iterative repair process and the first weights corresponding to the each first similar pixel, the spectral mapping relationship between each first reference pixel and the corresponding first target pixel is determined.

[0098] According to an embodiment of the present invention, the first weight corresponding to each first similar pixel point may be determined in the following manner.

[0099] First, first spatial distances between a plurality of first similar pixel points associated with each first reference pixel point and the first reference pixel point are determined.

[0100] Next, for each first similar pixel point, a first weight corresponding to the first similar pixel point is determined according to the first spatial distance.

[0101] It can be understood that as the first spatial distance increases, the change between the first similar pixel and the first reference pixel will increase significantly, and correspondingly, the change between the first similar pixel and the corresponding first target pixel will also increase significantly. This indicates that the closer first similar pixel is more likely to maintain similarity with the first target pixel, while the farther first similar pixel may show lower similarity. Therefore, the first similar pixel that is closer to the first target pixel in space contributes more to the prediction process of the first target pixel.

[0102] According to an embodiment of the present invention, the first weight is used to characterize the contribution of the first similar pixel to the restoration of the corresponding first target pixel. The first weight corresponding to each first similar pixel is negatively correlated with the first spatial distance corresponding to the first similar pixel.

[0103] In an example, the following formula (3) may be used to determine the first weight corresponding to each first similar pixel point associated with the first reference pixel point.

[0104] (3)

[0105] In formula (3), (x, y) represents the coordinates of the first reference pixel, (x r ,y r ) represents the coordinates of the rth first similar pixel associated with the first reference pixel, w r represents the first weight corresponding to the rth first similar pixel associated with the first reference pixel, n_r represents the number of first similar pixels associated with the first reference pixel, n_r is a positive integer, r=1, 2,…, n_r.

[0106] According to an embodiment of the present invention, the above-mentioned determination of the spectral mapping relationship between each first reference pixel point and the corresponding first target pixel point includes: determining the mapping parameters based on the spectral values ​​of multiple first similar pixels respectively associated with the multiple first reference pixels in the processed image and the reference image after the n-1th iterative repair processing, and the first weights corresponding to each first similar pixel point; and constructing the spectral mapping relationship based on the mapping parameters.

[0107] According to an embodiment of the present invention, the above mapping parameters may include, for example, sub-mapping parameters corresponding to each band, and correspondingly, the spectrum mapping relationship may include, for example, sub-spectrum mapping relationships corresponding to each band, wherein the sub-spectrum mapping relationships corresponding to each band are constructed based on the sub-mapping parameters corresponding to the band.

[0108] The following describes the process of determining the sub-mapping parameters and sub-spectrum mapping relationship corresponding to each band.

[0109] In the embodiment of the present invention, it is assumed that the number of first similar pixel points associated with the plurality of first reference pixel points is N, where N is a positive integer. The objective function can be constructed using the spectral values ​​of the N first similar pixel points on the bth (b=1, 2, …, p)th band in the processed image and the reference image after the n-1th iteration restoration process, and the first weights corresponding to the first similar pixel points, so as to determine the sub-mapping parameters corresponding to the bth band by solving the objective function.

[0110] According to an embodiment of the present invention, the objective function may be expressed using the following formula (4), for example.

[0111] (4)

[0112] In formula (4), represents the objective function of the sub-mapping parameters for the b-th band, (x j ,y j ) represents the jth first similar pixel among N first similar pixels, w j represents the first weight corresponding to the jth first similar pixel point, b represents the bth band among the p bands corresponding to the reference image and the image to be processed after the n-1th iterative restoration process, and Indicates the sub-mapping parameters corresponding to the b-th band, It represents the (x j ,y j ) is the spectral value of the jth first similar pixel point, Indicates the (x j ,y j ) at , j=1,2,…,N.

[0113] Next, a sub-spectrum mapping relationship corresponding to the b-th band is constructed based on the sub-mapping parameters corresponding to the b-th band.

[0114] For example, the following formula (5) can be used to express the sub-spectrum mapping relationship corresponding to the b-th band.

[0115] (5)

[0116] In formula (5), It represents the (xe ,y e ) at the first target pixel, Indicates the (x e ,y e ) is the spectral value of the first reference pixel point at .

[0117] Next, based on the spectral mapping relationship, the spectral value of the corresponding first target pixel is filled according to the spectral value of each first reference pixel in the reference image to obtain the nth iteration area corresponding to the nth iteration repair process.

[0118] According to an embodiment of the present invention, for each band, the sub-spectral mapping relationship corresponding to the band can be used to determine the spectral value of the corresponding first target pixel in the band according to the spectral value of each first reference pixel in the reference image in the band, and fill the first target pixel based on the spectral value of the first target pixel in the band. Through the above operation, the nth iteration area corresponding to the nth iteration repair process can be obtained. The nth iteration area corresponds to the spatial position of the problem area in the image to be processed.

[0119] Next, determine whether there are unprocessed pixels in the nth iteration area. The unprocessed pixels here refer to pixels in the nth iteration area other than the restored pixels and the pixels that were not successfully restored (i.e., the pixels that were abandoned for restoration because no similar pixels were found).

[0120] If it is determined that there are no unprocessed pixels in the nth iteration area, the image to be processed after the nth iteration restoration process is determined as the initial restoration image, and the nth iteration area corresponds to the initial restoration area in the initial restoration image.

[0121] If it is determined that there are still unprocessed pixels in the nth iteration area, determine the nth problem area in the nth iteration area. The area in the nth iteration area except the nth problem area is the nth repair area, which includes all the pixels restored in the previous n iteration repair processes. These restored pixels can be used as known information for subsequent filling. Then, according to the above-mentioned process of iteratively repairing the n-1th problem area, continue to iteratively repair the nth problem area, and repeat the operation until the problem area in the image to be processed is completely filled, or all the pixels in the problem area of ​​the image to be processed are traversed (regardless of whether the pixels are repaired), and the initial repair image containing the initial repair area is obtained.

[0122] It can be understood that, except for the pixels that have not been restored (if any), the other pixels in the problem area of ​​the image to be processed have been restored after multiple iterations of restoration. These restored pixels can be regarded as known pixels. In each iterative optimization process, these known pixels can be used as potential similar pixels to optimize the target pixels in the initial restoration area to reduce spectral distortion and restore more accurate spectral features, thereby improving the accuracy and reliability of image restoration.

[0123] In addition, compared with the image to be processed, the initial repair image contains a larger range of known pixels, so that more and closer potential similar pixels can be searched. Therefore, for the first target pixel that fails to search for the first similar pixel in each of the above iterative repair processes, the first target pixel can also be repaired based on multiple iterative optimization processes.

[0124] According to an embodiment of the present invention, performing multiple iterative optimization processes on the initial repair area in the initial repair image may be, for example, a process of starting from the edge of the initial repair area and gradually optimizing the internal space. This makes the distance between the target pixel and the potential similar pixel as close as possible during each iterative optimization process, thereby improving the effectiveness of the similar pixel, and further facilitating improving the optimization effect of the target pixel.

[0125] The following is a detailed introduction to the process of each iterative optimization process.

[0126] According to an embodiment of the present invention, each iterative optimization process includes the following process.

[0127] First, obtain the m-1th iterative optimization region obtained after performing the m-1th iterative optimization process on the initial repair region, and the m-1th iterative optimization region corresponds to the spatial position of the initial repair region in the initial repair image. The m-1th iterative optimization region includes the m-1th region to be optimized and the m-1th optimization region, wherein the m-1th region to be optimized is located in the internal space of the m-1th iterative optimization region. The m-1th optimization region refers to the region in the m-1th iterative optimization region where pixel optimization has been completed, and it includes all the pixels that have been optimized in the first m-1 iterative optimization processes, and these optimized pixels can be used as known information for subsequent optimization. The m-1th region to be optimized refers to the region in the m-1th iterative optimization region that has not yet been pixel optimized, and m is an integer greater than 1. In some embodiments, the m-1th region to be optimized, for example, can also be a pixel point region that has not been repaired during multiple iterative repair processes.

[0128] Next, the boundary pixel points of the (m-1)th area to be optimized are determined to obtain a plurality of second target pixel points.

[0129] According to an embodiment of the present invention, the plurality of second target pixels are pixels that need to be processed in the mth iteration optimization process. The determination method of the plurality of second target pixels is similar to the determination method of the plurality of first target pixels, which will not be repeated here.

[0130] Next, for each second target pixel point, a second similar pixel point associated with the second target pixel point is searched within a second neighborhood corresponding to the second target pixel point, and the target spectral value corresponding to the second target pixel point is determined based on the spectral values ​​corresponding to each of the searched multiple second similar pixel points in the initial repaired image after the m-1th iterative optimization process and the second weights corresponding to each of the multiple second similar pixel points.

[0131] According to an embodiment of the present invention, determining a second similar pixel point associated with a second target pixel point includes the following process.

[0132] First, a plurality of second neighboring pixel points associated with the second target pixel point within a second neighborhood range are determined.

[0133] According to an embodiment of the present invention, the second neighborhood is a neighborhood corresponding to the second target pixel point determined in the initial restoration image after the m-1th iterative optimization process. The second neighborhood is at least partially located in the m-1th optimization area. The second neighborhood includes the second target pixel point and a plurality of second neighboring pixels associated with the second target pixel point. The size, shape, etc. of the second neighborhood can be set according to actual needs and are not limited here.

[0134] Next, a second spectral deviation value between the second target pixel and each second adjacent pixel is determined.

[0135] According to an embodiment of the present invention, the second spectral deviation value between each second neighboring pixel point and the second target pixel point may, for example, include the spectral deviation values ​​of the second neighboring pixel point and the second target pixel point in all bands. The process of determining the spectral deviation value of each second neighboring pixel point and the second target pixel point in each band is similar to the process of determining the spectral deviation value of each first neighboring pixel point and the first reference pixel point in each band, and will not be repeated here.

[0136] Next, each second spectral deviation value is compared with the second spectral deviation threshold value. If it is determined that the second spectral deviation value is less than the second spectral deviation threshold value, the second adjacent pixel point corresponding to the second spectral deviation value is determined as the second similar pixel point; otherwise, the second adjacent pixel point corresponding to the second spectral deviation value is determined not to be the second similar pixel point.

[0137] According to an embodiment of the present invention, the second spectral deviation threshold may include, for example, a second sub-spectral deviation threshold corresponding to each band. The second sub-spectral deviation threshold corresponding to each band may be determined according to the spectral values ​​of all pixels in the second neighborhood in the band.

[0138] In one example, for example, the second local deviation value can be determined based on the spectral values ​​of all pixels (including the second target pixel and multiple second neighboring pixels) in each band within the second neighborhood, and the second local deviation value is used as the second sub-spectrum deviation threshold corresponding to the band. In an embodiment of the present invention, the process of determining the second sub-spectrum deviation threshold is similar to the process of determining the first sub-spectrum deviation threshold, which will not be repeated here.

[0139] According to an embodiment of the present invention, comparing the second spectral deviation value with the second spectral deviation threshold comprises comparing the spectral deviation value of the second neighboring pixel and the second target pixel in each band with the second sub-spectral deviation threshold corresponding to the band.

[0140] If the spectral deviation values ​​of the second neighboring pixel point and the second target pixel point in each band are both smaller than the second sub-spectral deviation threshold corresponding to the band, it is determined that the second spectral deviation value is smaller than the second spectral deviation threshold, and the second neighboring pixel point corresponding to the second spectral deviation value can be determined as the second similar pixel point.

[0141] If the spectral deviation value between the second neighboring pixel and the second target pixel in a certain band is not less than the second sub-spectral deviation threshold corresponding to the band, it is determined that the corresponding second neighboring pixel is not the second similar pixel.

[0142] Similarly, if sufficient second similar pixel points cannot be found within the second neighborhood, the second neighborhood may be expanded in the manner described above, so as to find a sufficient number of second similar pixel points in the expanded second neighborhood.

[0143] Through the above process, a plurality of second similar pixel points associated with the second target pixel point may be determined.

[0144] According to an embodiment of the present invention, the second weight corresponding to each second similar pixel point may be determined in the following manner.

[0145] First, second spatial distances between a plurality of second similar pixel points associated with a second target pixel point and the second target pixel point are determined.

[0146] Next, for each second similar pixel point, a second weight corresponding to the second similar pixel point is determined according to the second spatial distance.

[0147] According to an embodiment of the present invention, the second weight is used to characterize the contribution of the second similar pixel to the optimization of the corresponding second target pixel. The second weight corresponding to each second similar pixel is negatively correlated with the second spatial distance corresponding to the second similar pixel.

[0148] In an example, the following formula (6) may be used to determine the second weight corresponding to each second similar pixel point associated with the second target pixel point.

[0149] (6)

[0150] In formula (3), (x m ,y m ) represents the coordinates of the second target pixel, (x f ,y f ) represents the coordinates of the fth second similar pixel point associated with the second target pixel point, w f represents the second weight corresponding to the f-th second similar pixel associated with the second target pixel, n_f represents the number of second similar pixels associated with the second target pixel, n_f is a positive integer, f=1,2,…,n_f.

[0151] According to an embodiment of the present invention, the target spectral value corresponding to each second target pixel point includes sub-target spectral values ​​of the second target pixel point in each waveband.

[0152] The following describes a process for determining the sub-target spectral value of each second target pixel in each band.

[0153] In the embodiment of the present invention, it is assumed that the number of second similar pixels associated with each second target pixel is n_f, where n_f is a positive integer. The sub-target spectral value of the second target pixel in the bth band (b=1, 2, ..., p) of the initial restored image after the m-1th iterative optimization process can be determined by using the spectral values ​​of the n_f second similar pixels in the bth band (b=1, 2, ..., p) of the n_f second similar pixels and the second weights corresponding to the second similar pixels.

[0154] According to an embodiment of the present invention, the sub-target spectrum value of each second target pixel in the bth band can be expressed, for example, using the following formula (7).

[0155] (7)

[0156] In formula (7), (x m ,y m ) represents the coordinates of the second target pixel, n_f represents the number of second similar pixels associated with the second target pixel, (x f ,y f) represents the coordinates of the fth second similar pixel point associated with the second target pixel point, w f represents the second weight corresponding to the f-th second similar pixel point associated with the second target pixel point, represents the (x in the bth band in the initial repaired image after the m-1th iterative optimization process f ,y f ) at the spectral value of the fth second similar pixel point, represents the (x in the bth band in the initial repaired image after the m-1th iterative optimization process m ,y m ), f=1,2,…,n_f.

[0157] Next, the spectral value corresponding to each second target pixel point in the initial restoration image is optimized according to the target spectral value corresponding to each second target pixel point, so as to obtain the mth iterative optimization region corresponding to the mth optimization process.

[0158] According to an embodiment of the present invention, the target spectral value corresponding to each second target pixel point can be used to replace the spectral value corresponding to each second target pixel point in the initial restoration image, thereby obtaining the mth iterative optimization region corresponding to the mth optimization process. The mth iterative optimization region corresponds to the spatial position of the initial restoration region in the initial restoration image.

[0159] Next, it is determined whether there are any unoptimized pixels in the mth iterative optimization region.

[0160] If it is determined that there are no unoptimized pixels in the m-th iterative optimization area, the initial restoration image after the m-th iterative optimization process is determined as the target image.

[0161] If it is determined that there are still unoptimized pixels in the mth iterative optimization area, determine the mth area to be optimized in the mth iterative optimization area. The area other than the mth area to be optimized in the mth iterative optimization area is the mth optimization area, which includes all the pixels that have been optimized in the previous m iterative optimization processes, and these pixels can be used as known information for subsequent optimization processes. Then, according to the above process of iteratively optimizing the m-1th area to be optimized, continue to iteratively optimize the mth area to be optimized, and repeat the operation until all pixels in the initial repair area are optimized, thereby obtaining the target image.

[0162] Figure 3 4 is a block diagram of a remote sensing image restoration device according to an embodiment of the present invention.

[0163] like Figure 3As shown, the remote sensing image restoration device 300 includes an acquisition module 310 , an identification module 320 , a determination module 330 and a restoration module 340 .

[0164] The acquisition module 310 is used to acquire the to-be-processed image and the candidate image set of the target area. The to-be-processed image is a remote sensing image containing the problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times.

[0165] The identification module 320 is used to identify information of non-problem areas other than the problem area in the image to be processed, so as to determine the imaging time of the image to be processed.

[0166] The determination module 330 is used to determine a candidate remote sensing image that matches the image to be processed from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, so as to obtain a reference image.

[0167] The restoration module 340 is used to restore the image to be processed by using the reference image to obtain a target image.

[0168] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0169] Figure 4 A block diagram of an electronic device suitable for implementing a remote sensing image restoration method according to an embodiment of the present invention is schematically shown.

[0170] like Figure 4 As shown, the electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0171] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0172] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the I / O interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0173] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the remote sensing image restoration method according to the embodiment of the present invention is implemented.

[0174] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0175] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the remote sensing image restoration method provided by the embodiment of the present invention.

[0176] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the processor 401 executes the computer program. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0177] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0178] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0179] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0182] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A remote sensing image restoration method, characterized in that: include: Acquire an image to be processed and a candidate image set of a target area, wherein the image to be processed is a remote sensing image containing the problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times; Performing information recognition on non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed; According to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, a candidate remote sensing image matching the image to be processed is determined from the candidate image set to obtain a reference image; Restoring the image to be processed by using the reference image to obtain a target image; The repairing of the image to be processed by using the reference image to obtain the target image comprises: performing multiple iterative repair processes on the problem area in the image to be processed according to the reference image to obtain an initial repair image including an initial repair area, wherein the initial repair area corresponds to the problem area; performing multiple iterative optimization processes on the initial repair area in the initial repair image to obtain the target image; The iterative repair process includes: obtaining an n-1th iteration area obtained after performing the n-1th iterative repair process on the problem area, the n-1th iteration area corresponds to the problem area, the n-1th iteration area includes the n-1th problem area, and n is an integer greater than 1; determining the boundary pixel points of the n-1th problem area to obtain a plurality of first target pixel points; determining the pixel points corresponding to the plurality of first target pixel points in the reference image to obtain a plurality of first reference pixel points; for each of the first reference pixel points, searching for a first similar pixel point associated with the first reference pixel point within a first neighborhood corresponding to the first reference pixel point; based on the searched pixel points associated with the plurality of first reference pixel points, The spectral mapping relationship between each of the first reference pixels and the corresponding first target pixels is determined based on the spectral values ​​of the multiple first similar pixels in the to-be-processed image and the reference image after the n-1th iterative restoration process and the first weights corresponding to each of the first similar pixels; based on the spectral mapping relationship, the spectral value of the corresponding first target pixel is filled according to the spectral value of each of the first reference pixels in the reference image to obtain the nth iteration area corresponding to the nth iteration restoration process; in response to determining that there are no unprocessed pixels in the nth iteration area, the to-be-processed image after the nth iteration restoration process is determined as the initial restoration image, and the nth iteration area corresponds to the initial restoration area in the initial restoration image.

2. The method according to claim 1, characterized in that: The step of identifying information of non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed includes: Using the recognition model to identify information in the non-problem area of ​​the image to be processed, and obtaining time information and ground object categories corresponding to multiple objects in the non-problem area; wherein the recognition model is obtained by training the initial recognition model using sample remote sensing images carrying time information labels and ground object category labels; In response to determining that the ground object categories corresponding to the plurality of objects respectively include a target ground object category, the time information of the target object corresponding to the target ground object category is determined as the imaging time of the image to be processed.

3. The method according to claim 2, characterized in that The step of identifying information of non-problem areas other than the problem area in the image to be processed to determine the imaging time of the image to be processed further includes: In response to determining that the target object category is not included in the object categories corresponding to the plurality of objects, determining the number of time information falling into the same time range in the time information corresponding to the plurality of objects; The time information with the largest number is used as the imaging time of the image to be processed.

4. The method according to claim 2, characterized in that: The recognition model is trained in the following way: Acquire a sample remote sensing image, wherein the sample remote sensing image includes a sample object and a time information label and a ground object category label of the sample object; Extracting features from the sample remote sensing image to obtain multiple scale feature maps; Performing multi-scale fusion processing on the multiple scale feature maps to obtain multiple multi-scale fusion feature maps, and performing fusion processing on the multiple multi-scale fusion feature maps to obtain a target fusion feature map; Determining sample time information and sample ground object category corresponding to the sample object in the sample remote sensing image according to the target fusion feature map; The parameters of the initial recognition model are adjusted according to the sample loss to obtain the recognition model; wherein the sample loss is determined according to the sample time information, the time information label, the sample feature category and the feature category label.

5. The method according to claim 4, characterized in that The initial recognition model includes a feature extraction module, a feature fusion module, a ground object recognition module and a time recognition module; the parameters of the initial recognition model are adjusted according to the sample loss to obtain the recognition model including: The parameters of the feature extraction module, the feature fusion module, the ground object recognition module and the time recognition module are adjusted according to the sample loss.

6. The method according to claim 1, characterized in that The step of determining a candidate remote sensing image matching the image to be processed from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, and obtaining a reference image comprises: Selecting a candidate remote sensing image whose imaging time is consistent with the imaging time of the image to be processed from the candidate image set; The selected candidate remote sensing images are used as the reference images.

7. The method according to claim 6, characterized in that The step of determining a candidate remote sensing image matching the image to be processed from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, and obtaining the reference image further comprises: In response to not screening out a candidate remote sensing image whose imaging time is consistent with the imaging time of the image to be processed from the candidate image set, selecting a candidate remote sensing image whose imaging time is the shortest between that of the image to be processed and the image to be processed from the candidate image set as the reference image.

8. The method according to claim 1, characterized in that: The method further comprises: In response to determining that there are unprocessed pixels in the nth iteration area, determining an nth problem area in the nth iteration area; The n-1th problem area is updated based on the nth problem area, and the operation of determining whether the image to be processed after the nth iterative restoration process is the initial restoration image is repeatedly performed.

9. The method according to claim 1, characterized in that: The step of searching for a first similar pixel point associated with the first reference pixel point within a first neighborhood corresponding to the first reference pixel point comprises: Determine a plurality of first neighboring pixel points associated with the first reference pixel point within the first neighborhood; Determining a first spectral deviation value between the first reference pixel and each of the first adjacent pixels; In response to determining that the first spectral deviation value is less than a first spectral deviation threshold, a first neighboring pixel point corresponding to the first spectral deviation value is determined as the first similar pixel point.

10. The method according to claim 9, characterized in that The method further comprises: Determine a first sub-spectrum deviation threshold corresponding to the band according to the spectral values ​​of all pixels in the first neighborhood in the reference image in each band; The first spectrum deviation value is determined according to the first sub-spectrum deviation threshold corresponding to each band.

11. The method according to claim 1, characterized in that: The determining of the spectral mapping relationship between each of the first reference pixels and the corresponding first target pixel based on the spectral values ​​of the first similar pixels respectively associated with the first reference pixels after the n-1th iteration restoration process in the to-be-processed image and the reference image and the first weight corresponding to each of the first similar pixels comprises: Determine a mapping parameter based on the spectral values ​​of each of the first similar pixels in the to-be-processed image and the reference image after the n-1th iterative restoration process, and the first weight corresponding to each of the first similar pixels; Based on the mapping parameters, the spectral mapping relationship is constructed.

12. The method according to claim 1, characterized in that The method further comprises: Determine first spatial distances between a plurality of first similar pixels associated with the first reference pixel and the first reference pixel respectively; For each of the first similar pixel points, a first weight corresponding to the first similar pixel point is determined according to the first spatial distance.

13. The method according to claim 12, characterized in that The first spatial distance is negatively correlated with the first weight corresponding to the first similar pixel point.

14. The method according to claim 1, characterized in that The iterative optimization process includes: Obtaining an m-1th iterative optimization region obtained after performing an m-1th iterative optimization process on the initial repair region, wherein the m-1th iterative optimization region corresponds to the initial repair region, and the m-1th iterative optimization region includes the m-1th region to be optimized, where m is an integer greater than 1; Determine the boundary pixel points of the m-1th area to be optimized to obtain a plurality of second target pixel points; For each of the second target pixel points, search for a second similar pixel point associated with the second target pixel point within a second neighborhood range corresponding to the second target pixel point, and determine a target spectral value corresponding to the second target pixel point based on spectral values ​​corresponding to each of the plurality of second similar pixel points searched out in the initial restored image after the m-1th iterative optimization process and second weights corresponding to each of the plurality of second similar pixel points; Optimizing the spectral value corresponding to each second target pixel in the initial restoration image after the m-1th iterative optimization process according to the target spectral value corresponding to each second target pixel, to obtain the mth iterative optimization area corresponding to the mth optimization process; In response to determining that there are no unoptimized pixels in the m-th iterative optimization area, an initial restoration image after the m-th iterative optimization process is determined as the target image.

15. The method according to claim 14, characterized in that The method further comprises: In response to determining that there are unoptimized pixels in the m-th iterative optimization region, determining an m-th region to be optimized in the m-th iterative optimization region; The m-1th region to be optimized is updated based on the mth region to be optimized, and the operation of determining whether the initial restoration image after the mth iterative optimization process is the target image is repeatedly performed.

16. The method according to claim 14, characterized in that The step of searching for a second similar pixel point associated with the second target pixel point within a second neighborhood range corresponding to the second target pixel point comprises: Determine a plurality of second neighboring pixel points associated with the second target pixel point within the second neighborhood; Determining a second spectral deviation value between the second target pixel and each of the second adjacent pixels; In response to determining that the second spectral deviation value is less than a second spectral deviation threshold, a second neighboring pixel point corresponding to the second spectral deviation value is determined as the second similar pixel point.

17. The method according to claim 16, characterized in that The method further comprises: Determine a second sub-spectrum deviation threshold corresponding to the band according to the spectral values ​​of all pixels in the second neighborhood in each band in the initial restored image after the m-1th iterative optimization process; The second spectrum deviation threshold is determined according to the second sub-spectrum deviation threshold corresponding to each band.

18. The method according to claim 14, characterized in that The method further comprises: Determine second spatial distances between a plurality of second similar pixels associated with the second target pixel and the second target pixel respectively; For each of the second similar pixel points, a second weight corresponding to the second similar pixel point is determined according to the second spatial distance.

19. The method according to claim 18, characterized in that The second spatial distance is negatively correlated with the second weight corresponding to the second similar pixel point.

20. The method according to any one of claims 1 to 7, characterized in that The problem area includes one of a noise area, a null area, a thick cloud area, a thin cloud area, a fog area, a shadow area and a blurred area.

21. A remote sensing image restoration device, characterized in that: include: An acquisition module, used to acquire an image to be processed and a candidate image set of a target area, wherein the image to be processed is a remote sensing image containing the problem area, and the candidate image set includes candidate remote sensing images acquired at multiple imaging times; An identification module, used for identifying information of non-problem areas other than the problem area in the image to be processed, so as to determine the imaging time of the image to be processed; A determination module, configured to determine a candidate remote sensing image matching the image to be processed from the candidate image set according to the imaging time of the image to be processed and the imaging time of each candidate remote sensing image in the candidate image set, so as to obtain a reference image; A restoration module, used to restore the image to be processed by using the reference image to obtain a target image; The repair module is further used to: perform multiple iterations of repair processing on the problem area in the image to be processed according to the reference image to obtain an initial repair image including an initial repair area, wherein the initial repair area corresponds to the problem area; Performing multiple iteration optimization processes on the initial restoration area in the initial restoration image to obtain the target image; The iterative repair process includes: obtaining an n-1th iteration area obtained after performing the n-1th iterative repair process on the problem area, the n-1th iteration area corresponds to the problem area, the n-1th iteration area includes the n-1th problem area, and n is an integer greater than 1; determining the boundary pixel points of the n-1th problem area to obtain a plurality of first target pixel points; determining the pixel points corresponding to the plurality of first target pixel points in the reference image to obtain a plurality of first reference pixel points; for each of the first reference pixel points, searching for a first similar pixel point associated with the first reference pixel point within a first neighborhood corresponding to the first reference pixel point; based on the searched pixel points associated with the plurality of first reference pixel points, The spectral mapping relationship between each of the first reference pixels and the corresponding first target pixels is determined based on the spectral values ​​of the multiple first similar pixels in the to-be-processed image and the reference image after the n-1th iterative restoration process and the first weights corresponding to each of the first similar pixels; based on the spectral mapping relationship, the spectral value of the corresponding first target pixel is filled according to the spectral value of each of the first reference pixels in the reference image to obtain the nth iteration area corresponding to the nth iteration restoration process; in response to determining that there are no unprocessed pixels in the nth iteration area, the to-be-processed image after the nth iteration restoration process is determined as the initial restoration image, and the nth iteration area corresponds to the initial restoration area in the initial restoration image.

22. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 20.

24. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 20.

Images