Remote sensing image processing method, device, equipment and medium

Abstract

The present disclosure provides a remote sensing image processing method, which can be applied to the field of artificial intelligence. The remote sensing image processing model includes an image update module and a matrix update module, and the method includes: obtaining a cloud image to be declouded and a cloud mask matrix of the cloud image; inputting the cloud image and the cloud mask matrix into the image update module, eliminating clouds in a preset area of ​​the cloud image based on the cloud mask matrix, and outputting an updated cloud image; inputting the updated cloud image into the matrix update module, updating the cloud mask matrix, and outputting an updated cloud mask matrix; according to the updated cloud image and the updated cloud mask matrix, obtaining a cloud-removed image. The present disclosure also provides a remote sensing image processing device, equipment, storage medium, and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to the field of remote sensing image processing, and in particular to a remote sensing image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Currently, remote sensing technology can be used to obtain remote sensing images with high spatial resolution and high spectral resolution. However, remote sensing images are very susceptible to climate factors. Cloud occlusion is one of the main reasons for the loss of ground information. The ground information collected by remote sensing images will be degraded to varying degrees due to sparse or completely blurred cloud cover. Thick clouds in cloud occlusion generally appear as clouds. In remote sensing images, thick clouds are not only difficult to remove, but also the information blocked by thick clouds is completely invalid.

[0003] In the prior art, algorithms for removing thick clouds from remote sensing images include traditional cloud removal methods and high-resolution remote sensing image thick cloud removal methods based on deep learning. Among them, traditional cloud removal methods, such as those based on homomorphic filtering and those based on multispectral information, are more effective for thin clouds or small cloud areas, but have certain limitations in removing thick clouds; for high-resolution remote sensing image thick cloud removal methods based on deep learning, when using standard convolution to traverse the entire network input remote sensing image with clouds, its output is calculated based on valid pixels and masked pixels. Since masked pixels belong to interference information, the output image will produce visual artifacts such as distortion, blur, and color distortion. In addition, when using a mask (MASK) to cover the interference information, as the number of network layers increases, the network will ignore the valid information generated by the cloud boundary, which will then affect the final loss.

[0004] The existing technology has the problem that it is unable to eliminate various image distortion information that occurs in the process of restoring the information covered by thick clouds for remote sensing images, and is unable to accurately and effectively restore the covered surface information. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a remote sensing image processing method, device, equipment, medium and program product for processing clouds in remote sensing images, eliminating various artifacts that may appear when the U-Net network is used as a reconstruction network, and restoring the obscured surface information.

[0006] According to a first aspect of the present disclosure, a remote sensing image processing method is provided, wherein the remote sensing image processing model includes an image updating module and a matrix updating module, and the method includes: obtaining a cloud image to be declouded and a cloud mask matrix of the cloud image; inputting the cloud image and the cloud mask matrix into the image updating module, eliminating clouds in a preset area of ​​the cloud image based on the cloud mask matrix, and outputting an updated cloud image; inputting the updated cloud image into the matrix updating module, updating the cloud mask matrix, and outputting an updated cloud mask matrix; and obtaining a cloud-removed image according to the updated cloud image and the updated cloud mask matrix.

[0007] According to an embodiment of the present disclosure, a cloud image and a cloud mask matrix are input into an image update module, and clouds in a preset area of ​​the cloud image are eliminated based on the cloud mask matrix. The updated cloud image is output, including: determining first feature information based on the cloud image and the cloud mask matrix; adjusting the weight of a convolution kernel for feature extraction of the cloud image based on the first feature information to determine second feature information; and adjusting the amount of information of the second feature information using a scaling factor to output an updated cloud image.

[0008] According to an embodiment of the present disclosure, an updated cloud image is input into a matrix update module, the cloud mask matrix is ​​updated, and the updated cloud mask matrix is ​​output, including: determining a first area to be convolved in the cloud mask matrix; determining a second area corresponding to the first area in the updated cloud image; when there is a part in the second area that is not covered by clouds, updating the value corresponding to the center position of the mask convolution kernel to 1 to obtain a first area mask convolution kernel; convolving the first area with the first area mask convolution kernel to update the first area of ​​the cloud mask matrix; and when the convolution operation is completed in all areas of the cloud mask matrix, outputting the updated cloud mask matrix.

[0009] According to an embodiment of the present disclosure, obtaining a cloud-removed image based on an updated cloud image and an updated cloud mask matrix includes: performing E times of feature extraction on the updated cloud image and the updated cloud mask matrix using an image update module until all information covered by clouds in the cloud image is restored, and outputting the cloud-removed image, wherein a cloud image restoring preset area information is generated after each feature extraction, and E is an integer greater than or equal to 2; and wherein, after obtaining the cloud image restoring the preset area information, before performing the next feature extraction, using a matrix update module to update the updated cloud image to obtain a cloud mask matrix corresponding to the updated cloud image.

[0010] According to an embodiment of the present disclosure, the cloud image includes a remote sensing image of N bands, and the input of the remote sensing image processing model is the remote sensing image of N bands and a cloud mask matrix, wherein N is an integer greater than or equal to 2.

[0011] According to an embodiment of the present disclosure, the training sample set includes multiple data pairs, each data pair includes a cloud training image, a cloud-free training image and a cloud mask training matrix, and determining the remote sensing image processing model includes: rotating the cloud training image and the cloud mask training matrix according to a preset angle to obtain expanded cloud training data and a corresponding expanded cloud mask training matrix; and using the expanded cloud training data and the corresponding expanded cloud mask training matrix as inputs of the remote sensing image processing model to be trained to obtain the remote sensing image processing model.

[0012] According to an embodiment of the present disclosure, obtaining a cloud image to be de-clouded and a cloud mask matrix of the cloud image includes: based on the registered cloud image, segmenting the registered cloud image according to a preset step size to obtain the cloud image to be de-clouded, the cloud image to be de-clouded includes multiple image blocks of a preset step size; and determining a cloud mask matrix corresponding to the cloud image according to the cloud image to be de-clouded, wherein the clouded area in the cloud mask matrix is ​​marked as 0, and other areas are marked as 1.

[0013] According to a second aspect of the present disclosure, a remote sensing image processing device is provided, and the remote sensing image processing model includes an image updating module and a matrix updating module, and the device includes: an acquisition module, used to acquire a cloud image to be declouded and a cloud mask matrix of the cloud image; a first output module, used to input the cloud image and the cloud mask matrix into the image updating module, eliminate clouds in a preset area of ​​the cloud image based on the cloud mask matrix, and output an updated cloud image; a second output module, used to input the updated cloud image into the matrix updating module, update the cloud mask matrix, and output an updated cloud mask matrix; an elimination module, used to obtain a cloud-eliminated image based on the updated cloud image and the updated cloud mask matrix.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the above-mentioned remote sensing image processing method.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored, and when the instructions are executed by a processor, the processor implements the above-mentioned remote sensing image processing method.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product includes computer executable instructions, and the instructions are used to implement the above remote sensing image processing method when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The system architecture of the remote sensing image processing method according to the embodiment of the present disclosure is schematically shown;

[0019] Figure 2 The flowchart of the remote sensing image processing method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A flowchart of a method for outputting an updated cloud image according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 4 A flowchart of a method for outputting an updated cloud layer mask matrix according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 5 A flowchart of a method for obtaining a cloud-removed image according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 6 A remote sensing image processing model according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 7 A flowchart of a method for determining a remote sensing image processing model according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 8 A flowchart of a method for obtaining a cloud image to be declouded and a cloud mask matrix according to an embodiment of the present disclosure is schematically shown;

[0026] Fig. 9 A schematic diagram of a remote sensing image processing device according to an embodiment of the present disclosure is shown; and

[0027] Fig.10 A block diagram of an electronic device suitable for a remote sensing image processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] An embodiment of the present disclosure provides a training method for a remote sensing image processing model, comprising: determining a training sample set based on original cloud image data, the training sample set comprising multiple data pairs, each data pair comprising a cloud image, a cloud-free image and a cloud mask matrix; determining an updated cloud image based on the cloud image and the cloud mask matrix; updating the cloud mask matrix based on the updated cloud image to obtain an updated cloud mask matrix; obtaining a cloud-removed image based on the updated cloud image and the updated cloud mask matrix; and adjusting parameters of the remote sensing image processing model based on the cloud-removed image and the cloud-free image to obtain a trained remote sensing image processing model.

[0033] Figure 1 The system architecture of the remote sensing image processing method according to the embodiment of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be installed with various communication client applications, such as image processing applications, video processing applications, image editing applications, etc. (only examples).

[0036] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0037] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the training method of the abnormal behavior classification model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the abnormal behavior classification model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method of the abnormal behavior classification model provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the training device of the abnormal behavior classification model provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] The following will be based on Figure 1 The system architecture described by Figure 2 to Figure 7 The training method of the remote sensing image processing model of the disclosed embodiment is described in detail.

[0041] Figure 2 The flowchart of the remote sensing image processing method according to the embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the method includes operations S201 to S204.

[0043] In operation S201, a cloud image to be declouded and a cloud layer mask matrix of the cloud image are obtained.

[0044] According to the embodiments of the present disclosure, since the remote sensing images taken by remote sensing satellites are affected by clouds, there will be parts covered by clouds on the taken remote sensing images. After obtaining the cloud image to be de-clouded, the cloud image to be de-clouded is processed to obtain the corresponding cloud layer data in the cloud image, and the cloud layer mask matrix is ​​determined according to the corresponding cloud layer data.

[0045] According to an embodiment of the present disclosure, a cloud image includes parts covered by clouds and parts not covered by clouds. The cloud mask matrix corresponding to the cloud image can represent the initial cloud data in the cloud image. The cloud mask matrix can be used to restore some information at the connection between the cloud covered part and the uncovered part.

[0046] In operation S202, the cloud image and the cloud layer mask matrix are input into an image updating module, clouds in a preset area of ​​the cloud image are removed based on the cloud layer mask matrix, and an updated cloud image is output.

[0047] According to an embodiment of the present disclosure, after the cloud image and the cloud mask matrix are input into the image update module, the image update module can reduce the clouds in the cloud image from the outside to the inside according to the input cloud mask matrix, and output an updated cloud image. Specifically, the connection between the cloud-covered part and the uncovered part contains some implicit information. When the cloud image is feature extracted using the cloud mask matrix, the area of ​​the cloud-covered part in the cloud image can be reduced, and part of the information covered by the cloud can be restored. Since the connection between the cloud-covered part and the uncovered part is irregular, the area restored each time is a partial area of ​​the connection, so that the cloud layer shows a decreasing trend from the outside to the inside.

[0048] In operation S203, the updated cloud image is input into a matrix update module, the cloud mask matrix is ​​updated, and an updated cloud mask matrix is ​​output.

[0049] According to an embodiment of the present disclosure, since the cloud cover area in the updated cloud image has changed, the cloud information contained in the cloud mask matrix at this time is different from the cloud contained in the updated cloud image, and the updated cloud image needs to be input into the matrix update module, and the cloud mask matrix is ​​updated using the updated cloud image, and the updated cloud mask matrix is ​​output. The updated cloud mask matrix corresponds to the cloud information contained in the updated cloud image.

[0050] According to an embodiment of the present disclosure, the matrix updating module is capable of storing each updated cloud mask matrix so as to realize the next update of the cloud mask matrix by utilizing the input updated cloud image.

[0051] In operation S204, a cloud-removed image is obtained according to the updated cloud image and the updated cloud mask matrix.

[0052] According to the embodiments of the present disclosure, the updated cloud image restores part of the information covered by the clouds, but cannot restore all the information covered by the clouds. Therefore, it is necessary to continue to perform multiple feature extractions on the updated cloud image and the updated cloud mask matrix until all the information covered by the clouds in the cloud image is restored to obtain an image with the clouds removed.

[0053] According to an embodiment of the present disclosure, the remote sensing image processing model is a model pre-trained using remote sensing image data included in an open source database.

[0054] The present invention reduces the influence of interference information on the output results of the remote sensing image processing model, eliminates various visual artifacts, and improves the reconstruction accuracy of the network by extracting features from cloud images and cloud mask matrices. By automatically updating the cloud mask matrix, the influence of losses caused by failure to update the cloud mask matrix is ​​avoided, the processing accuracy of the remote sensing image processing model is improved, and the removal of thick clouds in remote sensing images is achieved.

[0055] According to the embodiments of the present disclosure, the remote sensing image processing method can be applied to the field of remote sensing image processing to determine accurate cloud removal images and improve the accuracy of remote sensing image processing. In addition, the remote sensing image processing method can also be applied to the financial field, which is beneficial for financial institutions and enterprises to achieve accurate asset evaluation and reduce asset evaluation losses caused by image processing distortion.

[0056] For example, with respect to agriculture, when evaluating the degree of vegetation greening, the remote sensing image processing method disclosed in the present invention can be used to obtain accurate images with clouds removed, thereby achieving an accurate evaluation of the degree of vegetation greening; with respect to financial evaluation, when an asset institution evaluates a physical construction project, the asset institution can determine the accurate progress of the physical construction project based on the accurate image with clouds removed, thereby evaluating the risk situation of the physical project.

[0057] For example, during the construction of building A, due to a shortage of funds, a loan needs to be obtained from bank B. Bank B needs to evaluate the location information, construction progress and other information of building A. When evaluating building A, multiple remote sensing images of building A are used.

[0058] Due to the cloud cover in the sky, part of the information of Building A is blocked by the clouds. At this time, the cloud image obtained cannot show the specific information of the covered part of Building A, and Bank B cannot make an accurate assessment of Building A based on the cloud remote sensing image. In addition, when using the existing cloud removal technology to process the remote sensing image covered by thick clouds, it is impossible to eliminate various image distortion information that occurs in the process of restoring the information covered by thick clouds, and it is still impossible to accurately and effectively restore the covered surface information. This will cause Bank B to deviate from the assessment results of the project, resulting in financial losses.

[0059] The cloud-removed image obtained by utilizing the remote sensing image processing method provided by the present invention can help banks or other asset institutions obtain accurate information on the cloud-removed image, thereby facilitating banks or other asset institutions to conduct accurate asset assessments and reduce the risk of capital loss.

[0060] Figure 3 The flowchart of the method for outputting an updated cloud image according to an embodiment of the present disclosure is schematically shown.

[0061] like Figure 3 As shown, the method includes operations S301 to S303.

[0062] In operation S301, first feature information is determined according to a cloud image and a cloud layer mask matrix.

[0063] According to an embodiment of the present disclosure, based on a cloud image and a cloud mask matrix input to a remote sensing image processing model, a dot product is performed between the cloud image and the cloud mask matrix to obtain a first feature information matrix for subsequent feature extraction.

[0064] In operation S302, the weight of a convolution kernel used for extracting features from a cloud image is adjusted based on the first feature information to determine second feature information.

[0065] According to an embodiment of the present disclosure, a convolution kernel is used to extract features from the first feature information. However, since there are differences in the extracted feature information, the weight of the convolution kernel is adjusted to process the first feature information to obtain the second feature information.

[0066] In operation S303, the information amount of the second feature information is adjusted using the scaling factor, and an updated cloud image is output.

[0067] According to an embodiment of the present disclosure, since the first feature information is obtained based on the cloud image and the cloud mask matrix, as the cloud mask matrix is ​​updated, the number of known elements in the area corresponding to the convolution kernel increases, and the input amount of different numbers of effective values ​​can be adjusted through the scaling factor to obtain an updated cloud image.

[0068] According to an embodiment of the present disclosure, feature extraction of an input cloud image and a cloud mask matrix may satisfy the following formula:

[0069]

[0070] Where W represents the weight of the convolution kernel, b represents the corresponding bias, X represents the input of the current convolution sliding window, x' represents the output after feature extraction of the current input, and M represents the corresponding binary cloud mask matrix. Represents the scaling factor.

[0071] Figure 4 A flowchart of a method for outputting an updated cloud layer mask matrix according to an embodiment of the present disclosure is schematically shown.

[0072] like Figure 4 As shown, the method includes operations S401 to S405.

[0073] In operation S401, a first region to be convolved in a cloud layer mask matrix is ​​determined.

[0074] According to an embodiment of the present disclosure, after feature extraction is performed on a cloud image and a cloud mask matrix, the cloud mask matrix needs to be updated to match the updated cloud information to the cloud mask matrix. Each update will reduce the area of ​​the cloud-covered portion, so it is necessary to determine the first area to be convolved in the unupdated cloud mask matrix. In practical applications, convolution of the portion of the cloud mask matrix that does not contain cloud information cannot extract information, so the first area to be convolved generally refers to the portion containing cloud information.

[0075] In operation S402 , a second area corresponding to the first area is determined in the updated cloud image.

[0076] According to an embodiment of the present disclosure, the cloud layer information contained in the updated cloud image will also be updated accordingly. After determining the first area to be convolved in the cloud mask matrix, a second area corresponding to the first area is determined in the updated cloud image, so as to determine the updated cloud layer information based on the updated cloud image.

[0077] In operation S403, when there is a portion in the second area that is not covered by clouds, the value corresponding to the center position of the mask convolution kernel is updated to 1 to obtain the mask convolution kernel of the first area.

[0078] According to an embodiment of the present disclosure, the existence of a portion of the second area that is not covered by clouds indicates that there is uncovered surface information in the current second area. At this time, the value corresponding to the center position of the mask convolution kernel used to update the cloud mask matrix is ​​updated to 1 to obtain the mask convolution kernel of the first area.

[0079] In operation S404, the first region is convolved using the first region mask convolution kernel to update the first region of the cloud layer mask matrix.

[0080] According to the embodiment of the present disclosure, since the center position of the first region mask convolution kernel is 1, when the first region mask convolution kernel is used to convolve the first region of the cloud mask matrix, the first region of the cloud mask matrix will be updated to a matrix containing surface information. As part of the forward transfer, if the updated information is applied, the cloud mask matrix will eventually become a matrix of all 1s, thereby completing the purpose of missing information completion.

[0081] In operation S405, when the convolution operation is completed for all regions of the cloud mask matrix, an updated cloud mask matrix is ​​output.

[0082] According to an embodiment of the present disclosure, after all first regions to be convolved of the cloud mask matrix are updated according to the updated cloud image, a cloud mask matrix containing updated cloud information can be obtained.

[0083] According to an embodiment of the present disclosure, updating the cloud mask matrix may satisfy the formula:

[0084]

[0085] Among them, M represents the corresponding binary cloud mask matrix, and m' represents the output after convolution of the current input.

[0086] Figure 5 The flowchart of the method for obtaining a cloud-removed image according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 5 As shown, the method includes operations S501-S502.

[0088] In operation S501, an image updating module is used to perform E times of feature extraction on the updated cloud image and the updated cloud mask matrix until all information covered by clouds in the cloud image is restored, and a cloud-removed image is output.

[0089] According to the embodiments of the present disclosure, since the information recovered by feature extraction of the updated cloud image and the updated cloud mask matrix is ​​limited, each feature extraction will generate a cloud image that restores the preset area information, so it is necessary to use the image update module to perform feature extraction E times until all the information covered by the clouds in the cloud image is restored and the cloud-removed image is output, where E is an integer greater than or equal to 2. For example, E is 5, and the input cloud image and cloud mask matrix are subjected to feature extraction 5 times, and the cloud-removed image is output.

[0090] In operation S502, after obtaining the cloud image that restores the specific preset area information, before performing the next feature extraction, the updated cloud image is updated using a matrix updating module to obtain a cloud mask matrix corresponding to the updated cloud image.

[0091] According to an embodiment of the present disclosure, the updated cloud image is a cloud image in which the preset area information has been restored. When the cloud image is updated next time, the information of the restored area is required, so the cloud mask matrix needs to be updated next time. The cloud mask matrix can be updated according to the updated cloud image so that the updated cloud mask matrix can be used to update the cloud image next time, that is, the updates of the cloud image and the cloud mask matrix are performed alternately.

[0092] According to an embodiment of the present disclosure, the training set for training the remote sensing image processing model comes from the Landsat8 remote sensing satellite, and each cloud image includes remote sensing images of N bands, so the input of the remote sensing image model includes remote sensing images of N bands and the cloud mask matrix, where N is an integer greater than or equal to 2. For example, the cloud image includes remote sensing images of 9 bands, and the input of the remote sensing image processing model is the remote sensing images of 9 bands and the cloud mask matrix.

[0093] Figure 6 A remote sensing image processing model according to an embodiment of the present disclosure is schematically shown.

[0094] According to the embodiments of the present disclosure, Figure 6 As shown in the figure, a U-Net network based on adaptive partial convolution is built using the Python+Pytorch framework. The input of the network is: the input of the network is N+1, the first N channels are remote sensing images, and the N+1th channel is the corresponding cloud mask matrix. U-Net has five layers, among which all convolution operations in the encoder are completed by adaptive partial convolution layers, and deconvolution is used for upsampling in the decoding stage. In order to achieve a stylistic fusion and approximation between the decoder and the encoder, the feature data extracted by the corresponding encoder and the cloud mask matrix are linked through jump links.

[0095] Figure 7 The flowchart of the method for determining a remote sensing image processing model according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 7 As shown, the method includes operations S701-S702.

[0097] In operation S701, the cloud training image and the cloud mask training matrix are rotated according to a preset angle to obtain augmented cloud training data and a corresponding augmented cloud mask training matrix.

[0098] According to an embodiment of the present disclosure, the training sample set includes a plurality of data pairs, each data pair including a cloud training image, a cloud-free training image, and a cloud mask training matrix.

[0099] According to an embodiment of the present disclosure, the training sample set is obtained from the open source dataset RICE. The open source dataset RICE includes a thin cloud dataset RICE1 and a thick cloud dataset RICE2, wherein the thick cloud dataset RICE2 comes from the Landsat8 remote sensing satellite. The original cloud layer image data obtained from the open source database contains 450 pairs of images, each pair of images includes cloud image data, cloud-free image data and cloud layer mask data.

[0100] According to an embodiment of the present disclosure, the cloud image data, cloud-free image data and cloud mask data of the original cloud image data are processed to obtain the corresponding cloud training images, cloud-free training images and cloud mask training matrices in the training sample set. For example, data expansion and registration processing are performed on the cloud image data and cloud-free image data; and the cloud mask data is binarized to obtain a binary cloud mask matrix.

[0101] According to an embodiment of the present disclosure, the remote sensing image processing model adopts a U-Net network, including an encoder and a decoder, and each convolution layer of the encoder is connected to the next convolution layer of the encoder, and is also connected to the corresponding convolution layer in the decoder.

[0102] According to an embodiment of the present disclosure, the remote sensing image processing model uses the OpenCV library of python to rotate the cloud training image and the cloud mask training matrix at a preset angle to obtain the expanded cloud training data and the corresponding expanded cloud mask training matrix. Specifically, the preset angles include 0°, 90°, and 180°, and the cloud training image and the cloud mask training matrix are randomly rotated at preset angles. For example, the cloud training image and the cloud mask training matrix are first rotated 90°, then 180°, and then rotated 90° in the other direction. During the rotation process, the rotation angles are random.

[0103] In operation S702, the augmented cloud training data and the corresponding augmented cloud mask training matrix are used as inputs of the remote sensing image processing model to be trained to obtain the remote sensing image processing model.

[0104] According to an embodiment of the present disclosure, after rotating the cloud training image and the cloud mask training matrix, the number of samples in the training sample set can be expanded, and then the expanded cloud training data and the corresponding expanded cloud mask training matrix are used as inputs of the remote sensing image processing model to be trained to train the remote sensing image processing model.

[0105] According to an embodiment of the present disclosure, a loss function of a remote sensing image processing model to be trained is obtained based on a cloud-removed image and a corresponding cloud-free training image obtained during a training process. Parameters of the remote sensing image processing model to be trained are adjusted based on the loss function, and a trained remote sensing image processing model is obtained when preset conditions are met.

[0106] According to the embodiment of the present disclosure, the distribution of the initial parameters of the network for training the remote sensing image processing model obeys a normal distribution with a mean of 0 and a standard deviation of 0.05, and Relu is the activation function between layers. The Adam algorithm is used for optimization, and the initial learning rate is set to 0.0001, with a total of 500,000 iterations, and the learning rate is changed to the original 0.8 every 20,000 iterations. The loss function of the remote sensing image processing model includes the loss of the effective area, the loss of the cloud occlusion area, and the regular constraint loss. The specific loss function satisfies:

[0107] L=L valid +pL hole +qL tv (3)

[0108] Among them, L is the loss function of the entire network, L valid is the loss function of the effective area, L hole is the loss function of the cloud-occluded area, L tv is the loss function of the regular constraint term, which is used to maintain the smoothness of the image, p is the training parameter of the loss function of the cloud occlusion area, and q is the training parameter of the loss function of the regular constraint term.

[0109] According to an embodiment of the present disclosure, after training, p is 5 and q is 0.1 in the loss function.

[0110] The loss function of the effective area satisfies:

[0111]

[0112] Where M represents the corresponding binary cloud mask matrix, f(w,b,x k ) represents the output result of the kth input of the model when the weight of the convolution kernel is W and the bias is b, y k represents the theoretical result corresponding to the kth input, and N represents a total of N inputs.

[0113] The loss function of the cloud-occluded area satisfies:

[0114]

[0115] Where M represents the corresponding binary cloud mask matrix, f(W,b,x k) represents the output result of the kth input of the model when the weight of the convolution kernel is W and the bias is b, y k represents the theoretical result corresponding to the kth input, and N represents a total of N inputs.

[0116] The loss function of the regular constraint term satisfies:

[0117]

[0118] Among them, x i，j Represents the input of the i-th pixel in the horizontal direction and the j-th pixel in the vertical direction.

[0119] Figure 8 The flowchart of the method for obtaining a cloud image to be declouded and a cloud mask matrix according to an embodiment of the present disclosure is schematically shown.

[0120] like Figure 8 As shown, the method includes operations S801-S802.

[0121] In operation S801, based on the registered cloud image, the registered cloud image is segmented according to a preset step length to obtain a cloud image to be de-clouded, and the cloud image to be de-clouded includes a plurality of image blocks with a preset step length.

[0122] According to the embodiments of the present disclosure, after obtaining the original cloud layer image data, the cloud image and the cloud-free image are registered using professional remote sensing image software. After the registration, the registered cloud image is segmented according to a fixed step size into a collection of image blocks of a fixed size to obtain the cloud image to be declouded.

[0123] In operation S802, a cloud mask matrix corresponding to the cloud image is determined according to the cloud image to be declouded, wherein the cloud area in the cloud mask matrix is ​​marked as 0, and other areas are marked as 1.

[0124] According to an embodiment of the present disclosure, the cloud mask is binarized based on the cloudy image to be declouded, the position corresponding to the cloudy area is marked as 0, and the other parts not covered by the clouds are marked as 1, so as to obtain a binary cloud mask matrix containing only 0 and 1.

[0125] According to an embodiment of the present disclosure, after registering the cloud image and the cloud-free image, the registered images are divided into a data set, a validation set, and a test set according to the ratio of 70%, 20%, and 10%, respectively. The training sample set includes a data set and a validation set, which are used to train the remote sensing image processing model, and the test set is used to test the trained remote sensing image processing model.

[0126] Fig. 9The structural block diagram of the remote sensing image processing device according to an embodiment of the present disclosure is schematically shown.

[0127] like Fig. 9 As shown, the remote sensing image processing model training device 900 of this embodiment includes an acquisition module 901 , a first output module 902 , a second output module 903 and an elimination module 904 .

[0128] The acquisition module 901 is used to acquire the cloud image to be declouded and the cloud mask matrix of the cloud image. In one embodiment, the acquisition module 901 can be used to perform the operation S201 described above, which will not be described in detail here.

[0129] The first output module 902 is used to input the cloud image and the cloud mask matrix into the image update module, eliminate the cloud layer in the preset area of ​​the cloud image based on the cloud mask matrix, and output an updated cloud image. In one embodiment, the first output module 902 can be used to perform the operation S202 described above, which will not be repeated here.

[0130] The second output module 903 is used to input the updated cloud image into the matrix update module, update the cloud mask matrix, and output the updated cloud mask matrix. In one embodiment, the second output module 903 can be used to perform the operation S203 described above, which will not be described in detail here.

[0131] The elimination module 904 is used to obtain a cloud-removed image according to the updated cloud image and the updated cloud mask matrix. In one embodiment, the elimination module 904 can be used to perform the operation S204 described above, which will not be described in detail here.

[0132] According to an embodiment of the present disclosure, the first output module 902 includes a first determining unit, a second determining unit and a third determining unit.

[0133] The first determination unit is used to determine the first feature information according to the cloud image and the cloud mask matrix. In one embodiment, the first determination unit can be used to perform the operation S301 described above, which will not be described in detail here.

[0134] The second determination unit is used to adjust the weight of the convolution kernel used for extracting features from the cloud image based on the first feature information to determine the second feature information. In one embodiment, the second determination unit can be used to perform the operation S302 described above, which will not be described in detail here.

[0135] The third determination unit is used to adjust the information amount of the second feature information by using the scaling factor, and output an updated cloud image. In one embodiment, the third determination unit can be used to perform the operation S303 described above, which will not be described in detail here.

[0136] According to an embodiment of the present disclosure, the second output module 903 includes a first matrix determination unit, a second matrix determination unit, a third matrix determination unit, a fourth matrix determination unit and a fifth matrix determination unit.

[0137] The first matrix determination unit is used to determine a first region to be convolved in the cloud layer mask matrix. In one embodiment, the first matrix determination unit may be used to perform the operation S401 described above, which will not be described in detail herein.

[0138] The second matrix determination unit is used to determine the second area corresponding to the first area in the updated cloud image. In one embodiment, the second matrix determination unit can be used to perform the operation S402 described above, which will not be described in detail here.

[0139] The third matrix determination unit is used to update the value corresponding to the center position of the mask convolution kernel to 1 when there is a part of the second area that is not covered by clouds, so as to obtain the mask convolution kernel of the first area. In one embodiment, the third matrix determination unit can be used to perform the operation S403 described above, which will not be repeated here.

[0140] The fourth matrix determination unit is used to convolve the first region using the first region mask convolution kernel to update the first region of the cloud layer mask matrix. In one embodiment, the fourth matrix determination unit can be used to perform the operation S404 described above, which will not be described in detail here.

[0141] The fifth matrix determination unit is used to output an updated cloud mask matrix when the convolution operation is completed for all regions of the cloud mask matrix. In one embodiment, the fifth matrix determination unit may be used to perform the operation S405 described above, which will not be described in detail herein.

[0142] According to an embodiment of the present disclosure, the elimination module 904 includes a feature extraction unit and an updating unit.

[0143] The feature extraction unit is used to perform E feature extractions on the updated cloud image and the updated cloud mask matrix using the image update module until all information covered by clouds in the cloud image is restored, and the cloud-removed image is output, wherein a cloud image restoring the preset area information is generated after each feature extraction, and E is an integer greater than or equal to 2. In one embodiment, the feature extraction unit can be used to perform the operation S501 described above, which will not be repeated here.

[0144] The updating unit is used to update the updated cloud image using the matrix updating module after obtaining the cloud image that restores the preset area information and before performing the next feature extraction to obtain the cloud layer mask matrix corresponding to the updated cloud image. In one embodiment, the updating unit can be used to perform the operation S502 described above, which will not be repeated here.

[0145] According to an embodiment of the present disclosure, the remote sensing image processing device further includes a preprocessing module, and the preprocessing module includes a first preprocessing unit and a second preprocessing unit.

[0146] The first preprocessing unit is used to rotate the cloud training image and the cloud mask training matrix according to a preset angle to obtain the expanded cloud training data and the corresponding expanded cloud mask training matrix. In one embodiment, the first preprocessing unit can be used to perform the operation S701 described above, which will not be repeated here.

[0147] The second preprocessing unit is used to use the augmented cloud training data and the corresponding augmented cloud mask training matrix as inputs of the remote sensing image processing model to be trained to obtain the remote sensing image processing model. In one embodiment, the second preprocessing unit can be used to perform the operation S702 described above, which will not be described in detail here.

[0148] According to an embodiment of the present disclosure, the acquisition module 901 includes a registration unit and a marking unit.

[0149] The registration unit is used to segment the registered cloud image according to a preset step size based on the registered cloud image to obtain a cloud image to be de-clouded, wherein the cloud image to be de-clouded includes multiple image blocks of the preset step size. In one embodiment, the registration unit can be used to perform the operation S801 described above, which will not be repeated here.

[0150] The marking unit is used to determine the cloud mask matrix corresponding to the cloud image to be declouded, wherein the cloud area in the cloud mask matrix is ​​marked as 0, and other areas are marked as 1, to obtain the cloud mask matrix. In one embodiment, the marking unit can be used to perform the operation S802 described above, which will not be repeated here.

[0151] Fig.10 A block diagram of an electronic device suitable for a remote sensing image processing model according to an embodiment of the present disclosure is schematically shown.

[0152] like Fig.10As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0153] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0154] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the I / O interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0155] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0156] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0157] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the training method of the remote sensing image processing model provided by the embodiment of the present disclosure.

[0158] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0159] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0160] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0161] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0162] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0164] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A remote sensing image processing method, wherein the remote sensing image processing model includes an image updating module and a matrix updating module, and the method includes: Acquire a cloud image to be declouded and a cloud mask matrix of the cloud image; Inputting the cloud image and the cloud layer mask matrix into an image updating module, eliminating clouds in a preset area of ​​the cloud image based on the cloud layer mask matrix, and outputting an updated cloud image; Inputting the updated cloud image into a matrix update module, updating the cloud mask matrix, and outputting an updated cloud mask matrix; Obtaining a cloud-removed image according to the updated cloud image and the updated cloud mask matrix; The step of inputting the cloud image and the cloud mask matrix into an image updating module, removing clouds in a preset area of ​​the cloud image based on the cloud mask matrix, and outputting an updated cloud image includes: Multiply the cloud image by the cloud mask matrix to obtain a first feature information matrix; adjusting the weight of a convolution kernel used for extracting features from the cloud image, and using the adjusted convolution kernel to process the first feature information matrix to obtain second feature information; and adjusting the information amount of the second feature information by using a scaling factor, and outputting an updated cloud image; The step of inputting the updated cloud image into a matrix update module, updating the cloud mask matrix, and outputting the updated cloud mask matrix comprises: Determining a first region to be convolved in the cloud mask matrix; determining a second area corresponding to the first area in the updated cloud image; When there is a part of the second area that is not covered by clouds, the value corresponding to the center position of the mask convolution kernel is updated to 1 to obtain the mask convolution kernel of the first area; Convolving the first region using the first region mask convolution kernel to update the first region of the cloud layer mask matrix; and When the convolution operation is completed for all regions of the cloud mask matrix, an updated cloud mask matrix is ​​output.

2. The method according to claim 1, wherein: The step of obtaining the cloud-removed image according to the updated cloud image and the updated cloud mask matrix comprises: Using the image update module, perform E times of feature extraction on the updated cloud image and the updated cloud mask matrix until all information covered by clouds in the cloud image is restored, and output a cloud-removed image, wherein a cloud image restoring the preset area information is generated after each feature extraction, and E is an integer greater than or equal to 2; and After obtaining the cloud image that restores the preset area information, before performing the next feature extraction, the updated cloud image is updated using the matrix update module to obtain a cloud mask matrix corresponding to the updated cloud image.

3. The method according to claim 1, wherein: The cloud image includes remote sensing images of N bands, and the input of the remote sensing image processing model is the remote sensing images of the N bands and the cloud mask matrix, wherein N is an integer greater than or equal to 2.

4. According to the method of claim 1, the training sample set includes a plurality of data pairs, each data pair includes a cloud training image, a cloud-free training image and a cloud mask training matrix, and the remote sensing image processing model is determined by the following steps: Rotating the cloud training image and the cloud mask training matrix according to a preset angle to obtain expanded cloud training data and a corresponding expanded cloud mask training matrix; and The expanded cloud training data and the corresponding expanded cloud mask training matrix are used as inputs of a remote sensing image processing model to be trained to obtain the remote sensing image processing model.

5. The method according to claim 1, wherein: The step of obtaining a cloud image to be declouded and a cloud mask matrix of the cloud image comprises: Based on the registered cloud image, segmenting the registered cloud image according to a preset step size to obtain a cloud image to be de-clouded, wherein the cloud image to be de-clouded includes a plurality of image blocks of the preset step size; and According to the cloud image to be declouded, a cloud layer mask matrix corresponding to the cloud image is determined, wherein the cloud area in the cloud layer mask matrix is ​​marked as 0, and other areas are marked as 1.

6. A remote sensing image processing device, wherein the remote sensing image processing model includes an image updating module and a matrix updating module, and the device includes: An acquisition module, used for acquiring a cloud image to be declouded and a cloud mask matrix of the cloud image; A first output module, configured to input the cloud image and the cloud layer mask matrix into an image updating module, eliminate clouds in a preset area of ​​the cloud image based on the cloud layer mask matrix, and output an updated cloud image; A second output module, configured to input the updated cloud image into a matrix update module, update the cloud mask matrix, and output an updated cloud mask matrix; an elimination module, which obtains a cloud-eliminated image according to the updated cloud image and the updated cloud mask matrix; The step of inputting the cloud image and the cloud mask matrix into an image updating module, removing clouds in a preset area of ​​the cloud image based on the cloud mask matrix, and outputting an updated cloud image includes: Multiply the cloud image by the cloud mask matrix to obtain a first feature information matrix; adjusting the weight of a convolution kernel used for extracting features from the cloud image, and using the adjusted convolution kernel to process the first feature information matrix to obtain second feature information; and adjusting the information amount of the second feature information by using a scaling factor, and outputting an updated cloud image; The step of inputting the updated cloud image into a matrix update module, updating the cloud mask matrix, and outputting the updated cloud mask matrix comprises: Determining a first region to be convolved in the cloud mask matrix; determining a second area corresponding to the first area in the updated cloud image; When there is a part of the second area that is not covered by clouds, the value corresponding to the center position of the mask convolution kernel is updated to 1 to obtain the mask convolution kernel of the first area; Convolving the first region using the first region mask convolution kernel to update the first region of the cloud layer mask matrix; and When the convolution operation is completed for all regions of the cloud mask matrix, an updated cloud mask matrix is ​​output.

7. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 5.

9. A computer program product, comprising computer executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

Images