Optical remote sensing image cloud detection model training method, detection method and device

Abstract

The present disclosure provides an optical remote sensing image cloud detection model training method, comprising: inputting an optical remote sensing image into an untrained cloud detection model to obtain a teacher model through training; fixing the parameters of the teacher model, inputting the optical remote sensing image into the teacher model and a preset student model respectively to output a first feature map, a first prediction probability distribution and a second feature map, a second prediction probability distribution respectively; determining first feature correlation knowledge according to the first feature map; determining second feature correlation knowledge according to the second feature map; calculating a loss value of the preset student model according to the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, a prediction probability value calculated by using the first prediction probability distribution and a true value label; updating the parameters of the preset student model according to the loss value until the preset student model converges, and taking the converged preset student model as a target optical remote sensing image cloud detection model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of optical remote sensing, and in particular to an optical remote sensing image cloud detection model training method, a detection method and apparatus. BACKGROUND

[0002] With the rapid development of remote sensing technology, satellite images are widely used in different fields such as weather warning, disaster monitoring, military supervision, etc. However, due to the influence of atmospheric density and cloud changes on the sensor, many images have cloud cover problems, making cloud detection a key to subsequent identification, classification and interpretation of images, and one of the foundations of remote sensing image data repair. The cloud detection method in the related art is based on a deep neural network to achieve high-precision prediction. However, the deep neural network model trained using the training method in the related art has the disadvantages of large parameter quantity, high floating point calculation, and high cost of storage and computing resources. SUMMARY

[0003] In view of the above problems, the present disclosure provides an optical remote sensing image cloud detection model training method, a detection method and apparatus that improve cloud detection efficiency.

[0004] According to a first aspect of the present disclosure, an optical remote sensing image cloud detection model training method is provided, comprising:

[0005] inputting an optical remote sensing image in an optical remote sensing image dataset into an untrained deep neural network-based cloud detection model to train a trained deep neural network-based cloud detection model as a teacher model;

[0006] in a case where model parameters of the teacher model are fixed, inputting the optical remote sensing image in the optical remote sensing image dataset into the teacher model and a preset student model respectively, and outputting a first feature map and a second feature map extracted by the models, and a first prediction probability distribution and a second prediction probability distribution predicted by the models, wherein a prediction probability value calculated from the first prediction probability distribution represents prediction ability knowledge of the teacher model;

[0007] determining first feature correlation knowledge according to the first feature map, and determining second feature correlation knowledge according to the second feature map;

[0008] calculating a loss value of the preset student model according to the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, a prediction probability value calculated from the first prediction probability distribution, and a true value label of the optical remote sensing image;

[0009] According to the loss value, the parameters of the preset student model are updated until the preset student model converges, and the converged preset student model is taken as the target optical remote sensing image cloud detection model.

[0010] According to an embodiment of the present disclosure, the first feature correlation knowledge is determined according to the first feature map, and the second feature correlation knowledge is determined according to the second feature map, including:

[0011] The true value label is down-sampled to adapt to the size of the first feature map and the second feature map, wherein the optical remote sensing image dataset includes the optical remote sensing image and the true value label of the manually labeled cloud region;

[0012] According to the first feature map, the feature mapping of the non-cloud region and the cloud region of the teacher model is determined, and according to the second feature map, the feature mapping of the non-cloud region and the cloud region of the preset student model is determined;

[0013] The feature centers of the non-cloud region and the cloud region of the teacher model and the feature centers of the non-cloud region and the cloud region of the preset student model are respectively calculated by using global average pooling;

[0014] According to the cosine distance between the feature mapping of the teacher model and the corresponding feature center, the first feature correlation knowledge is calculated, and according to the feature mapping of the preset student model and the corresponding feature center, the second feature correlation knowledge is calculated.

[0015] According to an embodiment of the present disclosure, the first feature correlation knowledge is calculated according to the cosine distance between the feature mapping of the teacher model and the corresponding feature center, including:

[0016] The feature correlation knowledge of the non-cloud region of the teacher model is calculated according to the cosine distance between the first feature mapping of the non-cloud region of the teacher model and the corresponding feature center;

[0017] The feature correlation knowledge of the cloud region of the teacher model is calculated according to the cosine distance between the second feature mapping of the cloud region of the teacher model and the corresponding feature center;

[0018] The feature correlation knowledge of the non-cloud region of the teacher model and the feature correlation knowledge of the cloud region of the teacher model are jointly taken as the first feature correlation knowledge; and

[0019] According to the feature mapping of the preset student model and the corresponding feature center, the second feature correlation knowledge is calculated, including:

[0020] According to the cosine distance between the third feature mapping of the non-cloud area of the preset student model and the corresponding feature center, the feature correlation knowledge of the non-cloud area of the preset student model is calculated.

[0021] According to the cosine distance between the fourth feature mapping of the cloud area of the preset student model and the corresponding feature center, the feature correlation knowledge of the cloud area of the preset student model is calculated.

[0022] The feature correlation knowledge of the non-cloud area of the preset student model and the feature correlation knowledge of the cloud area of the preset student model are combined as the second feature correlation knowledge.

[0023] According to an embodiment of the present disclosure, the loss value includes: a feature correlation loss value, a pixel adaptive distillation loss value, and a cross-entropy loss value.

[0024] The feature correlation loss value is calculated by applying the first feature correlation knowledge and the second feature correlation knowledge to the KL divergence, and the KL divergence is used as a loss function.

[0025] The pixel adaptive distillation loss value is calculated by: calculating a predicted probability value by calculating the inner product of the first predicted probability distribution and the One-hot true value; using the KL divergence as a loss function between the first predicted probability distribution and the second predicted probability distribution, multiplying the predicted probability value and the KL divergence to calculate a weighted loss value, and the weighted loss value is used as the pixel adaptive distillation loss value; and

[0026] The cross-entropy loss value is calculated by: inputting an optical remote sensing image in the optical remote sensing image dataset into the preset student model to output the second predicted probability distribution of the preset student model; and calculating the cross-entropy loss value by using the true value label of the optical remote sensing image and the second predicted probability distribution.

[0027] According to an embodiment of the present disclosure, the teacher model is composed of a deep neural network feature extractor and a predictor.

[0028] According to an embodiment of the present disclosure, the model parameters of the teacher model are fixed, including:

[0029] The network structure and the weight of the teacher model are fixed.

[0030] The second aspect of the present disclosure provides an optical remote sensing image cloud detection method, including:

[0031] The optical remote sensing image dataset is input into the target optical remote sensing image cloud detection model to output an optical remote sensing image cloud detection result.

[0032] The target optical remote sensing image cloud detection model is trained by the optical remote sensing image cloud detection model training method according to any one of the preceding embodiments.

[0033] The third aspect of the present disclosure provides an optical remote sensing image cloud detection model training device, comprising:

[0034] The training module is configured to input an optical remote sensing image in an optical remote sensing image dataset into an untrained deep neural network-based cloud detection model, and train a trained deep neural network-based cloud detection model as a teacher model.

[0035] The feature extraction module is configured to input the optical remote sensing image in the optical remote sensing image dataset into the teacher model and a preset student model respectively under the condition that the model parameters of the teacher model are fixed, and output a first feature map and a second feature map extracted by the models and a first prediction probability distribution and a second prediction probability distribution predicted by the models respectively, wherein a prediction probability value calculated from the first prediction probability distribution represents the prediction ability knowledge of the teacher model.

[0036] The determination module is configured to determine first feature correlation knowledge according to the first feature map, and determine second feature correlation knowledge according to the second feature map.

[0037] The calculation module is configured to calculate a loss value of the preset student model according to the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, a prediction probability value calculated from the first prediction probability distribution, and a true value label of the optical remote sensing image.

[0038] The update module is configured to update the parameters of the preset student model according to the loss value until the preset student model converges, and take the converged preset student model as a target optical remote sensing image cloud detection model.

[0039] The fourth aspect of the present disclosure provides an optical remote sensing image cloud detection device, comprising:

[0040] The processing module is configured to input the optical remote sensing image dataset into the target optical remote sensing image cloud detection model, and output an optical remote sensing image cloud detection result.

[0041] The target optical remote sensing image cloud detection model is trained by the optical remote sensing image cloud detection model training method according to any one of the preceding embodiments.

[0042] In a fifth aspect, the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the optical remote sensing image cloud detection model training method and the optical remote sensing image cloud detection method.

[0043] According to an embodiment of the present disclosure, by training a powerful teacher model in advance, feature correlation knowledge and model prediction ability knowledge of the teacher model are constructed respectively based on feature output and prediction probability distribution of the teacher model, and are migrated to a preset student model as supervision information of the teacher model, to obtain a lightweight cloud detection model. The lightweight cloud detection model trained by the training method has high cloud detection accuracy and cloud detection efficiency, and overcomes the defects of a large number of parameters, high floating point calculation, and a large amount of storage and computing resources of a deep model in the related art, to realize a lightweight and efficient lightweight cloud detection model. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0045] Figure 1 An application scenario diagram of the optical remote sensing image cloud detection model training method, the detection method, the device and the equipment according to an embodiment of the present disclosure is schematically shown;

[0046] Figure 2 A flowchart of the optical remote sensing image cloud detection model training method according to an embodiment of the present disclosure is schematically shown;

[0047] Figure 3 A flowchart of the optical remote sensing image cloud detection method according to an embodiment of the present disclosure is schematically shown;

[0048] Figure 4 A structure block diagram of the optical remote sensing image cloud detection model training device according to an embodiment of the present disclosure is schematically shown;

[0049] Figure 5 A structure block diagram of the optical remote sensing image cloud detection device according to an embodiment of the present disclosure is schematically shown;

[0050] Figure 6 A block diagram of an electronic device suitable for implementing the optical remote sensing image cloud detection model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0051] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have been omitted to avoid unnecessarily complicating the present disclosure with details that will be readily apparent to those skilled in the art.

[0052] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or components thereof.

[0053] All terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings that are consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0054] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any of them, to include, for example, only A, only B, only C, A and B, A and C, B and C, and A and B and C, etc.

[0055] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0056] At present, many lightweight small models are widely studied, but the cloud detection performance of these lightweight models is far inferior to that of deep neural network models. Therefore, it is of great significance to realize a lightweight and efficient optical remote sensing image cloud detection model.

[0057] Knowledge distillation is a commonly used method of model compression. Unlike pruning and quantization in model compression, knowledge distillation is to train a small model by using the supervision information of a large model with better performance, so as to achieve better performance and accuracy. It was first proposed by Hinton in 2015 and applied to classification tasks. The large model is called the teacher model, and the small model is called the student model. The supervision information from the teacher model output is called knowledge, and the process of the student model learning to transfer the supervision information from the teacher model is called distillation.

[0058] Based on this, the present disclosure provides an optical remote sensing image cloud detection model training method. The method first trains a teacher model with strong cloud detection performance using a deep neural network. The teacher model is used to model the cloud targets in the optical remote sensing image. The feature correlation knowledge and model prediction ability knowledge constructed are distilled to enable a lightweight student model to imitate the feature output and network output of the teacher model, thereby achieving performance improvement of the lightweight cloud detection model.

[0059] It should be noted that, unless it is explicitly stated that there is an execution order between different operations or there is an execution order between different operations in technical implementation, the execution order between multiple operations can not be distinguished, and multiple operations can be executed simultaneously.

[0060] Figure 1 An application scenario diagram of the optical remote sensing image cloud detection model training method, the detection method, the device and the equipment according to the embodiments of the present disclosure is schematically shown.

[0061] As shown in Figure 1 According to the application scenario 100 of the embodiment, the terminal devices 101, 102 and 103, the network 104 and the server 105 can be included.

[0062] The terminal devices 101, 102 and 103 can be various electronic devices with communication functions, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. The terminal devices 101, 102 and 103 can have a memory, and the memory can store an optical remote sensing image dataset used for model training. Alternatively, the terminal devices 101, 102 and 103 can be connected with an external storage device, and the external storage device can store an optical remote sensing image dataset used for model training.

[0063] The network 104 is a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0064] The server 105 can be a server that provides computing resource support for various computing tasks, for example, when a user trains a student model using the terminal devices 101, 102, 103, the server 105 can provide computing resource support for the training process of the student model.

[0065] It should be noted that the optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the optical remote sensing image cloud detection model training apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the optical remote sensing image cloud detection model training apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or by other terminal devices different from the terminal devices 101, 102, or 103. Accordingly, the optical remote sensing image cloud detection model training apparatus provided by the embodiments of the present disclosure can also be arranged in the terminal devices 101, 102, or 103, or in other terminal devices different from the terminal devices 101, 102, or 103.

[0066] For example, the optical remote sensing image dataset can originally be stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or on an external storage device and can be imported into the terminal device 101. Then, the terminal device 101 can execute the optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure locally to train a student network using the optical remote sensing image dataset to obtain a target optical remote sensing image cloud detection model, or the terminal device 101 can send the optical remote sensing image dataset to other terminal devices, servers, or server clusters, and the optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure can be executed by other terminal devices, servers, or server clusters that receive the optical remote sensing image dataset to train a student network using the optical remote sensing image dataset to obtain a target optical remote sensing image cloud detection model.

[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.

[0068] Figure 2 A flowchart of an optical remote sensing image cloud detection model training method according to an embodiment of the present disclosure is illustratively shown.

[0069] As Figure 2 The method can include operations S210-S250.

[0070] In operation S210, the optical remote sensing images in the optical remote sensing image dataset are input into an untrained deep neural network-based cloud detection model, and a trained deep neural network-based cloud detection model is obtained as a teacher model.

[0071] In operation S220, under the condition that the model parameters of the teacher model are fixed, the optical remote sensing images in the optical remote sensing image dataset are respectively input into the teacher model and a preset student model, and first and second feature maps extracted by the models and first and second prediction probability distributions predicted by the models are respectively output, wherein a prediction probability value calculated from the first prediction probability distribution represents the prediction ability knowledge of the teacher model.

[0072] In operation S230, first feature correlation knowledge is determined according to the first feature map, and second feature correlation knowledge is determined according to the second feature map.

[0073] In operation S240, a loss value of the preset student model is calculated according to the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, a prediction probability value calculated from the first prediction probability distribution, and a true value label of the optical remote sensing image.

[0074] In operation S250, according to the loss value, the parameters of the preset student model are updated using a stochastic gradient descent algorithm until the preset student model converges, and the converged preset student model is taken as a target optical remote sensing image cloud detection model.

[0075] According to an embodiment of the present disclosure, by training a powerful teacher model in advance, feature correlation knowledge and model prediction ability knowledge of the teacher model are constructed respectively according to feature outputs and prediction probability distributions of the teacher model, and are migrated to a preset student model as supervision information of the teacher model, to obtain a lightweight cloud detection model. The lightweight cloud detection model trained by the training method has high cloud detection accuracy and cloud detection efficiency, and overcomes the defects of a large number of parameters, high floating point calculation, and a large amount of storage and computing resources of a deep model in the related art, to realize a lightweight and efficient lightweight cloud detection model.

[0076] According to an embodiment of the present disclosure, the first feature correlation knowledge is determined according to the first feature map, and the second feature correlation knowledge is determined according to the second feature map, including:

[0077] The ground truth label is down-sampled to adapt to the size of the first feature map and the second feature map, wherein the optical remote sensing image dataset includes an optical remote sensing image and a ground truth label of an artificially labeled cloud region;

[0078] The feature mapping of the non-cloud region and the cloud region of the teacher model is determined according to the first feature map, and the feature mapping of the non-cloud region and the cloud region of the preset student model is determined according to the second feature map;

[0079] The feature center of the non-cloud region and the cloud region of the teacher model and the feature center of the non-cloud region and the cloud region of the preset student model are respectively calculated by using global average pooling;

[0080] The first feature correlation knowledge is calculated according to the cosine distance between the feature mapping of the teacher model and the corresponding feature center, and the second feature correlation knowledge is calculated according to the feature mapping of the preset student model and the corresponding feature center.

[0081] According to an embodiment of the present disclosure, the first feature correlation knowledge is calculated according to the cosine distance between the feature mapping of the teacher model and the corresponding feature center, including:

[0082] The feature correlation knowledge of the non-cloud region of the teacher model is calculated according to the cosine distance between the first feature mapping of the non-cloud region of the teacher model and the corresponding feature center;

[0083] The feature correlation knowledge of the cloud region of the teacher model is calculated according to the cosine distance between the second feature mapping of the cloud region of the teacher model and the corresponding feature center;

[0084] The feature correlation knowledge of the non-cloud region of the teacher model and the feature correlation knowledge of the cloud region of the teacher model are jointly used as the first feature correlation knowledge; and

[0085] According to the characteristic mapping and the corresponding characteristic center of the preset student model, the second feature correlation knowledge is calculated, including:

[0086] According to the cosine distance between the third characteristic mapping of the non-cloud area of the preset student model and the corresponding characteristic center, the feature correlation knowledge of the non-cloud area of the preset student model is calculated.

[0087] According to the cosine distance between the fourth characteristic mapping of the cloud area of the preset student model and the corresponding characteristic center, the feature correlation knowledge of the cloud area of the preset student model is calculated.

[0088] The feature correlation knowledge of the non-cloud area of the preset student model and the feature correlation knowledge of the cloud area of the preset student model are combined as the second feature correlation knowledge.

[0089] According to an embodiment of the present disclosure, the loss value includes: a feature correlation loss value, a pixel adaptive distillation loss value, and a cross-entropy loss value.

[0090] The feature correlation loss value is calculated by applying the first feature correlation knowledge and the second feature correlation knowledge to the KL divergence, wherein the KL divergence is used as a loss function.

[0091] The pixel adaptive distillation loss value is calculated by: calculating the inner product of the first prediction probability distribution and the One-hot true value to obtain a prediction probability value; using the KL divergence as a loss function between the first prediction probability distribution and the second prediction probability distribution, multiplying the prediction probability value by the KL divergence to calculate a weighted loss value, and taking the weighted loss value as the pixel adaptive distillation loss value; and

[0092] The cross-entropy loss value is calculated by: inputting the optical remote sensing image in the optical remote sensing image data set into the preset student model to output the second prediction probability distribution of the preset student model; and calculating the cross-entropy loss value using the true value label of the optical remote sensing image and the second prediction probability distribution.

[0093] According to an embodiment of the present disclosure, the teacher model is composed of a deep neural network feature extractor and a predictor; the preset student model is also composed of a feature extractor and a predictor, wherein the feature extractor of the preset student model is a neural network with less parameters and floating point operations, which can be ResNet18 or MobileNet.

[0094] According to an embodiment of the present disclosure, the fixed model parameters of the teacher model include: the network structure and the weight of the fixed teacher model.

[0095] According to an embodiment of the present disclosure, the data in the optical remote sensing image dataset is collected from an optical remote sensing satellite, and includes an optical remote sensing image and a ground truth label of a manually labeled cloud region, wherein the label is stored in a grayscale picture.

[0096] Figure 3 A flowchart of an optical remote sensing image cloud detection method according to an embodiment of the present disclosure is schematically shown.

[0097] As shown in Figure 3 , the optical remote sensing image cloud detection method can include step S310.

[0098] In step S310, the optical remote sensing image dataset is input into a target optical remote sensing image cloud detection model, and an optical remote sensing image cloud detection result is output; wherein the target optical remote sensing image cloud detection model is trained by the optical remote sensing image cloud detection model training method described above.

[0099] Based on the optical remote sensing image cloud detection model training method described above, the present disclosure further provides an optical remote sensing image cloud detection model training device, which will be described in detail below. Figure 4 .

[0100] Figure 4 A structural block diagram of an optical remote sensing image cloud detection model training device according to an embodiment of the present disclosure is schematically shown.

[0101] As shown in Figure 4 , the optical remote sensing image cloud detection model training device 400 of this embodiment includes a training module 410, a feature extraction module 420, a determination module 430, a calculation module 440, and an update module 450.

[0102] The training module 410 is configured to input the optical remote sensing image in the optical remote sensing image dataset into an untrained deep neural network-based cloud detection model, and train a trained deep neural network-based cloud detection model as a teacher model.

[0103] The feature extraction module 420 is configured to input the optical remote sensing image in the optical remote sensing image dataset into the teacher model and a preset student model respectively under the condition that the model parameters of the teacher model are fixed, and output a first feature map and a second feature map extracted by the models respectively, as well as a first prediction probability distribution and a second prediction probability distribution predicted by the models, wherein a prediction probability value calculated by using the first prediction probability distribution represents the prediction ability knowledge of the teacher model.

[0104] The determination module 430 is configured to determine first feature correlation knowledge according to the first feature map, and determine second feature correlation knowledge according to the second feature map.

[0105] The computing module 440 is configured to calculate a loss value of the preset student model according to the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, a prediction probability value calculated by using the first prediction probability distribution, and a true value label of the optical remote sensing image.

[0106] The updating module 450 is configured to update parameters of the preset student model according to the loss value until the preset student model converges, and take the converged preset student model as the target optical remote sensing image cloud detection model.

[0107] According to an embodiment of the present disclosure, the determining module 430 includes a processing sub-module, a determining sub-module, a first calculating sub-module, and a second calculating sub-module.

[0108] The processing sub-module is configured to down-sample the true value label to adapt to sizes of the first feature map and the second feature map, wherein the optical remote sensing image dataset includes the optical remote sensing image and the true value label of the artificially labeled cloud region.

[0109] The determining sub-module is configured to determine feature mappings of non-cloud regions and cloud regions of the teacher model according to the first feature map, and determine feature mappings of non-cloud regions and cloud regions of the preset student model according to the second feature map.

[0110] The first calculating sub-module is configured to calculate feature centers of the non-cloud regions and the cloud regions of the teacher model and feature centers of the non-cloud regions and the cloud regions of the preset student model respectively by using global average pooling.

[0111] The second calculating sub-module is configured to calculate the first feature correlation knowledge according to cosine distances between the feature mappings of the teacher model and corresponding feature centers, and calculate the second feature correlation knowledge according to the feature mappings of the preset student model and corresponding feature centers.

[0112] According to an embodiment of the present disclosure, the second calculating sub-module includes a first calculating unit and a second calculating unit.

[0113] The first calculating unit is configured to calculate the first feature correlation knowledge according to cosine distances between the feature mappings of the teacher model and corresponding feature centers.

[0114] The second calculating unit is configured to calculate the second feature correlation knowledge according to the feature mappings of the preset student model and corresponding feature centers.

[0115] According to an embodiment of the present disclosure, the first calculating unit includes a first calculating sub-unit, a second calculating sub-unit, and a first joint sub-unit.

[0116] The first computing subunit is configured to calculate the feature correlation knowledge of the non-cloud region of the teacher model according to cosine distances between first feature mappings of the non-cloud region of the teacher model and corresponding feature centers.

[0117] The first joint subunit is configured to jointly take the feature correlation knowledge of the non-cloud region of the teacher model and the feature correlation knowledge of the cloud region of the teacher model as first feature correlation knowledge.

[0118] According to an embodiment of the present disclosure, the second computing unit includes a third computing subunit, a fourth computing subunit and a second joint subunit.

[0119] The third computing subunit is configured to calculate the feature correlation knowledge of the non-cloud region of the preset student model according to cosine distances between third feature mappings of the non-cloud region of the preset student model and corresponding feature centers.

[0120] The fourth computing subunit is configured to calculate the feature correlation knowledge of the cloud region of the preset student model according to cosine distances between fourth feature mappings of the cloud region of the preset student model and corresponding feature centers.

[0121] The second joint subunit is configured to jointly take the feature correlation knowledge of the non-cloud region of the preset student model and the feature correlation knowledge of the cloud region of the preset student model as second feature correlation knowledge.

[0122] According to an embodiment of the present disclosure, the computing module 430 includes a third computing sub-module and a fourth computing sub-module.

[0123] The third computing sub-module is configured to apply the first feature correlation knowledge and the second feature correlation knowledge to the KL divergence to calculate a feature correlation loss value.

[0124] The fourth computing sub-module is configured to calculate a pixel adaptive distillation loss value according to the first predicted probability distribution and the second predicted probability distribution.

[0125] The fifth computing sub-module is configured to calculate a cross-entropy loss value according to the second predicted probability distribution and the true value label.

[0126] According to an embodiment of the present disclosure, the fourth computing sub-module includes a third computing unit and a fourth computing unit.

[0127] The third computing unit is configured to calculate a predicted probability value by taking an inner product of the first predicted probability distribution and a One-hot true value.

[0128] The fourth calculation unit is configured to use the KL divergence as a loss function between the first predicted probability distribution and the second predicted probability distribution, multiply the predicted probability value by the KL divergence, and calculate a weighted loss value as a pixel self-adaptive distillation loss value.

[0129] According to an embodiment of the present disclosure, the fifth calculation sub-module comprises a first output unit and a fifth calculation unit.

[0130] The first output unit is configured to input an optical remote sensing image in the optical remote sensing image dataset into the preset student model, and output a second predicted probability distribution of the preset student model.

[0131] The fifth calculation unit is configured to calculate a cross-entropy loss value by using the true value label of the optical remote sensing image and the second predicted probability distribution.

[0132] Based on the optical remote sensing image cloud detection method described above, the present disclosure further provides an optical remote sensing image cloud detection device, which will be described in detail below. Figure 5 The device is described in detail.

[0133] Figure 5 The structure block diagram of the optical remote sensing image cloud detection device according to an embodiment of the present disclosure is schematically shown.

[0134] As shown in Figure 5 The optical remote sensing image cloud detection device 500 of this embodiment comprises a processing module 510.

[0135] The processing module 510 is configured to input an optical remote sensing image dataset into a target optical remote sensing image cloud detection model, and output an optical remote sensing image cloud detection result; wherein the target optical remote sensing image cloud detection model is obtained by training according to the optical remote sensing image cloud detection model training method described above.

[0136] According to an embodiment of the present disclosure, any one or more of the training module 410, the feature extraction module 420, the determination module 430, the calculation module 440, and the updating module 450, or any one or more of the processing modules 510 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of the other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the training module 410, the feature extraction module 420, the determination module 430, the calculation module 440, and the updating module 450, or the processing module 510 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged with a circuit, or any one of software, hardware, and firmware or any appropriate combination of any of them. Alternatively, at least one of the training module 410, the feature extraction module 420, the determination module 430, the calculation module 440, and the updating module 450, or the processing module 510 can be at least partially implemented as a computer program module that can perform corresponding functions when executed.

[0137] It should be noted that the optical remote sensing image cloud detection model training device part in the embodiments of the present disclosure corresponds to the optical remote sensing image cloud detection model training method part in the embodiments of the present disclosure. The description of the optical remote sensing image cloud detection model training device part, and the training method of the target optical remote sensing image cloud detection model in the optical remote sensing image cloud detection device are specifically referred to the optical remote sensing image cloud detection model training method part, and will not be repeated here.

[0138] Figure 6 The block diagram of the electronic device suitable for implementing the optical remote sensing image cloud detection model training method according to an embodiment of the present disclosure is schematically shown. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0139] As Figure 6As shown, the electronic device 600 according to the embodiments of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded into a random access memory (RAM) 603 from a storage section 608. The processor 601 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and the like. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing the various actions of the method processes according to the embodiments of the present disclosure.

[0140] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method processes according to the embodiments of the present disclosure by executing the programs in the ROM 602 and / or the RAM 603. Note that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0141] According to the embodiments of the present disclosure, the electronic device 600 can further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable recording medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read therefrom is installed into the storage section 608 as necessary.

[0142] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0143] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include but is not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories, such as the ROM 602 and / or the RAM 603 described above, and / or one or more memory external to the ROM 602 and the RAM 603.

[0144] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in the computer system, the program codes are used to make the computer system implement the optical remote sensing image cloud detection model training method provided by the embodiments of the present disclosure.

[0145] The above functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 601. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0146] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on a network medium, and be downloaded and installed through the communication part 609, and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any appropriate combination thereof.

[0147] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the detachable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0148] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0149] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions executable by a machine such as a personal digital assistant (PDA), a laptop, a desktop computer, or a server. Alternatively, the computer program product can be a propagated signal per se generated by using the program code, and the program code can be stored on a machine-readable medium that can be read by a machine such as a personal digital assistant (PDA), a laptop, a desktop computer, or a server. The program code of the computer program product defines a function, preferably a corresponding action, and / or implements the elements / object / means for performing this function or a part thereof as disclosed in the embodiments of the present disclosure. The program code can be executed by using one or more processors.

[0150] Those skilled in the art will appreciate that features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such combinations or integrations are not expressly disclosed in the present disclosure. In particular, the features of the various embodiments and / or claims of the present disclosure can be combined and / or integrated with one another in any manner, without departing from the spirit and scope of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0151] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A method for training a cloud detection model in optical remote sensing images, comprising: Optical remote sensing images from the optical remote sensing image dataset are input into an untrained deep neural network-based cloud detection model to train a trained deep neural network-based cloud detection model, which serves as the teacher model. With the model parameters of the teacher model fixed, the optical remote sensing images in the optical remote sensing image dataset are input into the teacher model and the preset student model respectively, and the first feature map and the second feature map extracted by the model are output respectively, as well as the first prediction probability distribution and the second prediction probability distribution predicted by the model. The prediction probability value calculated using the first prediction probability distribution represents the prediction ability knowledge of the teacher model. Based on the first feature map, determine the first feature association knowledge; and based on the second feature map, determine the second feature association knowledge; Based on the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, the prediction probability value calculated using the first prediction probability distribution and the ground truth label of the optical remote sensing image, the loss value of the preset student model is calculated. Based on the loss value, the parameters of the preset student model are updated until the preset student model converges, and the converged preset student model is used as the target optical remote sensing image cloud detection model. Wherein, determining the first feature correlation knowledge based on the first feature map and determining the second feature correlation knowledge based on the second feature map includes: The ground truth labels are downsampled to fit the size of the first feature map and the second feature map, wherein the optical remote sensing image dataset includes the optical remote sensing images and ground truth labels of manually annotated cloud regions; Based on the first feature map, the feature mappings of the non-cloud region and the cloud region of the teacher model are determined; and based on the second feature map, the feature mappings of the non-cloud region and the cloud region of the preset student model are determined. Using global average pooling, the feature centers of the non-cloud and cloud regions of the teacher model and the feature centers of the non-cloud and cloud regions of the preset student model are calculated respectively. The first feature correlation knowledge is calculated based on the cosine distance between the feature mapping of the teacher model and the corresponding feature center; and the second feature correlation knowledge is calculated based on the feature mapping of the preset student model and the corresponding feature center.

2. The training method according to claim 1, wherein, Based on the cosine distance between the feature mapping of the teacher model and the corresponding feature center, the first feature correlation knowledge is calculated, including: Based on the first feature mapping of the non-cloud region of the teacher model and the cosine distance between the corresponding feature centers, the feature correlation knowledge of the non-cloud region of the teacher model is calculated. Based on the second feature mapping of the cloud region of the teacher model and the cosine distance between the corresponding feature centers, the feature correlation knowledge of the cloud region of the teacher model is calculated. The feature correlation knowledge of the non-cloud region of the teacher model and the feature correlation knowledge of the cloud region of the teacher model are combined as the first feature correlation knowledge; and Based on the feature mapping and corresponding feature centers of the preset student model, the second feature correlation knowledge is calculated, including: Based on the third feature mapping of the non-cloud region of the preset student model and the cosine distance between the corresponding feature center, the feature correlation knowledge of the non-cloud region of the preset student model is calculated. Based on the fourth feature mapping of the cloud region of the preset student model and the cosine distance between the corresponding feature centers, the feature correlation knowledge of the cloud region of the preset student model is calculated. The feature correlation knowledge of the non-cloud region of the preset student model and the feature correlation knowledge of the cloud region of the preset student model are combined as the second feature correlation knowledge.

3. The training method according to claim 1, wherein, The loss values ​​include: feature correlation loss value, pixel adaptive distillation loss value, and cross-entropy loss value; The feature correlation loss value is calculated by applying the first feature correlation knowledge and the second feature correlation knowledge to the KL divergence, where the KL divergence is used as the loss function. The pixel adaptive distillation loss value is calculated as follows: The predicted probability value is obtained by calculating the inner product of the first predicted probability distribution and the One-hot ground truth value; KL divergence is used as the loss function between the first and second predicted probability distributions, and the predicted probability value is multiplied by the KL divergence to calculate the weighted loss value, which is used as the pixel adaptive distillation loss value; and The cross-entropy loss value is calculated as follows: the optical remote sensing images in the optical remote sensing image dataset are input into a preset student model, and the second prediction probability distribution of the preset student model is output; the cross-entropy loss value is calculated using the ground truth labels of the optical remote sensing images and the second prediction probability distribution.

4. The training method according to claim 1, wherein, The teacher model consists of a deep neural network feature extractor and a predictor.

5. The training method according to claim 1, wherein, The fixed model parameters of the teacher model include: The network structure and weights of the teacher model are fixed.

6. A method for cloud detection in optical remote sensing images, comprising: The optical remote sensing image dataset is input into the target optical remote sensing image cloud detection model, and the optical remote sensing image cloud detection result is output. The target optical remote sensing image cloud detection model is obtained by training the optical remote sensing image cloud detection model training method according to any one of claims 1 to 5.

7. An optical remote sensing image cloud detection model training device, used to implement the optical remote sensing image cloud detection model training method according to any one of claims 1 to 5, wherein the optical remote sensing image cloud detection model training device comprises: The training module is used to input optical remote sensing images from the optical remote sensing image dataset into an untrained deep neural network-based cloud detection model, and train the trained deep neural network-based cloud detection model as a teacher model. The feature extraction module is used to input optical remote sensing images from the optical remote sensing image dataset into the teacher model and the preset student model respectively, while fixing the model parameters of the teacher model, and output the first feature map and the second feature map extracted by the model, as well as the first prediction probability distribution and the second prediction probability distribution predicted by the model respectively, wherein the prediction probability value calculated by the first prediction probability distribution represents the prediction ability knowledge of the teacher model. The determining module is configured to determine first feature correlation knowledge based on the first feature map; and to determine second feature correlation knowledge based on the second feature map. The calculation module is used to calculate the loss value of the preset student model based on the first feature correlation knowledge, the second feature correlation knowledge, the first prediction probability distribution and the second prediction probability distribution, the prediction probability value calculated using the first prediction probability distribution and the ground truth label of the optical remote sensing image. The update module is used to update the parameters of the preset student model according to the loss value until the preset student model converges, and the converged preset student model is used as the target optical remote sensing image cloud detection model.

8. An optical remote sensing image cloud detection device, comprising: The processing module is used to input the optical remote sensing image dataset into the target optical remote sensing image cloud detection model and output the optical remote sensing image cloud detection result; The target optical remote sensing image cloud detection model is obtained by training the optical remote sensing image cloud detection model training method according to any one of claims 1 to 5.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

Images