Method for extracting cloud features in high-resolution image

Abstract

The invention relates to the field of image processing, in particular to a method for extracting cloud features in a high-resolution image, a network G adopts an encoding and decoding structure, and an encoder is responsible for encoding input data and extracting advanced feature representation of the input data. In this way, the data dimension can be reduced while the key information is reserved. In the decoder, the feature maps of the encoder and the decoder are fused layer by layer. In order to better transfer the bottom and high level features, the coding and decoding structure usually uses jump connection. Jump connection connects feature maps of certain layers in the encoder with feature maps of corresponding layers in the decoder. According to the method, the model G capable of accurately identifying the cloud features is obtained through optimization training of the coding and decoding network G. In the training process, positive and negative samples which are close in proportion and are not subjected to color homogenizing processing are selected, and the samples are marked according to the characteristics of the cloud. Therefore, the identified cloud features are closer to the actual situation, and the accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a cloud extraction method under complex background in high-resolution images. Background Art

[0002] In recent years, with the rapid development of remote sensing technology, remote sensing images have been rapidly applied to fields such as earth observation, resource survey, and natural disaster prediction. High-resolution images also play an important role in environmental monitoring. They can provide detailed information on land use, vegetation cover, meteorological observation, etc.

[0003] The difference in atmospheric density and cloud cover cause most remote sensing images to have varying degrees of cloud pollution, which not only greatly reduces the quality and utilization of remote sensing data, but also brings great difficulties to the interpretation, analysis and application of remote sensing images. In addition, with the continuous development of earth observation satellites, the amount of image data has gradually increased. Therefore, quickly and accurately extracting the cloud cover range from the image has important theoretical value and practical significance for the quality inspection of satellite images and subsequent image interpretation and analysis.

[0004] Traditional cloud extraction methods often do not work well in complex environments or when there are multiple noise interferences. It is difficult to accurately distinguish between clouds and snow with very similar spectral and texture features. Due to the poor generalization ability of band reflectance information for different extraction scenarios, the accuracy of traditional threshold methods is relatively low. Some machine learning-based methods have begun to emerge. Machine learning-based algorithms use manually selected image texture, brightness and other features.

[0005] With the continuous advancement of deep learning algorithms, more and more remote sensing problems can be effectively solved through deep learning methods. The excellent feature extraction and learning capabilities of deep learning can solve the problems of low efficiency and poor adaptability of traditional remote sensing cloud extraction processing methods, improve the accuracy of remote sensing image cloud extraction, and enhance the generalization ability of cloud extraction models.

[0006] At present, the mainstream data sources for cloud feature extraction are various types of satellite remote sensing images, that is, by finding one or more types of features that meet specific conditions from digital images, and restoring the geographical information of the cloud from the pixels where the features are located, the commonly used cloud features are summarized into the following three categories based on the review research of Xhiwei Li et al. in 2022 and other literature:

[0007] 1) Spectral characteristics: Clouds have specific optical properties in terms of reflection, scattering, and absorption of light. Different types of clouds have different characteristic reflectance curves on the spectrum. The spectral characteristics of clouds can be obtained by analyzing the reflectance of clouds in different bands. Cloud detection is often performed by using a combination of multi-band thresholds or by using prior knowledge such as the surface reflectance set.

[0008] 2) Spatial texture features: The texture features of clouds belong to random textures. Although its texture primitives are changeable and unpredictable, they are different from the statistical texture characteristics of the underlying surface objects. The grayscale level of the cloud area is higher than that of the underlying surface. The edge contains more grayscale jump features similar to steps, and has a considerable area. The local and overall clouds have a certain degree of similarity, so the cloud has a certain fractal self-similarity.

[0009] 3) Temporal features: Methods based on temporal features are mainly based on the spectrum of a single pixel and its temporal variation. Such methods usually treat cloud extraction as a variation extraction problem. Specifically, pixels with a sudden increase in reflectance in the time series are more likely to be clouds, while pixels with a sudden decrease in reflectance are likely to be cloud shadows.

[0010] In early remote sensing image applications, cloud detection and identification mainly relied on experienced staff to make judgments through visual inspection and careful analysis. This method is restricted by many factors such as the reader's experience and familiarity with the identification area. It also takes a lot of time, is labor-intensive, and has a long information acquisition cycle. Most traditional classification methods are shallow structures and rely on manual experience to extract sample features. Deep learning transforms the feature representation of the sample in the original space into a new feature space by performing layer-by-layer feature transformation on the original signal, and automatically learns to obtain a hierarchical feature representation, which is more conducive to classification or feature visualization. Early cloud extraction methods based on deep learning cut remote sensing images into multiple image blocks and defined the cloud extraction task as an image classification process. Image segmentation-based methods use convolutional neural networks to automatically extract image features, which can usually achieve higher cloud extraction accuracy than classical machine learning methods.

[0011] According to the analysis of articles on cloud extraction algorithms in recent years, cloud extraction methods of remote sensing images are divided into two categories according to the basic theories and research ideas on which the methods are based: artificial empirical algorithms based on physical features and algorithms based on machine learning. The first type of algorithm is mainly based on the spectrum, texture and other features of the physical characteristics of clouds. By analyzing the spectral differences between clouds and the surface, thresholds or rules are formulated; or based on the texture feature differences between clouds and ground objects in remote sensing images, statistical methods are used to extract the texture features and spatial features of remote sensing images to achieve cloud detection and recognition, thereby achieving cloud extraction. Second, based on the classic machine learning methods, the classic machine learning models such as support vector machines and random forests are trained using artificially selected image texture, spectrum and other features to classify image blocks / pixels and achieve cloud extraction. In the machine learning algorithm, cloud extraction is regarded as a binary classification task of image segmentation based on deep learning related algorithms. This method uses a large amount of remote sensing image data, establishes a deep neural network model, automatically extracts data features, and obtains high-precision cloud extraction results. Although the above traditional cloud extraction methods effectively reduce the cost of manual extraction and improve the extraction efficiency, they still have the following disadvantages:

[0012] 1) Due to the spectral complexity and spatiotemporal variability of surface landscapes, it is generally difficult for such algorithms to fully consider all influencing factors, and the universality of their thresholds or rules limits the stability of their detection performance on different landscape images.

[0013] 2) For areas such as the edge of clouds, the detection effect is poor due to the complex underlying surface. The cloud morphology and thickness in remote sensing images vary greatly. It is usually difficult to detect and distinguish thin clouds and cloud boundaries (which makes it difficult to manually label clouds / no clouds, and the data labels themselves have a lot of noise, affecting the results).

[0014] 3) High demand for manual intervention and post-processing: Traditional methods usually require a lot of manual pre-processing and post-processing during cloud detection, including manual selection of thresholds, manual correction and filling of loopholes, etc. This increases labor costs and reduces the automation and efficiency of cloud detection. Summary of the invention

[0015] In order to make up for the deficiencies of the prior art, the purpose of the present invention is to provide a method for extracting cloud features in high-resolution images, which has simple requirements for image preprocessing, can further refine the boundary division of the cloud layer and capture its spatial position information, and the recognition effect is more accurate.

[0016] First, to achieve the above purpose, a method for extracting cloud features in high-resolution images is proposed. The network G adopts an encoding and decoding structure. The encoder is responsible for encoding the input data and extracting its high-level feature representation. Generally, the encoder consists of multiple convolutional layers and pooling layers, gradually reducing the size and number of channels of the feature map. This can reduce the data dimension while retaining key information. In the decoder, the feature maps of the encoder and decoder are fused layer by layer. Convolution or upsampling operations are usually used to splice or weighted fusion feature maps. In order to better transfer low-level and high-level features, the encoding and decoding structure usually uses jump connections. Jump connections connect the feature maps of certain layers in the encoder with the feature maps of the corresponding layers in the decoder. The steps include: the first step is to obtain a remote sensing image, slice it according to a fixed size, and obtain a remote sensing image slice x; the second step is to input the remote sensing image slice into the encoding and decoding network G for training, and learn and output the cloud features in the image. In the second step, the encoding and decoding network G is trained according to the following steps: Step 1, the loss function of the network is the Dice loss function, which is an evaluation indicator and loss function commonly used in image segmentation tasks, and is used to measure the similarity between the predicted result and the true label. It is based on the Dice coefficient, and the relationship between the two is 1-DiceCoefficient. Assume that the predicted cloud feature is Y pred , the real cloud feature label is Y true , the Dice loss can be expressed as Among them, ∑ represents the summation operation, and ∈ is a very small positive number, which is used to stabilize the numerical value to avoid the denominator being zero. During the calculation process, the predicted cloud elements and the actual cloud element labels are regarded as binary images, in which the pixel values ​​are 0 or 1. The numerator calculates the intersection of the predicted result and the actual label, and the denominator calculates the union of the predicted result and the actual label. The value range of the Dice loss function is between 0 and 1. The closer the value is to 1, the more similar the predicted result is to the actual label, and the closer the value is to 0, the greater the difference between the predicted result and the actual label. When using the Dice loss function for model training, the goal is to minimize the Dice loss, that is, to maximize the overlap between the predicted result and the actual label by adjusting the model parameters. Step 2, set the optimization target as ,P data(x) Indicates that the distribution of all remote sensing image slices x is a real sample, and L represents the loss function for the overall data distribution during training. Where G(x) is the distribution of generated samples obtained by model G; Step 3, input the labeled image slice samples into network G, according to Calculate the model loss. Step 4, perform BP back propagation operation on the loss obtained by performing forward propagation operation on the network G in step 4; optimize the network parameters; Step 5, repeat steps 3 to 4, train the network, optimize its network parameters, until the loss of the network G gradually stabilizes, and output the network model G at this time.

[0017] Optionally, in the above extraction method, in the second step, the parameters are initialized to construct the network G, and the training of steps 2 to 5 is performed, and the network parameters are updated by back-propagation errors to generate the final model G for forward propagation operations to obtain cloud features.

[0018] Optionally, in the above extraction method, the model G uses a coding and decoding structure, the goal of which is to retain the information of the input data as much as possible and reconstruct an output similar to the original input data during the decoding process. Through the optimization of the loss function during the training process, the coding and decoding structure can learn effective data representation and reconstruction capabilities. The coding and decoding network is in the form of y=D(E(x)), where E(x) is the encoder function.

[0019] Optionally, in the above extraction method, in step 1, the ratio of positive samples to negative samples in the image slice sample x is close.

[0020] Optionally, in the above extraction method, the image slice sample x or the remote sensing image slice is not subjected to color uniformity processing.

[0021] Optionally, in the above extraction method, the image slice sample x or the remote sensing image slice is a binary image of equal size.

[0022] Beneficial effects of the present invention:

[0023] The present invention obtains a model G that can accurately identify cloud features by optimizing the training of the encoding and decoding network G. During the training process, positive and negative samples with similar proportions and without uniform color processing are selected, and the samples are marked according to the characteristics of the cloud. The cloud features thus identified are closer to the actual situation and have higher accuracy.

[0024] In order to further refine the boundary division of the cloud layer and capture its spatial position information, the present invention introduces a multi-head attention module containing a soft attention module based on U-Net, which is represented by x soft-attention =MA(g,x0), MA is a multi-head attention module, g is the output of the previous convolutional block, and x0 is the input from the skip connection. Figure 3In the module, the x input from the skip connection and the g input generated by the previous layer are fed into a 1x1 convolution to convert them into the same number of channels. Since g comes from the next layer of x and its size is 1 / 2 of x, x is downsampled. They are then accumulated and passed through another 1x1 convolution and sigmoid activation function through ReLU to obtain an importance score from 0 to 1, which is assigned to each part of the feature map. Then this attention map is multiplied by the x input of the skip connection to produce the final output of this attention block. The output results of the same 8 soft attention modules are concatenated, and then the different attention heads are average pooled to obtain the final output.

[0025] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0027] Figure 1 Schematic diagram of the process of cloud feature extraction method according to the present invention.

[0028] Figure 2 Schematic diagram of the encoding and decoding network model according to the present invention.

[0029] Figure 3 This is a structural diagram of the multi-head attention module according to the present invention.

[0030] Figure 4 It is a data source slice obtained by the present invention in the original remote sensing image.

[0031] Figure 5 It is the cloud feature marked in the above data source slice.

[0032] Figure 6 is the cloud feature extracted by the above MA_UNet model. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0034] Figure 1 The method for extracting cloud features from high-resolution images according to the present invention comprises the following steps:

[0035] The first step is to obtain a remote sensing image and slice it according to a fixed size to obtain a remote sensing image slice x;

[0036] In the second step, the remote sensing image slice is input into the trained encoding and decoding network G for forward propagation operation to obtain the cloud features therein and output the cloud features.

[0037] The above extraction method can extract cloud elements from original high-resolution remote sensing images. This method is obtained by optimizing and improving related links based on the U-Net model. The method mainly includes the following innovations:

[0038] 1: U-Net structure optimization and sample labeling process

[0039] 2: MA_UNet operation process

[0040] The above innovations are analyzed one by one below.

[0041] 1: The main improvement of the sample labeling process calculated by MA_UNet in this invention is that the deep learning model used in this patent is based on the standard U-Net structure, introduces a multi-head attention module containing a soft attention module, and optimizes the network structure and training process according to the cloud extraction task and high-resolution image features. Figure 2 As shown. The model contains the basic features of the standard encoding and decoding structure, that is, there is an encoder and a decoder. In order to better learn the high-resolution and variability image cloud features at all levels and the underlying principles of data generation, the encoder in the model used in this patent is composed of VGG16. A large number of remote sensing image slice samples x with a picture size of 256×256 are put into the encoder as input, and the cloud element distribution samples G(x) generated according to the original image features are obtained. Correspondingly, the real cloud element distribution samples obtained by manual annotation of sample x are recorded as x, and the two types of distribution samples are represented by binary images of the same size as the original image slices, where cloud elements are used as positive samples and represented as 1 in the image, and non-cloud elements are used as negative samples and represented as 0 in the image. The decoder contains a basic upsampling module, and a multi-head attention module is added to the upsampling module to strengthen the attention to cloud details. The decoder formula is expressed as

[0042] x upsample =upsample(g,scale=2) (1)

[0043] output = conv(concat(x soft-attention ,x0)) (2) Thus, a complete MA_UNet model structure is obtained. The application process of this model includes two parts: training and extraction. The training part includes two links: sample set production and sample training.

[0044] 2: Like other deep learning models, the MA_UNet model needs to undergo a large amount of sample training before it can be applied in actual scenarios, and making a sample set that meets the model structure requirements is a necessary condition for sample training. Therefore, in the present invention, for the second step, the encoding and decoding network model G can be specifically trained according to the following steps to obtain a network model G with a more ideal feature extraction effect:

[0045] Step 1: The loss function of the network is the Dice loss function. The Dice loss function is an evaluation indicator and loss function commonly used in image segmentation tasks. It is used to measure the similarity between the predicted result and the true label. It is based on the Dice coefficient, and the relationship between the two is 1-DiceCoefficient. Assume that the predicted cloud feature is Y pred , the real cloud feature label is Y true , the Dice loss can be expressed as Among them, ∑ represents the summation operation, and ∈ is a very small positive number used to stabilize the numerical value to avoid the denominator being zero. During the calculation process, both the predicted cloud elements and the actual cloud element labels are regarded as binary images, where the pixel values ​​are 0 or 1. The numerator calculates the intersection of the predicted result and the actual label, and the denominator calculates the union of the predicted result and the actual label. The value range of the Dice loss function is between 0 and 1. The closer the value is to 1, the more similar the predicted result is to the actual label, and the closer the value is to 0, the greater the difference between the predicted result and the actual label. When using the Dice loss function for model training, the goal is to minimize the Dice loss, that is, to maximize the overlap between the predicted result and the actual label by adjusting the model parameters.

[0046] Step 2: Set the optimization goal as P data(x) Indicates that the distribution of all remote sensing image slices x is a real sample, and L represents the loss function for the overall data distribution during training. Where G(x) is the distribution of generated samples obtained by model G;

[0047] Step 3: Input the labeled image slice samples into the network G, according to Calculate the model loss.

[0048] Step 4, perform BP back propagation operation on the loss obtained by performing forward propagation operation on the network G in step 4; optimize the network parameters;

[0049] Step 5, repeat steps 3 to 4, train the network, optimize its network parameters, until the loss of the network G gradually stabilizes, and output the network model G at this time.

[0050] In a specific implementation of the present invention:

[0051] Some slices in Resource No. 3 are used as sample data sources to extract cloud features.

[0052] refer to Figure 4 The data source slices shown in the figure are labeled for the sample slices in each scene image in ArcGIS software to form the sample real road element data distribution sample x, that is, the remote sensing image slice, such as Figure 5 As shown in Figure 2. According to the model requirements, 5266 256×256 resolution remote sensing image slices covering various types of terrain and containing cloud elements are selected as training data x. In order to enable the model to obtain better training results, when selecting samples, the ratio of positive and negative samples in the slice area should not differ too much.

[0053] Therefore, the present invention is trained according to the above samples, and features are extracted through MA_UNet model training to obtain a model G that can accurately identify cloud features. The purpose of sample training is to enable the model to optimize its own structure during a large number of learning processes, and finally obtain a generator with good adaptability and accuracy to generate cloud elements based on remote sensing images. In the actual training process, it is necessary to continuously adjust the parameters according to the data set. The specific process of model training is as follows:

[0054] Step 1: Set the loss function: The loss function is used to evaluate the changing trend of the gap between the generated data and the actual data during the training process, so as to reflect whether the training optimization of the model is adjusting towards the desired goal. First, refer to the standard U-Net model to set the loss function for the training process as shown in Formula 3:

[0055]

[0056] During the calculation process, both the predicted cloud elements and the true cloud element labels are treated as binary images, where the pixel values ​​are either 0 or 1. The numerator calculates the intersection of the predicted result and the true label, and the denominator calculates the union of the predicted result and the true label. The value range of the Dice loss function is between 0 and 1. The closer the value is to 1, the more similar the predicted result is to the true label, and the closer the value is to 0, the greater the difference between the predicted result and the true label. When using the Dice loss function for model training, the goal is to minimize the Dice loss, that is, to maximize the overlap between the predicted result and the true label by adjusting the model parameters.

[0057] Step 2: Set the optimization goal as P data(x) Indicates that the distribution of all remote sensing image slices x is the real sample, and E represents the loss function for the overall data distribution during the training process. Where G(x) is the distribution of generated samples obtained by model G;

[0058] Step 3: Modify the hyperparameters according to the MA_UNet model: Adapt the learning rate and optimizer to the MA_UNet for the Resource 3 dataset.

[0059] Step 4: Forward propagation to calculate loss: Input the labeled sample training set into the network and calculate the model loss according to Formula 1.

[0060] Step 5: Back propagation optimization network: Perform BP back propagation training model according to the forward propagation loss to optimize network parameters.

[0061] Step 6: Repeat the training until the optimization goal is achieved: Repeat the training process in steps 4 and 5 until the optimization result or the gradient of the model loss function no longer changes. At this time, a model that can generate high-precision cloud elements from remote sensing images is obtained. The test results are as follows: Figure 6 The cloud features extracted by the model are basically consistent with the real cloud conditions.

[0062] The deep learning and other artificial intelligence technologies on which the present invention is based use a multi-layer network to learn the features of cloud pixels at different levels of remote sensing images, integrate multi-level processing into the internal model optimization, and combine the advantages of multi-head attention modules and convolutional neural networks to improve the recognition ability of cloud extraction and enhance the model's perception of global information. The multi-head attention module can obtain inter-class distance information and strengthen the control of important features, so that the model has better spatial adaptability. This technology reduces the links of manual intervention and makes full use of various elements within high-resolution images. On the premise of having sufficient sample quantity support, the feature extraction model provided by the present invention can also obtain strong migration application capabilities, which well copes with the shortcomings of traditional methods in image preprocessing and recognition accuracy.

[0063] Those skilled in the art can understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for extracting cloud features from high-resolution images, characterized in that the steps include: The first step is to obtain a remote sensing image and slice it according to a fixed size to obtain a remote sensing image slice x; The second step is to slice the remote sensing image into the trained network G for forward propagation operation to obtain the cloud features therein and output the cloud features; In the second step, the network G is trained according to the following steps: Step 1: The loss function of the network is the Dice loss function. The Dice loss function is an evaluation indicator and loss function commonly used in image segmentation tasks. It is used to measure the similarity between the predicted result and the true label. It is based on the Dice coefficient, and the relationship between the two is 1-DiceCoefficient. Assume that the predicted cloud feature is Y pred , the real cloud feature label is Y true , the Dice loss can be expressed as Among them, ∑ represents the summation operation, ∈ is a very small positive number used to stabilize the value to avoid the denominator being zero; in the calculation process, the predicted cloud elements and the actual cloud element labels are regarded as binary images, in which the pixel value is 0 or 1; the numerator calculates the intersection of the predicted result and the actual label, and the denominator calculates the union of the predicted result and the actual label; the value range of the Dice loss function is between 0 and 1, the closer the value is to 1, the more similar the predicted result is to the actual label, and the closer the value is to 0, the greater the difference between the predicted result and the actual label; when using the Dice loss function for model training, the goal is to minimize the Dice loss, that is, to maximize the overlap between the predicted result and the actual label by adjusting the model parameters; Step 2: Set the optimization goal as P data(x) Indicates that the distribution of all remote sensing image slices x is a real sample, and L represents the loss function for the overall data distribution during training. Where G(x) is the distribution of generated samples obtained by model G; Step 3: Input the labeled image slice samples into the network G, according to Calculate model loss; Step 4, perform BP back propagation operation on the loss obtained by performing forward propagation operation on the network G in step 4; optimize the network parameters; Step 5, repeat steps 3 to 4, train the network, optimize its network parameters, until the loss of the network G gradually stabilizes, and output the network model G at this time.

2. The method for extracting cloud features in high-resolution images according to claim 1, characterized in that: In the second step, the parameters are initialized to construct the network G, and the training of steps 2 to 5 is performed. The network parameters are updated by back-propagating the error to generate the final model G, and the forward propagation operation is performed to obtain the cloud features.

3. A method for extracting cloud features in high-resolution images according to claim 1, characterized in that: Model G uses an encoding-decoding structure, the goal of which is to retain the information of the input data as much as possible and reconstruct an output similar to the original input data during the decoding process; through the optimization of the loss function during the training process, the encoding-decoding structure can learn effective data representation and reconstruction capabilities; the encoding-decoding network form is y=D(E(x)), where E(x) is the encoder function.

4. The method for extracting cloud features in high-resolution images according to claim 3, characterized in that: In step 1, the ratio of positive samples to negative samples in the image slice sample x is close.

5. The method for extracting cloud features in high-resolution images according to claim 3, characterized in that: The image slice sample x or the remote sensing image slice has not been subjected to color uniformity processing.

6. The method for extracting cloud features in high-resolution images according to claim 5, characterized in that: The image slice samples x or the remote sensing image slices are binary images of equal size.

Images