Satellite remote sensing image cloud and snow detection method based on hybrid feature network

Abstract

The application discloses a satellite remote sensing image cloud and snow detection method based on a mixed feature network and belongs to the technical field of satellite remote sensing image processing and application. The application aims to solve the problem of low cloud and snow detection precision under the condition of coexistence of cloud and snow in satellite remote sensing images and the problem of inability to accurately obtain details of snow and cloud distribution states. The application uses satellite remote sensing images as a data source, uses the GLCM, LBP and SEEDS methods to extract shallow features of the images, extracts deep heat map features of the images through a deep feature extraction network, then fuses the shallow features and the deep heat map features to form mixed features, simultaneously sends the mixed features into a cloud and snow detection network based on the mixed features for training and verification, and finally obtains cloud and snow detection results by applying a trained cloud and snow detection model. The method can accurately detect cloud and snow in satellite remote sensing images under the condition of coexistence of cloud and snow and in satellite remote sensing images with different spatial resolutions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of satellite remote sensing image processing and application. BACKGROUND

[0002] Snow, as a major component of the cryosphere, plays an important role in the earth's climate system. Obtaining accurate snow information is of great significance to climate change research and water cycle and hydrological resources investigation. Optical remote sensing satellites have been used to detect snow conditions, and in the study of optical remote sensing snow monitoring, the presence of clouds has brought great interference to snow dynamic monitoring. With the continuous improvement of optical data availability, cloud detection methods based on optical remote sensing images have been proposed. At present, the main methods are rule-based cloud detection methods and deep learning-based cloud detection methods.

[0003] Rule-based methods use the physical properties of clouds, such as surface spectral threshold, texture features, shape features, etc., to detect clouds from images. This kind of rule-based method is usually used to process specific types of satellite images. Threshold rule-based methods obtain the threshold range required for extracting different band cloud snow on the basis of traditional empirical values or a large number of statistical values. Texture rule-based methods are often used for cloud detection. Because the texture features of various objects are different, cloud can also be detected according to texture features. Ghosh et al. proposed a threshold-based cloud detection technology to extract clouds from NOAA-AVHRR data with time series. (See Ghosh RR, Ali MS, Hena A, Rahman H (2015) A simple cloud detection algorithm using NOAA-AVHRR satellite data. Int J Sci Eng Res 3(6)); Zhang proposed a method for high-precision detection of high-resolution RSI clouds using multiple shallow features such as color, texture, shape, etc. (See Zhang J, Zhou Q, Shen X, Li Y (2019) Cloud detection in high resolution remote sensing images using multi-features of ground objects. J Geovis Spat Anal 3:14);

[0004] In recent years, deep learning methods have been widely used in cloud feature extraction and cloud detection by using different size convolution kernels to extract deep and high-level abstract features from input images. Wieland proposed a cloud extraction method based on improved U-Net, namely MU-net, to realize the detection of clouds. (See M. Wieland, Y. Li, and S. Martinis, “Multi-sensor cloud and cloud shadow segmentation with a convolutional neural network,” Remote Sens. Environ., vol. 230, Sep. 2019, Art. no. 111203.) Chai improved SegNet (MSegNet) based on deep convolutional neural network to realize the extraction of cloud and shadow from Landsat image. (See D. Chai, S. Newsam, H. K. Zhang, Y. Qiu, and J. Huang, “Cloud and cloud shadow detection in Landsat imagery based on deep convolutional neural networks,” Remote Sens. Environ., vol. 225, pp. 307-316, May 2019.)

[0005] So far, scholars at home and abroad have proposed many cloud detection methods, but there are still some obvious defects: (1) The detection accuracy of thin clouds and irregular clouds is low. (2) In the areas similar to cloud spectral reflectance (such as snow and ice), it is difficult to detect clouds. (3) When the near-infrared band and short-wave infrared band are lacking, the accuracy of cloud snow detection will also be reduced. SUMMARY

[0006] In order to solve the problems that the existing cloud detection methods cannot obtain high-precision cloud snow detection results under the condition of cloud snow coexistence, and cannot realize accurate cloud snow detection in various different spatial resolution multispectral remote sensing image data and natural images. The present application uses the methods of gray level co-occurrence matrix (GLCM), local binary pattern (LBP) and energy-driven sampling method (SEEDS) to extract the shallow features of the image, extracts the deep heat map features of the image through the deep feature extraction network, then fuses the shallow features and the deep heat map features to form the hybrid features, and at the same time, the hybrid features are sent into the cloud snow detection network based on hybrid features (CSD-HFnet) for training and verification, finally the detection results of cloud and snow in satellite remote sensing image are obtained.

[0007] The technical scheme adopted by the present application is as follows:

[0008] The satellite remote sensing image cloud and snow detection method based on the hybrid feature network has the following specific steps:

[0009] Step one, image preprocessing: the satellite remote sensing image is radiometrically calibrated, atmospherically corrected, spliced and cropped to select the experimental area.

[0010] (a) The method of radiometric calibration: radiometric calibration is the process of converting the digital quantization value (DN) of the image into physical quantities such as radiance, reflectivity or surface temperature. Radiometric calibration parameters are generally placed in the metadata file. The radiometric calibration tool (Radiometric Calibration) of the complete remote sensing image processing platform (ENVI) can automatically read the parameters from the metadata file, thereby completing the radiometric calibration.

[0011] (b) The method of atmospheric correction: the atmospheric correction tool (FLASSH) of the complete remote sensing image processing platform (ENVI) is used to eliminate the radiation error caused by atmospheric influence and to retrieve the real surface reflectivity of the ground object.

[0012] (c) The method of MOD09GA data splicing: the MODIS Reprojection Tool (MRT) software is used to splice different strip number data to form complete data covering the experimental area.

[0013] (d) Image cropping is used to obtain the research area of interest (ROI).

[0014] Step two, extract the shallow feature: the gray level co-occurrence matrix (GLCM), local binary pattern (LBP) and superpixel segmentation (SEEDS) method are used to obtain the shallow feature of the image.

[0015] (a) Extract the GLCM feature of the image

[0016] The GLCM texture feature can count the gray scale information contained in the image and can calculate the frequency of each gray scale information. The advantage of GLCM texture feature is that it can accurately predict and reflect the comprehensive information of image gray scale direction, adjacent interval and change range, etc. The GLCM features mainly include mean, standard deviation, energy, contrast, entropy, inverse difference moment, homogeneity and difference.

[0017] The formula is as follows:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] where (i,j) represents the position of the image pixel pair, N is the image size, p(i,j) represents the frequency of the pixel pair at a specific position, and (i-j) represents the gray level difference of the pixel pair.

[0027] Mean reflects the regularity of the texture, and the value is small when the image texture is chaotic, and the value is large when the regularity is strong and easy to describe. STD measures the dispersion degree of the GLCM element value with the mean value as the center. ASM can be used to reflect the uniformity of the gray scale distribution in the image region. The coarse texture energy value is large, and the fine texture is small. CON reflects the definition of the image and the depth of the texture groove. The larger the contrast, the deeper the groove of the texture, and the clearer the effect. The smaller the contrast, the shallower the groove of the texture, and the more blurred the effect. ENT reflects the randomness of the image content, and reflects the information amount and complexity of the image. If the texture is complex, the entropy value is large, and vice versa. If the gray scale in the image is uniform, the texture is simple, and the entropy value is small. IDM reflects the smoothness of the image distribution. The smaller the uniformity value, the finer the texture; the larger the uniformity value, the coarser the texture; HOMO measures the uniformity of the local intensity of the gray scale image; and DIS describes the local change of the image.

[0028] (b) extracting the LBP feature of the image

[0029] LBP is a gray scale invariant texture feature extraction method. It has the characteristics of rotation invariance, good robustness to illumination change, and extreme sensitivity to noise. The LBP operator takes the difference between the gray scale values of the adjacent 8 pixels and the center pixel. If the value of the adjacent pixel is greater than that of the center pixel, the value of the pixel is marked as 1, otherwise it is marked as 0, and then matrix encoding is performed to convert the 8-bit binary number to a decimal number as the LBP value of the center pixel. LBP can be calculated by the following formula.

[0030]

[0031] where (x c ,y c ) represents the center pixel point, h C represents the gray value of the center pixel point. The circular region with the pixel point (x c ,y c ) as the center and R as the radius is defined as the neighborhood of the center pixel point, N represents the number of pixels in the field, and h n is the gray value of the neighborhood pixel point.

[0032] (c) Extracting SEEDS features of the image

[0033] The SEEDS algorithm divides the image with a regular grid. For the SEEDS algorithm, the grid is the initial superpixel, and then gradually moves the edges of the current superpixel to make the pixels inside the superpixel have as uniform color features as possible, thereby quickly completing the segmentation. SEEDS defines an energy function as the objective function to measure the value of each superpixel.

[0034] E(s) = H(s) + γG(s) … … … … … … … … … … (10)

[0035] where H(s) is a color expression representing the color information of the superpixel. The more uniform the color in the superpixel, the more concentrated the color distribution, and the larger the value of H(s); G(s) is a shape expression used to represent the regularity of the superpixel; and γ is an adaptive coefficient used to adjust the weight of the color expression and the shape expression.

[0036] The specific steps of the SEEDS algorithm are as follows:

[0037] (1) Pre-segment the image using K grids. K is the number of superpixels finally segmented. It is stipulated that the kth superpixel region is represented by A k .

[0038] (2) Calculate the color expression term H(s). First, establish a color histogram for each superpixel region, divide the color space into several regions, and each region is represented by a box in the histogram, denoted as H j . Define as the distribution of pixels in A k in the color histogram, as shown in equation (11).

[0039]

[0040] where I(i) represents the color of pixel i; Z is a normalization factor to normalize the equation; δ() is an indicator function that takes the value 1 when the color of pixel i falls in bin j of the color histogram. Define another function to measure the color distribution in each superpixel region. As shown in equation (12).

[0041]

[0042] The value is larger when the pixels in a superpixel region are more concentrated in the color histogram, and reaches the maximum value of 1 when all the pixels are concentrated in the same bin of the color histogram. Based on this, the sum of all superpixel color terms can be obtained, as shown in equation (13).

[0043]

[0044] (3) Calculate the edge expression term G(s). G(s) helps to generate edges that fit the target contour by penalizing the local irregularity between superpixel edges. First, divide the image with an N x N grid, and then build a histogram for each grid. Each histogram has K bins, representing K superpixels. Then count the distribution of pixels in each grid in the histogram, and define N i to represent the grid containing pixel i, then the histogram can be represented by equation (14).

[0045]

[0046] To make the superpixels have better and smoother edges, the grid containing only one superpixel should be as many as possible. Therefore, define the edge term G(s) by equation (15).

[0047]

[0048] When the pixels in the grid N i reach the maximum value of G(s) when the pixels in the grid belong to only one superpixel.

[0049] (4) Iterative implementation of superpixel segmentation. In each iteration, select the superpixel with the lowest energy, and move the pixels near the edge into the neighboring superpixel to make the color distribution more concentrated. Finally, set the number of iterations.

[0050] Step three, extracting depth heat map features: the application obtains the depth heat map features of the image by constructing a deep feature extraction network, and realizes the preliminary distinction between clouds and snow under the condition of cloud and snow coexistence. The extraction of depth heat map features consists of two parts: first, the deep feature extraction network extracts the depth features of the image, obtains the deep abstract features of clouds, snow and other ground objects, and then obtains the depth heat map features of the image through the Heatmap module. Through the features, clouds, snow and other ground objects can be effectively distinguished.

[0051] (1) Obtain depth features: the deep feature extraction network adopts a residual neural network structure, the first part of the residual neural network is composed of a convolution layer, a BN layer, a Relu layer and a Maxpool in series, and the purpose is to compress the image. The second part is four construction layers, each construction layer is composed of a CONV BLOCK module and multiple IDENTITY BLOCK modules in series. The IDENTITY BLOCK module is a residual block containing a group of three convolution layers connected in series and a Relu layer as a residual line. The input and output channel numbers of the IDENTITY BLOCK module are the same, and the purpose is to deepen the network and realize the extraction of deep features; the CONV BLOCK module is a residual block formed by a group of three convolution layers connected in series and a group of convolution layers and Relu connected in parallel. Among them, a group of convolution layers and Relu connected in series is taken as a residual line. The input and output channel numbers of the CONV BLOCK module are not the same, and the purpose is to change the size and channel number of the image. The purpose of combining multiple CONV BLOCK and IDENTITY BLOCK is to change the size and channel number of the image, and the step is used as the distinction. When the step is 1, the image size is not changed and only the image channel number is changed. When the step is not equal to 1, not only the image size is changed, but also the image channel number is changed.

[0052] (2) Obtain the depth heat map feature: the depth feature is obtained through the depth feature extraction network in step three (1), but in the back end of the depth feature extraction network, the 2D average pooling layer, the full connection layer and the Heatmap layer are used to obtain the heat map feature of the depth feature, and the feature size is self-adaptive to be consistent with the input image size. The heat map feature is obtained through the Heatmap module, which can simply aggregate a large amount of data and intuitively display spatial data. Many target detection tasks are carried out through the heat map feature, which can visualize the process of extracting the depth feature and make it easier to understand the network for cloud and snow detection. The Heatmap module actually matches the features in the spatial dimension, and uses the convolution kernel to slide on the feature map plane, so as to pay more attention to and use local information. The Heatmap is calculated one by one through the use of the convolution kernel, and then it is judged whether each local information meets the characteristics of the key point. The larger the calculated response is, the more likely it is a key point. Finally, the depth heat map feature consistent with the size of the input image can be obtained at the output of the depth feature extraction network. With the help of the depth heat map feature, the difficulty of cloud and snow detection can be reduced.

[0053] Step four, feature fusion and cloud and snow detection: the shallow layer feature and the deep heat map feature are fused to form a mixed feature, and the mixed feature cloud and snow detection network is trained and verified. In the case of cloud and snow coexistence, the trained cloud and snow detection network model is applied to cloud and snow extraction of remote sensing data images with different spatial resolutions.

[0054] The mixed feature cloud and snow detection network mainly consists of a feature fusion layer, four LSTM modules and a random forest classifier. First, the shallow layer feature obtained by the shallow layer feature extraction module, the deep heat map feature obtained by the deep layer feature extraction network and the original data band are combined to form the cloud and snow detection mixed feature through the use of the CONCAT function in the feature fusion layer of the cloud and snow detection network. Then the mixed feature is sent to the mixed feature cloud and snow detection network with the best number of units for training and verification to extract cloud and snow. The structure of each LSTM module mainly includes n LSTM units and dropout, where n represents the depth of the LSTM. The n LSTM units are used to extract the high-level nonlinear information available in the mixed feature. The last LSTM module contains a full connection layer at the output end to improve the generalization ability. The last layer of the cloud and snow detection network is the random forest classifier, which contains three neurons corresponding to three categories (cloud, snow and other), and finally the cloud and snow detection results of different remote sensing images can be obtained.

[0055] The beneficial effects of the present application are:

[0056] The present application adopts the method of extracting superpixel by using the gray level co-occurrence matrix, local binary pattern and energy-driven sampling to extract the shallow features of the image, and the shallow features can protect the texture information of cloud and snow. The deep heat map features of the image are extracted by the deep feature extraction network, and the deep heat map features extracted by the deep feature network are a kind of deep and abstract features, which can preliminarily distinguish the easily confused cloud and snow. Then the shallow features and the deep heat map features are fused to form mixed features, and the mixed features are sent into the cloud and snow detection network for training and verification, and finally the detection results of cloud and snow in the satellite remote sensing image are obtained. The method of the present application can accurately detect cloud and snow under the condition of coexistence of cloud and snow, and can realize cloud and snow detection in remote sensing images with different spatial resolutions, and provide reliable analysis data for snowmelt runoff analysis, agricultural irrigation and ecological environment protection. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is the research data used by the present application.

[0058] Figure 2 is a flow chart of the satellite remote sensing image cloud and snow detection method based on the mixed feature network of the present application.

[0059] Figure 3 is an example diagram of the shallow features obtained by using the gray level co-occurrence matrix of the present application.

[0060] Figure 4 is an example diagram of the shallow features obtained by using the local binary pattern of the present application.

[0061] Figure 5 is an example diagram of the shallow features obtained by using the SEEDS superpixel segmentation of the present application.

[0062] Figure 6 is a structure diagram of the deep feature extraction network used by the present application.

[0063] Figure 7 is an example diagram of the deep heat map features obtained by the present application.

[0064] Figure 8 is a structure diagram of the mixed feature cloud and snow detection network used by the present application.

[0065] Figure 9 is a cloud and snow detection result diagram obtained by using the satellite remote sensing image cloud and snow detection method based on the mixed feature network of the present application. DETAILED DESCRIPTION

[0066] The technical solutions of the present application will be further explained and described in the form of specific embodiments.

[0067] The experimental data are 1 Landsat 8 satellite data on January 27, 2019, 1 MOD09GA satellite data on February 12, 2019, 1 FY3D-MERSI satellite data on February 12, 2019, 1 GF satellite data from the GF satellite cloud and snow detection data set, and 1 ZY-3 satellite data from the ZY-3 satellite cloud and snow detection data set Figure 1 The detailed parameters of each experimental data are shown in Table 1.

[0068] Table 1

[0069] Satellite data Experimental bands Spatial resolution / temporal resolution rate Landsat 8 R, G, B, NIR, SW1, SW2 30m / 16day FY3D MERSI R, G, B, NIR 250m / 1day MOD09GA R, G, B, NIR, SW1, SW2 500m / 1day GF R, G, B 1m / 4day ZY-3 R, G, B 2m / 5day

[0070] The cloud and snow detection method based on the mixed feature network provided in the present application extracts the shallow features of the experimental data using the gray level co-occurrence matrix, the local binary pattern and the energy-driven sampling method to extract the superpixel method, and then extracts the deep heat map features of the image through the deep feature extraction network. The cloud and snow detection network first fuses the shallow features and the deep heat map features to form mixed features, and at the same time, the mixed features are sent into the cloud and snow detection network for training and verification. Finally, the cloud and snow detection results of the Landsat 8 satellite image, the MOD09GA satellite image, the FY3D-MERSI satellite image, the GF satellite image and the ZY-3 satellite image are obtained, and the cloud and snow detection accuracy of the higher resolution image corresponding to the satellite data is verified. The overall accuracy of the cloud and snow detection result obtained by the present application can reach 95.36%. Referring to the overall flowchart of the present example Figure 2

[0071] Step 1: Image preprocessing

[0072] Radiometric calibration, atmospheric correction, splicing and cutting of Landsat 8 satellite remote sensing image, MOD09GA spectral reflectance image, FY3D-MERSI satellite image, GF satellite image and ZY-3 satellite image are carried out.

[0073] (a) Radiometric calibration method: Radiometric calibration is a process of converting digital quantization value (DN) of the image into physical quantities such as radiance value, reflectivity or surface temperature. Radiometric calibration parameters are generally placed in the metadata file. The radiometric calibration tool (Radiometric Calibration) of the complete remote sensing image processing platform (ENVI) can automatically read the parameters from the metadata file, so as to complete the radiometric calibration.

[0074] (b) Atmospheric correction method: The atmospheric correction tool (FLASSH) of the complete remote sensing image processing platform (ENVI) is used to eliminate the radiation error caused by atmospheric influence and to retrieve the real surface reflectivity of the ground object. ​

[0075] (c) The method of splicing MOD09GA data: the splicing of data with different band numbers is realized by using MODIS Reprojection Tool (MRT) software to form complete data covering the experimental area.

[0076] (d) Image cropping is used to obtain the research area of interest (ROI).

[0077] Step two, extracting shallow features:

[0078] The shallow features of the experimental data are obtained by methods such as the gray level co-occurrence matrix, local binary pattern and superpixel segmentation. The shallow features can provide basic information of the image, such as the texture and shape of the ground objects, which provides basic information for cloud and snow detection. Eight gray level co-occurrence matrix texture features (mean, standard deviation, energy, contrast, entropy, inverse difference moment, homogeneity and difference) are obtained through the gray level co-occurrence matrix; the LBP features of the image are obtained through the local binary pattern; the shape features of the image are obtained through the SEEDS superpixel segmentation, which can protect the shape information of the clouds and snow and other objects, wherein the number of superpixels (K value) is set to 2000, the network N×N is set to 5×5, and the iteration number is set to 10 by default. Figure 3 Figure 4 Figure 5

[0079] Step three: extracting deep heat map features:

[0080] The deep heat map feature is an abstract feature, which can make the distinction between clouds and snow more obvious and reduce the difficulty of detecting clouds and snow. The deep heat map features of the image are obtained by constructing a deep feature extraction network, so as to preliminarily distinguish clouds and snow under the condition of coexistence of clouds and snow. The extraction of deep heat map features consists of two parts: first, the deep feature extraction network is used to extract the deep features of the image, so as to obtain the deep abstract features of clouds and snow and other objects; then, the Heatmap module in the deep feature extraction network can obtain the deep heat map features of the image, which can effectively distinguish clouds, snow and other objects. Figure 6 Figure 7

[0081] Step four: feature fusion and cloud and snow detection:

[0082] The shallow features and the deep heat map features will provide effective information for the detection of clouds and snow. First, the shallow features and the deep heat map features are fused to form a hybrid feature, and then the hybrid feature cloud and snow detection network is trained and verified. In the case of coexistence of clouds and snow, the trained hybrid feature cloud and snow detection network model is applied to different spatial resolution remote sensing data images for cloud and snow extraction.

[0083] ​​​​​The mixed feature cloud and snow detection network mainly consists of a feature fusion layer, four LSTM modules and a random forest classifier. Figure 8 Firstly, the shallow features obtained by the shallow feature extraction module, the deep heat map features obtained by the deep feature extraction network and the original data bands are combined into cloud and snow detection mixed features by the CONCAT function in the feature fusion layer of the cloud and snow detection network. Then the mixed features are sent to the multi-feature LSTM cloud and snow detection network with the best number of units for training and verification, and the trained network model is applied to realize cloud and snow detection. The structure of each LSTM module mainly includes n LSTM units and a dropout unit, where n represents the depth of LSTM, i.e. the number of mixed features. For Landsat 8 satellite data, n is 17, for FY3D-MERSI satellite data, n is 15, and for high-resolution satellite and resource satellite data, n is 14. The use of n LSTM units is to extract the available high-level nonlinear information in the mixed features. The last LSTM module contains a fully connected layer at the output end to improve the generalization ability. The last layer of the cloud and snow detection network is a random forest classifier, which contains three neurons corresponding to three categories (cloud, snow and other). Based on the mixed feature cloud and snow detection network, the learning rate is set to 0.001, the dropout rate is set to 0.5, and the number of iterations is set to 10000. Finally, the cloud and snow detection results of Landsat 8 satellite data, MOD09GA satellite data, FY3D-MERSI satellite data, high-resolution satellite data and resource satellite data can be obtained. Figure 9

[0084] The present application extracts different shallow features such as texture features and shape features of images by using the gray level co-occurrence matrix, local binary pattern and energy-driven sampling method to extract superpixel, and simultaneously uses deep learning network to realize image deep heat map feature extraction, feature fusion and cloud and snow detection. The shallow features can protect the basic ground object information of the image, the deep learning feature extraction network can obtain deep abstract information, and the present application can consider the basic information and abstract information of the ground object by mixing the shallow features and the deep heat map features on the basis of the original satellite data bands, thereby improving the cloud and snow detection precision under the cloud and snow coexistence condition. The present application can finally obtain the cloud and snow detection results in different spatial resolution satellite data by using the mixed feature cloud and snow detection network sensitive to feature changes, thereby providing effective reference data for realizing dynamic snow monitoring. The overall precision of the cloud and snow detection results obtained by the present application compared with the real verification data can reach 95.36%

[0085] ​Experimental results: In order to illustrate the accuracy of the cloud and snow detection method based on the mixed features, satellite data with higher spatial resolution than the experimental data are selected as verification data to verify the accuracy of the cloud and snow detection results obtained by the application. The application can mix deep thermal map features and shallow features in multiple layers and non-linearly to represent cloud, snow, thin cloud and irregularly shaped cloud features, and can realize high-precision cloud and snow detection. At the same time, the application can effectively and accurately detect cloud and snow in satellite data with different spatial resolutions, and provide real and reliable data for snow research and hydrological analysis.

Claims

1. A method for detecting clouds and snow in satellite remote sensing images based on hybrid feature networks, characterized in that, The specific steps of this method are as follows: Step 1: Image preprocessing: Radiometric calibration, atmospheric correction, mosaicking, and cropping of satellite remote sensing images to select experimental areas; Step 2: Extract shallow features: Obtain shallow features of the image through gray-level co-occurrence matrix, local binary mode, and superpixel segmentation methods; Step 3: Extracting Depth Heatmap Features: This step consists of two parts. First, a deep feature extraction network extracts the image's depth features to obtain deep abstract features of clouds and snow. Then, the Heatmap module generates the image's depth heatmap features. The deep feature extraction network adopts a residual neural network structure. The heatmap module is connected to the deep feature extraction network and consists of a 2D average pooling layer, a fully connected layer and a heatmap layer. Step 4: Feature Fusion and Cloud / Snow Detection: The shallow features and deep heatmap features are fused to form a hybrid feature, and the hybrid feature cloud / snow detection network is trained and validated. In the case of coexistence of clouds and snow, the trained cloud / snow detection network model is applied to extract clouds and snow from remote sensing data images with different spatial resolutions. The hybrid feature cloud and snow detection network consists of a feature fusion layer, four LSTM modules, and a random forest classifier. First, the feature fusion layer of the cloud and snow detection network uses the CONCAT function to combine the shallow features obtained by the shallow feature extraction module, the deep heatmap features obtained by the deep feature extraction network, and the original data bands to form a hybrid feature for cloud and snow detection. Subsequently, the mixed features are sent to a mixed feature cloud and snow detection network with a multi-feature LSTM having the optimal number of units for training and validation to extract clouds and snow. The structure of each LSTM module includes n LSTM units and dropout. The last LSTM module contains a fully connected layer at the output. Finally, a random forest classifier is used to classify the results into three categories: clouds, snow, and others, to obtain the cloud and snow detection results of the remote sensing image.

2. The method for detecting clouds and snow in satellite remote sensing images based on a hybrid feature network according to claim 1, characterized in that, In step one, the image preprocessing steps are as follows: (a) Radiometric calibration method: Radiometric calibration is the process of converting the digital quantization value of an image into a radiance value, reflectance or surface temperature physical quantity. Radiometric calibration parameters are stored in a metadata file. The radiometric calibration tools of a complete remote sensing image processing platform can automatically read the parameters from the metadata file to complete the radiometric calibration. (b) Atmospheric correction method: Use the atmospheric correction tools of the complete remote sensing image processing platform to eliminate radiation errors caused by atmospheric effects and retrieve the true surface reflectance of ground objects; (c) Method of MOD09GA data splicing: Use MODIS Reprojection Tool software to splice data of different strip numbers to form complete data covering the experimental area; (d) Image cropping is used to obtain the region of interest.

3. The method for detecting clouds and snow in satellite remote sensing images based on a hybrid feature network according to claim 1, characterized in that, The specific steps for extracting shallow features in step two are as follows: (a) Extracting GLCM features from the image GLCM texture features can statistically analyze the grayscale information contained in an image and calculate the frequency of each grayscale value. GLCM features include Mean, Standard Deviation (STD), Energy (ASM), Contrast (CON), Entropy (ENT), Inverse Differential Moment (IDM), Homogeneity (HOMO), and Dissimilarity (DIS), as shown in the following formulas: Where (i,j) represents the position of the pixel pair in the image, N is the image size, p(i,j) represents the frequency of the pixel pair under a specific positional relationship, and (ij) represents the gray level difference of the pixel pair. (b) Extracting LBP features from the image The LBP operator takes the difference between the gray values ​​of the 8 adjacent pixels and the center pixel. If the value of an adjacent pixel is greater than the value of the center pixel, the value of that pixel is marked as 1; otherwise, it is marked as 0. Then, matrix encoding is performed to convert the 8-bit binary number into a decimal number as the LBP value of the center pixel. LBP can be calculated using the following formula. Among them, (x c ,y c ) represents the center pixel, h C Represents the grayscale value of the center pixel; in pixels (x c ,y c A circular region centered at R and with radius R is defined as the neighborhood of the center pixel, where N represents the number of pixels within the field, and h... n The grayscale value of the neighboring pixels; (c) Extract SEEDS features from the image The specific steps of the SEEDS algorithm are as follows: (1) Use K grids to pre-segment the image; K is the number of superpixels in the final segmentation; the kth superpixel region is defined as A. k express; (2) Calculate the color representation term H(s); First, build a color histogram for each superpixel region, dividing the color space into several regions. Each region is represented by a bin in the histogram, denoted as H. j ;definition As A k The distribution of the middle pixels in the color histogram is shown in formula (11); Where I(i) represents the color of pixel i; Z is a normalization factor used to normalize the formula; δ() is an indicator function that takes the value 1 when the color of pixel i falls into bin j of the color histogram; and a function ψ(C) is defined. Ak ), used to measure the color distribution in each superpixel region; as shown in Equation (12); The more concentrated the pixels in the superpixel region are in the color histogram, the larger the value. When all the pixels are concentrated in the same bin of the color histogram, the value reaches the maximum, which is 1. Based on this, the sum of all superpixel color terms can be obtained, as shown in Equation (13). (3) Calculate the edge representation term G(s); G(s) helps generate edges that fit the target contour by penalizing local irregularities between superpixel edges; first, the image is divided into N×N grids, and then a histogram is built for each grid; each histogram has K bins, representing K superpixels. Then, the distribution of pixels in each grid in the histogram is statistically analyzed, and N is defined as... i If a grid contains a pixel i, then the histogram can be represented by equation (14); In order to make the superpixel have better and smoother edges, we need as many meshes as possible that contain only one superpixel; therefore, we define the edge term G(s) using equation (15). When N i The value of G(s) reaches its maximum when a pixel in the superpixel belongs to only one superpixel. (4) Iterative implementation of superpixel segmentation; in each iteration, select the superpixel with the lowest energy and move the pixels near the edge into the neighboring superpixel to make the color distribution more concentrated; finally, set the number of iterations.

4. The method for detecting clouds and snow in satellite remote sensing images based on a hybrid feature network according to claim 1, characterized in that, Step 3 extracts features from the deep heatmap. The first part of the residual neural network consists of a convolutional layer, a BN layer, a ReLU layer, and a Maxpool layer connected in series. The second part consists of four building layers. Each building layer is composed of a CONV BLOCK module and multiple IDENTITY BLOCK modules connected in series. A CONV BLOCK module is a residual block formed by a combination of three concatenated convolutional layers and a concatenated ReLU layer, with one concatenated convolutional layer and ReLU layer serving as the residual line. An IDENTITY BLOCK module is a residual block containing a combination of three concatenated convolutional layers and a ReLU layer serving as the residual line. The first build layer contains 2 IDENTITY BLOCK modules, the second build layer contains 3 IDENTITY BLOCK modules, the third build layer contains 5 IDENTITY BLOCK modules, and the fourth build layer contains 2 IDENTITY BLOCK modules.

5. The method for detecting clouds and snow in satellite remote sensing images based on a hybrid feature network according to claim 1, characterized in that, Step 4: Feature Fusion and Cloud / Snow Detection. The optimal number of units in the LSTM of the hybrid feature cloud / snow detection network is 17 for Landsat 8 satellite data, 15 for FY3D-MERSI satellite data, and 14 for Gaofen satellite and resource satellite data.

Images