Machine learning-based sand and dust intensity inversion method for static meteorological satellite
Through convolutional neural networks and stacked ensemble learning models, combined with multi-channel radiometer data and multi-source surface parameters, the problems of large errors and time lag in dust intensity monitoring in existing technologies are solved, and high-precision and continuous dust intensity distribution monitoring is achieved.
Patent Information
- Application Number
- CN202510963734.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-14
- Publication Date
- 2025-09-19
AI Technical Summary
Existing technologies in sandstorm monitoring have problems such as insufficient utilization of spectral information, poor feature generalization, and inability to quantitatively output visibility, resulting in large errors and time lags in sandstorm intensity classification. In addition, existing methods are severely interfered with by clouds, ice and snow, and surface albedo.
A convolutional neural network combined with multi-channel radiometer data is used to identify dust areas, and spectral indices and multi-source atmospheric-surface parameters are integrated to construct high-dimensional features. A stacked integrated learning model is used to output visibility and map it to dust intensity levels according to industry standards.
It has achieved high-efficiency and continuous monitoring of dust intensity distribution, improved the accuracy of dust area segmentation and the ability to correct background surface differences, and realized quantitative classification and minute-level updates of dust intensity.
Smart Images
Figure CN120670833A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of remote sensing meteorology and environmental monitoring technology, and in particular to a method for inverting dust intensity using a geostationary meteorological satellite based on machine learning. Background Art
[0002] Dust storms and blowing sand have significant impacts on traffic safety, air quality, human health, and the climate system. Industry authorities urgently need timely and accurate dust intensity monitoring products to support disaster warnings, air traffic control, and carbon budget assessments. Geostationary meteorological satellites, with their ability to continuously capture large-scale observations, are an ideal platform for real-time dust monitoring.
[0003] Existing methods mostly use (1) single-channel or dual-channel brightness temperature threshold methods, which are highly empirical and vary greatly with the underlying surface; (2) rely on aerosol optical thickness inversion, but are easily affected by cloud, ice, snow, and surface albedo; and (3) are based on polar-orbiting satellite machine learning models, which have insufficient spatiotemporal resolution. These methods, due to insufficient use of spectral information, poor feature generalization, and inability to quantitatively output visibility, result in large errors in dust intensity classification and delayed effectiveness. Summary of the Invention
[0004] In response to the many problems existing in the above-mentioned existing technologies, the present invention provides a method for inverting dust intensity from a geostationary meteorological satellite based on machine learning. The present invention uses a convolutional neural network combined with multi-channel radiometer data to identify dust areas; then, the spectral index and multi-source atmospheric-surface parameters are fused within the mask range to construct high-dimensional features; the stacked ensemble learning model is trained with visibility labels to output visibility; finally, the visibility is mapped to dust intensity levels according to industry standards to form a continuous and highly timely dust intensity distribution.
[0005] A method for inverting dust intensity from a geostationary meteorological satellite based on machine learning, comprising the following steps:
[0006] Collect multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters, and perform temporal and spatial registration on the multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters to form training samples;
[0007] The convolutional neural network model uses multi-channel radiometer observation data from geostationary meteorological satellites to identify dust areas and generate dust masks. Multiple spectral difference indices and ratio indices are calculated within the area defined by the dust mask. These indices are then combined with the normalized vegetation index, the planetary boundary layer height, and the solar altitude to generate a feature vector. This feature vector is then mapped to contemporaneous ground visibility observations using temporal and spatial coordinates to obtain dust samples with visibility labels.
[0008] A stacked ensemble learning model is trained based on dust samples with visibility labels. The stacked ensemble learning model consists of a random forest regression model, a support vector machine regression model, a gradient boosting tree regression model, and a sparse regression model, and is used to output visibility prediction values. Based on the visibility threshold in the industry dust monitoring standard, the visibility prediction value is converted into a dust intensity level.
[0009] Preferably, the multi-channel radiometer is an AGRI multi-channel radiometer carried by a geostationary meteorological satellite, and the multi-source surface and atmospheric parameters include normalized vegetation index, surface albedo, planetary boundary layer height, digital elevation, and satellite and sun observation geometric angle.
[0010] Preferably, the time registration is performed based on the satellite hourly observation time to match adjacent moments, and the spatial registration uses nearest neighbor interpolation to unify the spatial resolution of all data.
[0011] Preferably, the convolutional neural network model includes multiple convolutional layers and pooling layers, adopts ReLU activation function, and adopts Dropout method for regularization during training.
[0012] Preferably, the multiple spectral difference indices and ratio indices include a brightness temperature difference index, a ratio dust index, a normalized dust index, and a multi-channel infrared dust index.
[0013] Preferably, before generating the feature vector, the input variables are evaluated for feature importance based on Shapley values, and the variables with the highest importance ranking are selected as the input of the stacked ensemble learning model.
[0014] Preferably, when training the stacked ensemble learning model, the hyperparameters of the random forest regression model, the support vector machine regression model, and the gradient boosting tree regression model are optimized using grid search combined with K-fold cross validation.
[0015] Preferably, the gradient boosting tree regression model is implemented by the LightGBM algorithm.
[0016] Preferably, the industry sand and dust monitoring standard divides the sand and dust intensity into floating dust or blowing sand level, sandstorm level, strong sandstorm level and extremely strong sandstorm level according to the visibility prediction value.
[0017] Compared with the prior art, the advantages and beneficial effects of the present invention are:
[0018] Through the convolutional neural network dust identification technology, accurate segmentation of dust areas is achieved;
[0019] By using the feature construction technology of multi-source surface and atmospheric parameter fusion, the synchronous correction of background surface difference and atmospheric thickness is achieved;
[0020] The quantitative classification of dust intensity was achieved through stacked ensemble learning visibility inversion technology.
[0021] Through the full-process business deployment design technology, a regional sand and dust intensity monitoring service with minute-level updates is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 It is the overall flow chart of the present invention;
[0023] Figure 2 It is a schematic diagram of the processing process of the present invention;
[0024] Figure 3 This is a schematic diagram of the sandstorm intensity classification according to the present invention. DETAILED DESCRIPTION
[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0028] A method for inverting dust intensity from a geostationary meteorological satellite based on machine learning, comprising the following steps:
[0029] Collect multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters, and perform temporal and spatial registration on the multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters to form training samples;
[0030] Preferably, the multi-channel radiometer is an AGRI multi-channel radiometer carried by a geostationary meteorological satellite, and the multi-source surface and atmospheric parameters include normalized vegetation index, surface albedo, planetary boundary layer height, digital elevation, and satellite and sun observation geometric angle.
[0031] The present invention collects historical sandstorm weather process data from 2020 to 2024, and counts the time of each sandstorm weather process; sorts out satellite and ground station observation data according to the statistical time and matches them to form multi-source data such as satellite and dust intensity original data samples;
[0032] According to the time of dust occurrence, the visible light channel of FY-4's AGRI is used to generate an RGB image. The dust and non-dust areas are marked based on the RGB image to obtain the dust and non-dust areas. The channel data of AGRI is used as the input variable of the convolutional neural network (CNN), and the marked samples are used as targets for training. A dust recognition model is constructed to identify dust areas.
[0033] Collect the vegetation index query product NDVI of MERSI of FY-3D with the same history; collect the surface albedo MCD43 product of MODIS with the same history; collect the boundary layer height product of ERA5 with the same history; collect the FY-4 satellite observation and solar geometry parameters (azimuth and zenith angle) with the same history.
[0034] Based on the historical summary of sandstorm weather processes, we collected data from the FY-4 geostationary satellite, EAR5 boundary layer height data, surface albedo, satellite and solar observations, and azimuth angles during the same period. We integrated and matched these data based on space and time to construct a sandstorm intensity inversion dataset. The specific process includes:
[0035] First, the FY-4 AGRI multi-channel data was combined with a convolutional neural network (CNN) to identify dust and sand, and determine the spatial range information based on the dust and sand occurrence area.
[0036] According to the sandstorm weather process, the visibility data of each weather process station is sorted out, and the satellite and other auxiliary data of each time are extracted to complete the spatial information matching;
[0037] By integrating data that matches temporal and spatial information and statistically analyzing the extracted data based on the visibility range of the station, a raw data sample for inverting dust intensity is formed. This raw data sample provides a reliable data foundation for subsequent model construction, training, and verification.
[0038] The raw data samples of Fengyun-4 (geostationary meteorological satellite) multi-band data, dust index, and dust scene visibility were preprocessed to form a dust intensity dataset; the dust intensity dataset contains information such as the dust index, multi-channel, and boundary layer height based on the visibility of each station. The data sample format is shown in Table 1. In the machine learning process, the dataset is usually divided into a training set and a test set in a certain ratio. For example, in this embodiment, a 7:3 ratio is used to divide the dataset into a dust intensity training set and a dust intensity test set. This provides guarantees for subsequent machine learning methods to provide prediction accuracy, reduce the risk of overfitting, accelerate model training, and increase model interpretability.
[0039] Table 1 Values of various indicators under different visibility conditions in dust scenes
[0040]
[0041]
[0042] The SHAP model was used to screen the input variables and select important variables as model input variables. Finally, 20 variables were selected as model input values, some of which are shown in Table 1.
[0043] Preferably, the time registration is performed based on the satellite hourly observation time to match adjacent moments, and the spatial registration uses nearest neighbor interpolation to unify the spatial resolution of all data.
[0044] Due to the different temporal and spatial resolutions of BLH, NDDI, DEM, and surface albedo (SA) data from FY-4AGRI, GEO, and ERA05, the 15-minute resolution AGRI data is matched to the nearest hourly data. BLH is the hourly data, NDDI is the ten-day product, and SA is the daily product. All data are matched to the hourly data. All products are generated using the nearest point difference method to generate AGRI data with the same spatial resolution.
[0045] The multi-channel radiometer observation data of the geostationary meteorological satellite is input into the convolutional neural network model to identify dust areas and generate dust masks; multiple spectral difference indices and ratio indices are calculated within the area defined by the dust mask, and feature vectors are generated by combining the normalized vegetation index, planetary boundary layer height and solar altitude angle. The feature vectors are then matched with the ground visibility observations of the same period according to the time and space coordinates to obtain dust samples with visibility labels.
[0046] Preferably, the convolutional neural network model includes multiple convolutional layers and pooling layers, adopts ReLU activation function, and adopts Dropout method for regularization during training.
[0047] The CNN dust mask calculation method first uses an RGB true-color composite image to mark dust and non-dust areas. A total of 120,000 samples were collected, of which 57,432 were labeled dust and 62,568 were labeled non-dust. To improve the model's accuracy in high-albedo areas, some data were rotated. Dropout was used for regularization to prevent overfitting, and the ReLU activation function implemented a nonlinear transformation. Convolutional kernel sizes were set to 32, 64, and 128, and the loss function used was binary cross-entropy. The input sample tensor T had the size of (number of samples, length, width, number of bands). In the experiment, the length and width were both set to 3, and the number of bands was set to 14. The pooling layer size was set to (2, 2), and the iteration size was set to 30.
[0048] Preferably, the multiple spectral difference indices and ratio indices include a brightness temperature difference index, a ratio dust index, a normalized dust index, and a multi-channel infrared dust index.
[0049] The brightness temperature difference (BTD), the ratio dust index (DRI), the normalized difference dust index (NDDI), and the multi-channel infrared dust index (MIDI) were calculated using the FY-4AGRI.
[0050] BTD=BT i -BT j
[0051]
[0052]
[0053] MIDI=(BT 8.5μm +BT 12μm ) / (2×BT 10.8μm )×1000
[0054] Where R represents the band reflectance value, BT represents the brightness temperature, i represents the visible near-infrared band, j represents the thermal infrared band, BT8.5μm, BT12μm and BT10.8μm represent the brightness temperature values at the center wavelengths of 8.5μm, 12μm and 10.8μm, respectively.
[0055] Preferably, before generating the feature vector, the input variables are evaluated for feature importance based on Shapley values, and the variables with the highest importance ranking are selected as the input of the stacked ensemble learning model.
[0056] Set the number of input samples to n, the input feature dimension to m, and the input feature matrix X to be defined as: X = [x1, x2, ..., x n ] 2 , Where a single sample X i The characteristic representation is:
[0057] X i =[BTD, NDDI, DRI, ..., SA, BLH, other auxiliary features]
[0058] The SHAP model is used to evaluate the importance of different input factors, and the top 25 important factors are selected as model training factors; the optimal parameters of the machine learning model are trained and the model is automatically adjusted through GridSearch to optimize each model.
[0059] A stacked ensemble learning model is trained based on dust samples with visibility labels. The stacked ensemble learning model consists of a random forest regression model, a support vector machine regression model, a gradient boosting tree regression model, and a sparse regression model, and is used to output visibility prediction values. Based on the visibility threshold in the industry dust monitoring standard, the visibility prediction value is converted into a dust intensity level.
[0060] Ensemble learning is used to achieve visibility inversion under dust weather conditions, thereby dividing the intensity of sandstorms. Based on CNN dust recognition area, Stacking is a special ensemble learning method that uses the prediction results of multiple models as new features and inputs them into a new model to further improve the prediction accuracy. Stacking mainly includes two steps: first, multiple base models are constructed and their prediction results are used as new features, and then a new model is used to predict these new features. The base models include random forest (RF), LightGBM and support vector machine (SVM). The Stacking model uses the least absolute shrinkage and selection operator (LASSO) model to integrate the three inversion results and finally output the visibility prediction results.
[0061] Preferably, when training the stacked ensemble learning model, the hyperparameters of the random forest regression model, the support vector machine regression model, and the gradient boosting tree regression model are optimized using grid search combined with K-fold cross validation.
[0062] The stacking ensemble learning model includes random forest (RF), LightGBM, and support vector machine (SVM). The stacking model uses the least absolute shrinkage and selection operator (LASSO) model to integrate the three inversion results and finally output the visibility prediction result.
[0063] Visibility in dust-affected scenarios is inverted using three heterogeneous learners: RF, SVM, and LightGBM. The inversion process requires tuning each learner's parameters to achieve optimal performance. K-fold cross-validation is used to output the results. K-fold cross-validation is used to effectively avoid overfitting. The outputs of the three learners are synthesized into new input features and trained on a LASSO model. The Lasso model then outputs the final inverted visibility for dust-affected scenarios.
[0064] Random Forest (RF), which is the average of the outputs of each decision tree:
[0065]
[0066] Where T is the total number of decision trees, f t is the prediction function of the t-th tree.
[0067] The construction of a single decision tree and the node partitioning criteria are as follows: At each node, feature j and segmentation threshold s are selected to minimize the weighted mean square error of the node:
[0068]
[0069] Where, and are the mean visibility values of the left and right nodes respectively.
[0070]
[0071] Where T is the number of trees, is the predicted value of the t-th tree.
[0072] Support Vector Machine (SVM):
[0073]
[0074] Where a i , is the Lagrange multiplier, and K is the kernel function.
[0075] Preferably, the gradient boosting tree regression model is implemented by the LightGBM algorithm.
[0076] LightGBM model, the optimized objective function combines the loss function and the regularization term:
[0077]
[0078] Where y i is the true visibility value of the i-th sample, is the predicted value of the model, T is the total number of trees, T tis the number of leaf nodes in the t-th tree, w t·j is the output weight of the jth leaf node of the tth tree, γ and λ are the number of leaf nodes and the regularization coefficient respectively.
[0079] The LightGBM visibility prediction result is the sum of all tree output results:
[0080]
[0081] Where h t (X i ) represents the t-th tree for sample X i The prediction result of , η is the learning rate that controls the contribution weight of each tree.
[0082] LASSO combines the least squares method with the L1 regularization term:
[0083]
[0084] Or simplified as:
[0085]
[0086] Where RSS represents the residual sum of squares, that is:
[0087]
[0088] Used to measure the degree of fit of the model; λ||β|| l , L1 regularization term, λ is the regularization parameter (λ≥0), It represents the sum of the absolute values of the regression coefficients.
[0089] CNN is used to identify dust areas and extract visibility inversion results of the area; according to the visibility range, the dust areas are divided according to the dust intensity classification standards.
[0090] Preferably, the industry sand and dust monitoring standard divides the sand and dust intensity into floating dust or blowing sand level, sandstorm level, strong sandstorm level and extremely strong sandstorm level according to the visibility prediction value.
[0091] The accuracy, success rate (CSI, CriticalSuccessIndex) and detection rate (Detection) of the three indicators were used to evaluate the fine precision of sand and dust identification using different methods including DSD, NDDI, DRI, MIDI and CNN (Table 2).
[0092]
[0093] Where YY means that both the satellite and ground observation points are dusty; NN means that neither the satellite nor the ground observation points are dusty; YN means that the satellite observations are dusty but the ground observations are not dusty; NY means that the satellite observations are dusty but the ground observations are dusty.
[0094] As can be seen in Table 2, CNN significantly improves dust detection accuracy compared to other algorithms, such as DSD products. CNN's accuracy, CIS, and detection rates reached 90.21%, 25.54%, and 78.75%, respectively.
[0095] Table 2 Comparison of dust identification accuracy of different algorithms
[0096]
[0097] The accuracy of the visibility inverted under different dust intensity conditions was evaluated using the correlation coefficient (R), root mean square error (RMSE) and mean absolute error (MAE). The R values of the constructed CNN_HDPF_Stacking for inverting extremely strong sandstorms, strong sandstorms, sandstorms, blowing sand or floating dust were 0.15, 0.31, 0.52 and 0.75, respectively; the RMSE values were 0.23, 0.74, 0.79 and 1.06, respectively; and the MAE values were 0.14, 0.55, 0.62 and 1.13, respectively. The test results passed the 95% confidence interval and the p<0.05 significance correlation test (Table 3). Compared with the case without CNN dust mask (Table 4), the accuracy of sandstorm level inversion was significantly improved. This shows that the CNN_HDPF_Stacking algorithm (i.e., the "stacked ensemble learning model" in the present invention) can invert the dust intensity level.
[0098] Table 3 Inversion accuracy based on CNN_HDPF_Stacking method at different dust intensity levels
[0099]
[0100] Table 4. Visibility inversion accuracy under different dust intensities without CNN mask
[0101]
[0102] The model and process of the present invention were tested using actual historical dust process data. 200 sets of dust intensity data, generated based on historical dust processes, were input into the dust intensity model. The model's predicted values were then compared with the corresponding 200 sets of actual dust intensity values. The final results showed a root mean square error of ≤0.17 km and a mean absolute error of ≤0.8 km. Table 4 shows a comparison between the first 15 sets of actual dust intensity values and the model's predictions. The test results demonstrate that the present invention is capable of fitting dust intensity distributions.
[0103] The intensity of sandstorms is classified into different levels based on the visibility range (floating dust / blowing sand: VIS < 10 km; sandstorm: VIS < 1 km; strong sandstorm: VIS < 500 m; extremely strong sandstorm: VIS < 50 m) (e.g. Figure 3 shown).
[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware.
[0105] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for inverting dust intensity from geostationary meteorological satellites based on machine learning, characterized in that: The following steps are involved: Collect multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters, and perform temporal and spatial registration on the multi-channel radiometer observation data from geostationary meteorological satellites and multi-source surface and atmospheric parameters to form training samples; The convolutional neural network model uses multi-channel radiometer observation data from geostationary meteorological satellites to identify dust areas and generate dust masks. Multiple spectral difference indices and ratio indices are calculated within the area defined by the dust mask. These indices are then combined with the normalized vegetation index, the planetary boundary layer height, and the solar altitude to generate a feature vector. This feature vector is then mapped to contemporaneous ground visibility observations using temporal and spatial coordinates to obtain dust samples with visibility labels. A stacked ensemble learning model is trained based on dust samples with visibility labels. The stacked ensemble learning model consists of a random forest regression model, a support vector machine regression model, a gradient boosting tree regression model, and a sparse regression model, and is used to output visibility prediction values. Based on the visibility threshold in the industry dust monitoring standard, the visibility prediction value is converted into a dust intensity level.
2. The method according to claim 1, characterized in that The multi-channel radiometer is an AGRI multi-channel radiometer carried by a geostationary meteorological satellite, and the multi-source surface and atmospheric parameters include normalized vegetation index, surface albedo, planetary boundary layer height, digital elevation, and satellite and sun observation geometric angle.
3. The method according to claim 1, characterized in that Temporal registration uses the satellite hourly observation time as the benchmark to match adjacent moments, and spatial registration uses nearest neighbor interpolation to unify the spatial resolution of all data.
4. The method according to claim 1, wherein The convolutional neural network model includes multiple convolutional layers and pooling layers, adopts ReLU activation function, and adopts Dropout method for regularization during training.
5. The method according to claim 1, wherein The multiple spectral difference indices and ratio indices include a brightness temperature difference index, a ratio dust index, a normalized dust index, and a multi-channel infrared dust index.
6. The method according to claim 1, characterized in that Before generating feature vectors, the feature importance evaluation based on Shapley value is used for the input variables, and the variables with the highest importance ranking are selected as the input of the stacked ensemble learning model.
7. The method according to claim 1, characterized in that When training the stacked ensemble learning model, the hyperparameters of the random forest regression model, support vector machine regression model, and gradient boosted tree regression model are optimized using grid search combined with K-fold cross validation.
8. The method according to claim 1, characterized in that The gradient boosting tree regression model is implemented by the LightGBM algorithm.
9. The method according to claim 1, characterized in that The industry sand and dust monitoring standard divides the sand and dust intensity into floating dust or blowing sand level, sandstorm level, strong sandstorm level and extremely strong sandstorm level according to the visibility forecast value.