Sand-dust identification method and device based on multi-modal neural network and storage device

Abstract

The application discloses a kind of sand and dust identification method, equipment and storage device based on multi-modal neural network, comprising the following steps: collecting multi-source data related to sand and dust;The multi-source data is preprocessed, and the processed data is obtained;According to the processed data, a sand and dust identification model training set is constructed;Sand and dust identification model is constructed;The sand and dust identification model includes backbone network, output network and fusion network;The training set is used to train the sand and dust identification model, and the final model is obtained;According to the final model, sand and dust identification is completed;Equipment and storage device are used to realize the method.The application has the beneficial effects that compared with the traditional method in the past, the sand and dust identification speed and identification accuracy have been greatly improved, and the sand and dust identification continuity has also been partially improved.

Description

Technical Field

[0001] This invention relates to the field of dust prediction, and in particular to a method, device and storage device for dust identification based on a multimodal neural network. Background Technology

[0002] The identification and detection of sudden sandstorm events has always been a bottleneck problem restricting weather forecasting.

[0003] Previous dust storm identification methods primarily relied on satellite remote sensing, combined with various dust detection indices. Specifically, they utilized the different spectral characteristics of dust particles from multiple channels to extract relevant information using a thresholding method. This involved comparing satellite remote sensing pixel values ​​to a certain threshold; values ​​exceeding the threshold were considered to represent that type of dust. However, in practical applications, determining the threshold is subject to significant uncertainties. Satellite images from different imaging times may have different threshold values, and different ground features may share the same threshold, thus the accuracy falls short of practical requirements. Summary of the Invention

[0004] To improve the accuracy of dust identification, this invention comprehensively considers various factors related to dust, including satellite data, MODIS inversion data, and ground observation data, and proposes a dust identification method based on a multimodal neural network. The method specifically includes the following steps:

[0005] S1. Collect multi-source data related to sandstorms;

[0006] S2. Preprocess the multi-source data to obtain the processed data;

[0007] S3. Construct a training set for the sand and dust identification model based on the processed data;

[0008] S4. Construct a dust identification model; the dust identification model includes a backbone network, an output network, and a fusion network;

[0009] S5. Train the dust recognition model using the training set to obtain the final model;

[0010] S6. Complete the dust identification based on the final model.

[0011] A storage device that stores instructions and data for implementing a sand and dust identification method based on a multimodal neural network.

[0012] A dust identification device based on a multimodal neural network includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a dust identification method based on a multimodal neural network.

[0013] The beneficial effects provided by this invention are: compared with the traditional methods, it has greatly improved the speed and accuracy of sand and dust identification, and has also improved the continuity of sand and dust identification to some extent. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0015] Figure 2 This is a schematic diagram of the hardware device working in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0017] Please refer to Figure 1 , Figure 1 This is a flowchart of the method of the present invention.

[0018] This invention provides a method for identifying sand and dust based on a multimodal neural network, comprising the following steps:

[0019] S1. Collect multi-source data related to sandstorms;

[0020] The multi-source data includes: imaging spectrometer inversion data, ground meteorological observation station data, and dust event image data;

[0021] As one embodiment, this invention collects Moderate Resolution Imaging Spectroradiometer (MODIS) inversion data, ground meteorological observation station data, China Meteorological Administration CLDAS data, and European Centre for Medium-Range Weather Forecasts (ERA5) data for the identified area from 2018 to 2022. It also collects dust event image data from 14 channels of the Fengyun-4 (FY-4) satellite Geostationary Radiometric Imager (AGRI) for the study area from 2018 to 2022.

[0022] S2. Preprocess the multi-source data to obtain the processed data;

[0023] The preprocessing described in step S2 specifically includes: fusing multi-source data to obtain fused data; and combining the normalized difference and thermal infrared dust index of the dust event image data to obtain a comprehensive dust discrimination index.

[0024] As one embodiment, the present invention fuses collected FY-4 image data, MODIS inversion data, ground observation station data, China Meteorological Administration CLDAS data, and European Centre for Medium-Range Weather Forecasts (ERA5) data according to a defined identification grid.

[0025] In addition, this invention uses a threshold method to calculate the comprehensive dust discrimination index (Dust Distinguish Index) by combining the normalized difference dust index and the thermal infrared dust index of the MODIS data in the corresponding grid points.

[0026] S3. Construct a training set for the sand and dust identification model based on the processed data;

[0027] The process of constructing the training set for the dust recognition model in step S3 is as follows:

[0028] Spectral analysis was performed on the channels corresponding to the image data of the dust event to obtain channels suitable for distinguishing dust.

[0029] In channels suitable for distinguishing dust storms, the corresponding dust storm event image data are manually labeled to differentiate between dust storm areas and non-dust storm areas, thus completing the construction of the training set.

[0030] As one embodiment, the specific processing flow of the present invention is as follows:

[0031] S31. Perform spectral analysis (band reflectance, brightness temperature) on multiple channels of FY-4 satellite, and statistically determine the channels suitable for distinguishing sand and dust from other objects, namely channels 1, 2, 3, 5, 6, 11, 12, and 13, by comparing and analyzing the results.

[0032] S32. Manually mark the dusty and non-dusty areas on the collected FY-4 image data from different channels. Randomly select half of the marked dusty image data and manually add cloud images to the dusty center and edge areas of the images to cover them. These image data are denoted as Cover Data, and the corresponding original data are denoted as Source Data.

[0033] S4. Construct a dust identification model; the dust identification model includes a backbone network, an output network, and a fusion network;

[0034] In step S4, the backbone network is a UNET network; the UNET network outputs the confidence level of the dust category to which each pixel belongs;

[0035] The output network uses an XGboost extreme gradient boosting tree; the XGboost extreme gradient boosting tree outputs the confidence level that the grid points corresponding to the region to be identified are sand and dust regions.

[0036] The fusion network uses a Bayesian method to fuse the output of the UNET network and the output of the XGboost gradient boosting tree to obtain the final result;

[0037] As one embodiment, the construction process of the dust identification model of the present invention is as follows:

[0038] S41. Divide the labeled image data from step S32 into grid point data and use it as input to the U-NET neural network to construct the UNET network. The backbone feature extraction network in the UNET network has five convolutional layers and five pooling layers. The enhancement feature extraction network in the UNET network has an upsampling and stacking structure that matches the convolutional and pooling layers in the backbone feature extraction network. Finally, the output of the UNET network is the confidence score of each pixel for the dust category.

[0039] S42. The data from the China Meteorological Administration's CLDAS, the European Centre for Medium-Range Weather Forecasts (ERA5), ground station observations, DDI data, and MODIS inversion data within the corresponding grid points of the satellite image identification area input in step S41 are used as inputs to XGboost. The final XGboost output is the confidence level that the grid points corresponding to the identified area are indeed dust storm regions.

[0040] S43. A Bayesian method for backend fusion is used to perform feature fusion between the UNET output and the XGboost output.

[0041] S5. Train the dust recognition model using the training set to obtain the final model;

[0042] In step S5, the Adam optimization algorithm is used to optimize the model parameters during the training of the dust identification model.

[0043] As one embodiment, step S5 of the present invention is specifically as follows:

[0044] S51. Split the labeled training data into a training set and a test set in an 8:2 ratio. Use the training set to optimize the parameters of the classification model and use the test set to evaluate the performance of the trained classification model. The evaluation metric is the F1 score, which is commonly used in machine learning for classification tasks.

[0045] S52. Select the Adam optimization algorithm, set the learning rate to 1e-3 and Beta1 to 0.9 to train the recognition model, and finally select the model with the best performance on the test set as the final recognition model.

[0046] S6. Complete the dust identification based on the final model.

[0047] As one embodiment, step S6 is as follows:

[0048] S61. Collect inversion data from the Moderate Resolution Imaging Spectroradiometer (MODIS), data from ground meteorological observation stations, data from the China Meteorological Administration's CLDAS, data from the European Centre for Medium-Range Weather Forecasts (ERA5), and data from the Geostationary Radiation Imager (AGRI) of the Fengyun-4 (FY-4) satellite for the area to be identified.

[0049] S62. After data preprocessing in the second part of the method, the identification area data is processed into a data format that can be input into the identification model.

[0050] S63. Input the data into the well-trained recognition model, and the recognition model infers and outputs the recognition result.

[0051] S64. Visualize the model recognition results on a map to ultimately achieve the goal of automatically identifying sandstorm weather based on a multimodal neural network recognition model.

[0052] Please see Figure 2 , Figure 2 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a sand and dust identification device 401 based on a multimodal neural network, a processor 402, and a storage device 403.

[0053] A dust identification device 401 based on a multimodal neural network: The dust identification device 401 based on a multimodal neural network implements the dust identification method based on a multimodal neural network.

[0054] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the sand and dust identification method based on a multimodal neural network.

[0055] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the sand and dust identification method based on a multimodal neural network.

[0056] The beneficial effects of this invention are as follows: Compared with the shortcomings of traditional methods in terms of single data utilization, it integrates multi-source observation data and uses deep neural networks to effectively extract features from data from different sources; compared with traditional methods, it has significantly improved recognition speed and accuracy, and has also made some improvements in recognizing the continuity of sand and dust; previous methods based on convolutional neural networks in machine learning were mostly image-level classification, while the UNET neural network used in this invention is for pixel-level classification, which has a better effect on processing cloud-covered sand and dust areas.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying sand and dust based on a multimodal neural network, characterized in that: Includes the following steps: S1. Collect multi-source data related to dust storms; the multi-source data includes: imaging spectrometer inversion data, ground meteorological observation station data, and dust storm event image data; S2. Preprocess the multi-source data to obtain the processed data; S3. Construct a training set for the sand and dust identification model based on the processed data; S4. Construct a dust identification model; the dust identification model includes a backbone network, an output network, and a fusion network; In step S4, the backbone network is a UNET network; the UNET network outputs the confidence level of the dust category to which each pixel belongs; The output network uses an XGboost extreme gradient boosting tree; the XGboost extreme gradient boosting tree outputs the confidence level that the grid points corresponding to the region to be identified are sand and dust regions. The fusion network uses a Bayesian method to fuse the output of the UNET network and the output of the Xgboost extreme gradient boosting tree to obtain the final result; S5. Train the dust recognition model using the training set to obtain the final model; S6. Complete the dust identification based on the final model; The preprocessing described in step S2 specifically includes: fusing multi-source data to obtain fused data; and combining the normalized difference and thermal infrared dust index of the dust event image data to obtain a comprehensive dust discrimination index. Data collected from the MODIS medium resolution imaging spectrometer, ground meteorological observation stations, CLDAS data from the China Meteorological Administration, and ERA5 data from the European Centre for Medium-Range Weather Forecasts (ECMWF) for the identified region during the four years from 2018 to 2022; and dust event image data from 14 channels of the AGRI geostationary radiometric imager on the Fengyun-4 FY-4 satellite for the study region during the four years from 2018 to 2022. The collected FY-4 image data, MODIS inversion data, ground observation station data, China Meteorological Administration CLDAS data, and European Centre for Medium-Range Weather Forecasts ERA5 data were fused according to the defined identification grid. The threshold method is used to calculate the comprehensive dust discrimination index by combining the normalized difference dust index and the thermal infrared dust index of the MODIS data in the corresponding grid points. Spectral analysis was performed on multiple channels of FY-4 satellite data. By comparing the analysis results, channels 1, 2, 3, 5, 6, 11, 12, and 13 were identified as suitable for distinguishing sand and dust from other objects. On the FY-4 image data collected from different channels, dusty areas and non-dusty areas were manually marked. Half of the marked dusty image data was randomly selected, and cloud images were manually added to the dusty center area and edge position of the image to cover it. The labeled image data is divided into grid point data as input to the U-NET neural network. The output of the UNET network is the confidence score of each pixel for the dust category. The input data for XGboost includes CLDAS data from the China Meteorological Administration, ERA5 data from the European Centre for Medium-Range Weather Forecasts (ERA5), ground station observation data, DDI data, and MODIS inversion data within the grid points corresponding to the identified satellite image area. The final output of XGboost is the confidence level that the grid points corresponding to the identified area are dusty areas.

2. The dust identification method based on a multimodal neural network as described in claim 1, characterized in that: The process of constructing the training set for the dust recognition model in step S3 is as follows: perform spectral analysis on the channels corresponding to the dust event image data to obtain channels suitable for distinguishing dust; in the channels suitable for distinguishing dust, manually label the corresponding dust event image data to distinguish dust areas and non-dust areas, and complete the construction of the training set.

3. The dust identification method based on a multimodal neural network as described in claim 1, characterized in that: In step S5, the Adam optimization algorithm is used to optimize the model parameters during the training of the dust identification model.

4. A storage device, characterized in that: The storage device stores instructions and data to implement any of the sand and dust identification methods based on multimodal neural networks as described in claims 1 to 3.

5. A dust identification device based on a multimodal neural network, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement any one of the sand and dust identification methods based on a multimodal neural network as described in claims 1 to 3.

Images