Method and system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data

Abstract

The invention discloses a method and system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, and relates to the field of atmospheric environment, and the method comprises the steps: collecting multi-source data, and carrying out the quality control and preprocessing; the method comprises the following steps: taking multi-source data as an independent variable, taking an orbit layered aerosol extinction coefficient of a cloud-aerosol laser radar and an infrared path finder satellite as a dependent variable, inputting the independent variable and the dependent variable into an extreme gradient boosting model XGBoost, generating a prediction result, combining the prediction result with an original independent variable, forming an enhanced feature set, and inputting the enhanced feature set into a lightweight gradient boosting machine learning model LightGBM, generating a process prediction set; and fusing the process prediction set and the global atmosphere reanalysis data set by adopting a wavelet transform method, and extracting dominant information to obtain an accurate prediction data set of the high-resolution three-dimensional full-coverage aerosol extinction coefficient. According to the method, the reliability, the adaptability and the expansion capability of the inversion method are effectively improved, and the obtained data set can comprehensively, accurately and detailedly represent the aerosol three-dimensional space distribution condition.

Description

Technical Field

[0001] The present invention relates to the field of atmospheric environment technology, and more particularly to a method and system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data. Background Art

[0002] Aerosols not only reflect the atmospheric pollution situation, but also significantly affect the earth's weather and climate system through direct and indirect effects. Accurate characterization of the spatial distribution and dynamic change characteristics of aerosols, especially the vertical distribution characteristics of aerosols, plays a key role in the assessment of atmospheric environmental evolution, radiation balance and climate change response. However, current observation methods and related data products are unable to provide high-resolution, three-dimensional, and full-coverage spatial distribution information of aerosols. From the perspective of observation methods, ground-based radars can only continuously observe the vertical distribution of aerosols at a certain location, and cannot provide a large-scale vertical distribution information of aerosols; aircraft-borne radars can only observe the vertical distribution of aerosols in a certain area at a certain time, and cannot provide long-term and large-scale vertical distribution information of aerosols; traditional satellite-borne sensors can only observe the concentration of aerosols or pollutants in the entire atmosphere or near the ground, and cannot provide vertical stratification information; although satellite-borne lidars can continuously observe a large range of aerosol vertical distribution information, their observation range only covers the orbital strips where the satellite passes, and cannot provide seamless and full-coverage vertical distribution information of aerosols. From the existing data products, whether it is remote sensing inversion products or reanalysis data, most of them can only provide ground or whole-layer atmospheric aerosol concentrations, and cannot provide vertical stratification information; although some products can provide full coverage of aerosol vertical stratification, their horizontal spatial resolution is extremely low (resolution is less than 55 km × 55 km), and their application prospects are limited. Therefore, there is still a lack of data that can comprehensively, accurately and detailedly reflect the vertical and horizontal distribution information of aerosols.

[0003] In recent years, more and more studies have used machine learning methods to obtain high-quality remote sensing data products. Machine learning algorithms have shown significant advantages in processing large-scale, complex data and integrating data from multiple sources. This method is not limited by the time and space of the instrument, nor does it depend on the specific source and collection method of the data. It can enhance the credibility of the results and improve the accuracy of predictions by providing a diverse information perspective using multi-source data. Therefore, many studies have tried to use machine learning methods to invert the spatial distribution data of atmospheric pollutants. At present, how to comprehensively, accurately and in detail characterize the three-dimensional spatial distribution of aerosols is still a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] In view of this, the present invention provides a method and system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, which solves the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data comprises the following steps:

[0007] Collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and construct preprocessed data sets;

[0008] The multi-source data in the preprocessed dataset are used as independent variables, and the orbital stratified aerosol extinction coefficient of the cloud-aerosol lidar and infrared pathfinder satellite is used as the dependent variable, which is input into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient.

[0009] The XGBoost prediction results are combined with the original independent variables to form an enhanced feature set and input into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients;

[0010] The wavelet transform method is used to fuse the process prediction set with the global atmospheric reanalysis dataset, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

[0011] Optional, multi-source data include: cloud-aerosol lidar and infrared pathfinder satellite data, China multi-scale emission inventory model products, global climate and weather reanalysis datasets, China high-resolution and high-quality near-surface air pollutant datasets, digital elevation model data, land use type data and global atmospheric reanalysis datasets.

[0012] Optional quality control and pre-processing operations include the following steps:

[0013] Perform quality checks on multi-source data to remove missing values, outliers, and invalid data;

[0014] The bilinear interpolation method is used to unify the spatial resolution of the data, unifying the resolution of different data sets to the required high-resolution grid to ensure the spatial consistency of the data;

[0015] Time alignment of data with inconsistent time scales to ensure that all data are compared and analyzed within the same time step;

[0016] The time-aligned data are standardized and normalized to construct a preprocessed data set with consistent data dimensions.

[0017] Optional, independent variables include: PM of China Multiscale Emission Inventory Model2.5 , sulfur dioxide, nitrogen oxides and ammonia emissions, 10-meter wind speed, 2-meter air temperature, surface temperature and pressure, and precipitation from the Global Climate and Weather Reanalysis dataset, and PM2.5 from China's high-resolution and high-quality near-surface air pollutant dataset. 2.5 concentration, land use type data, and digital elevation model data.

[0018] Optionally, the extreme gradient boosting model XGBoost adopts a structure that integrates multiple decision trees, and each decision tree is gradually optimized through the gradient boosting algorithm to minimize the loss function.

[0019] Optionally, the lightweight gradient boosting machine learning model LightGBM uses a decision tree-based gradient boosting framework to fit complex data patterns by building multiple decision trees, and uses gradient clipping, sample bucketing, cross-validation, and early stopping strategies for model training.

[0020] Optionally, obtaining an accurate prediction data set includes the following steps:

[0021] The process prediction dataset and the global atmospheric reanalysis dataset are decomposed by wavelet to obtain the sub-band coefficients of different frequency bands;

[0022] Weighted fusion is performed on each frequency band, and the spatial distribution characteristics of the process prediction dataset and the global atmospheric reanalysis dataset are effectively fused using the localized characteristics of wavelet transform.

[0023] Through inverse wavelet transform, a high-resolution three-dimensional full-coverage aerosol extinction coefficient accurate prediction data set is reconstructed.

[0024] A system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, using a method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data as described in any one of the above items, comprising:

[0025] Data acquisition and preprocessing module, used to collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and build preprocessed data sets;

[0026] The prediction module is used to input the multi-source data in the preprocessed data set as independent variables and the orbital stratified aerosol extinction coefficient of the cloud-aerosol lidar and infrared pathfinder satellite as dependent variables into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient;

[0027] The first generation module is used to combine the XGBoost prediction results with the original independent variables to form an enhanced feature set and input it into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients;

[0028] The second generation module is used to fuse the process prediction set with the global atmospheric reanalysis dataset through the wavelet transform method, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

[0029] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a method and system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, which has the following beneficial effects: combining the two machine learning models of XGBoost and LightGBM can effectively improve the reliability, suitability and scalability of the inversion method. This XGBoost-LightGBM-Wavelet (XLW) method can be used to accurately and reliably produce high-resolution three-dimensional full-coverage aerosol extinction coefficient spatial distribution, so as to comprehensively, accurately and in detail characterize the three-dimensional spatial distribution of aerosols, and provide a methodological basis for the subsequent batch production of related data products. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 A flow chart of a method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data provided by the present invention;

[0032] Figure 2 A flowchart for implementing the technical solution provided by the present invention;

[0033] Figure 3 A system structure diagram for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data provided by the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The embodiment of the present invention discloses a method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0036] Collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and construct preprocessed data sets;

[0037] The multi-source data in the preprocessed dataset are used as independent variables and the orbital stratified aerosol extinction coefficient of the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite (CALIPSO) is used as the dependent variable. These are input into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient.

[0038] The XGBoost prediction results are combined with the original independent variables to form an enhanced feature set and input into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients;

[0039] The wavelet transform method is used to fuse the process prediction set with the global atmospheric reanalysis dataset, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

[0040] Furthermore, the multi-source data include: Cloud-Aerosol Lidar and Infrared Pathfinder Satellite (CALIPSO) data, China Multiscale Emission Inventory Model (MEIC) products, Global Climate and Weather Reanalysis Dataset (ERA5), China High-resolution and High-quality Near-surface Air Pollutant Dataset (CHAP), Digital Elevation Model data (DEM), Land Use Type Data (CLCD) and Global Atmospheric Reanalysis Dataset (MERRA2).

[0041] Furthermore, the quality control and pre-processing operations specifically include the following steps:

[0042] Perform quality checks on multi-source data to remove missing values, outliers, and invalid data;

[0043] The bilinear interpolation method is used to unify the spatial resolution of the data, unifying the resolution of different data sets to the required high-resolution grid to ensure the spatial consistency of the data;

[0044] Time alignment of data with inconsistent time scales to ensure that all data are compared and analyzed within the same time step;

[0045] The time-aligned data are standardized and normalized to construct a preprocessed data set with consistent data dimensions for subsequent fusion and analysis.

[0046] Furthermore, the independent variables include: PM 2.5 , sulfur dioxide, nitrogen oxides and ammonia emissions, 10-meter wind speed, 2-meter air temperature, surface temperature and pressure, precipitation from the Global Climate and Weather Reanalysis Dataset (ERA5), PM from the China High-Resolution and High-Quality Near-Surface Air Pollutants Dataset (CHAP) 2.5 Concentration, land use type data (CLCD) and digital elevation model data (DEM) and other multi-source data.

[0047] Furthermore, the extreme gradient boosting model XGBoost adopts a structure that integrates multiple decision trees, and each decision tree is gradually optimized by the gradient boosting algorithm to minimize the loss function. In the specific design of this embodiment, aerosol-related multidimensional input variables (such as temperature, humidity, wind speed, aerosol extinction coefficient, etc.) are combined as features, and the nonlinear fitting ability of the model is used for efficient prediction. Improvements are made on the basis of the prior art, and the model hyperparameters are optimized by feature selection and cross-validation to improve prediction accuracy and generalization ability. Compared with traditional methods, the XGBoost model can better capture the complex nonlinear relationship of aerosol distribution, especially in high-resolution data processing. It shows higher accuracy. Before using the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient, the model is pre-trained with a historical data set to ensure its stability and accuracy, which has obvious advantages over traditional methods in prediction effect and computational efficiency.

[0048] Furthermore, the lightweight gradient boosting machine learning model LightGBM adopts a gradient boosting framework based on decision trees. It fits complex data patterns by building multiple decision trees, and uses strategies such as gradient clipping, sample bucketing, cross-validation, and early stopping for model training. Unlike traditional gradient boosting methods, LightGBM improves training speed and memory efficiency through histogram optimization and data splitting. In terms of model design, LightGBM adopts a leaf-first tree growth method, which can better capture complex feature relationships and improve the prediction effect of high-resolution aerosol spatial distribution. The model uses gradient clipping, sample bucketing and other strategies during training to effectively avoid overfitting and improve prediction accuracy. Compared with other machine learning methods, especially training on large-scale data sets, LightGBM has shown significant computational efficiency advantages. Hyperparameters are tuned during model training, and cross-validation and early stopping strategies are used to ensure the efficiency and generalization ability of the model.

[0049] Furthermore, the acquisition of the accurate prediction data set specifically includes the following steps:

[0050] The process prediction dataset and the global atmospheric reanalysis dataset are decomposed by wavelet to obtain the sub-band coefficients of different frequency bands;

[0051] Weighted fusion is performed on each frequency band, and the spatial distribution characteristics of the process prediction dataset and the global atmospheric reanalysis dataset are effectively fused using the localized characteristics of wavelet transform.

[0052] Through inverse wavelet transform, a high-resolution three-dimensional full-coverage aerosol extinction coefficient accurate prediction data set is reconstructed.

[0053] and Figure 1 Corresponding to the method described above, the embodiment of the present invention also provides a system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, which is used for Figure 1 The specific implementation of the method in the embodiment of the present invention is that a system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data can be applied to computer terminals or various mobile devices, such as Figure 3 As shown, specifically including:

[0054] Data acquisition and preprocessing module, used to collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and build preprocessed data sets;

[0055] The prediction module is used to input the multi-source data in the preprocessed data set as independent variables and the orbital stratified aerosol extinction coefficient of the cloud-aerosol lidar and infrared pathfinder satellite as dependent variables into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient;

[0056] The first generation module is used to combine the XGBoost prediction results with the original independent variables to form an enhanced feature set and input it into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients;

[0057] The second generation module is used to fuse the process prediction set with the global atmospheric reanalysis dataset through the wavelet transform method, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

[0058] In summary, this embodiment combines the two machine learning models of XGBoost and LightGBM to effectively improve the reliability, adaptability and scalability of the inversion method. The prediction results of this XGBoost-LightGBM-Wavelet (XLW) method are accurate and effective, with a correlation of up to 0.94 with ground observation data. It can comprehensively, accurately and in detail characterize the three-dimensional spatial distribution of aerosols, providing a methodological basis for the subsequent batch production of related data products.

[0059] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0060] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, characterized in that: The following steps are involved: Collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and construct preprocessed data sets; The multi-source data in the preprocessed dataset are used as independent variables, and the orbital stratified aerosol extinction coefficient of the cloud-aerosol lidar and infrared pathfinder satellite is used as the dependent variable, which is input into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient. The XGBoost prediction results are combined with the original independent variables to form an enhanced feature set and input into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients; The wavelet transform method is used to fuse the process prediction set with the global atmospheric reanalysis dataset, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

2. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The multi-source data include: cloud-aerosol lidar and infrared pathfinder satellite data, China's multi-scale emission inventory model products, global climate and weather reanalysis datasets, China's high-resolution and high-quality near-surface air pollutant datasets, digital elevation model data, land use type data and global atmospheric reanalysis datasets.

3. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The quality control and pre-processing operations specifically include the following steps: Perform quality checks on multi-source data to remove missing values, outliers, and invalid data; The bilinear interpolation method is used to unify the spatial resolution of the data, unifying the resolution of different data sets to the required high-resolution grid to ensure the spatial consistency of the data; Time alignment of data with inconsistent time scales to ensure that all data are compared and analyzed within the same time step; The time-aligned data are standardized and normalized to construct a preprocessed data set with consistent data dimensions.

4. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The independent variables include: PM2.5 in China's Multi-Scale Emission Inventory Model 2.5 , sulfur dioxide, nitrogen oxides and ammonia emissions, 10-meter wind speed, 2-meter air temperature, surface temperature and pressure, and precipitation from the Global Climate and Weather Reanalysis dataset, and PM2.5 from China's high-resolution and high-quality near-surface air pollutant dataset. 2.5 concentration, land use type data, and digital elevation model data.

5. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The extreme gradient boosting model XGBoost adopts a structure that integrates multiple decision trees. Each decision tree is gradually optimized through the gradient boosting algorithm to minimize the loss function.

6. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The lightweight gradient boosting machine learning model LightGBM adopts a gradient boosting framework based on decision trees. It fits complex data patterns by building multiple decision trees and uses gradient clipping, sample bucketing, cross-validation, and early stopping strategies for model training.

7. The method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data according to claim 1, characterized in that: The acquisition of accurate prediction data set includes the following steps: The process prediction dataset and the global atmospheric reanalysis dataset are decomposed by wavelet to obtain the sub-band coefficients of different frequency bands; Weighted fusion is performed on each frequency band, and the spatial distribution characteristics of the process prediction dataset and the global atmospheric reanalysis dataset are effectively fused using the localized characteristics of wavelet transform. Through inverse wavelet transform, a high-resolution three-dimensional full-coverage aerosol extinction coefficient accurate prediction data set is reconstructed.

8. A system for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data, using a method for inverting high-resolution three-dimensional full-coverage aerosol spatial distribution data as described in any one of claims 1 to 7, characterized in that: include: Data acquisition and preprocessing module, used to collect multi-source data and perform quality control and preprocessing, unify the horizontal and vertical resolutions of different types of data, and build preprocessed data sets; The prediction module is used to input the multi-source data in the preprocessed data set as independent variables and the orbital stratified aerosol extinction coefficient of the cloud-aerosol lidar and infrared pathfinder satellite as dependent variables into the extreme gradient boosting model XGBoost to generate the XGBoost prediction results of the aerosol extinction coefficient; The first generation module is used to combine the XGBoost prediction results with the original independent variables to form an enhanced feature set and input it into the lightweight gradient boosting machine learning model LightGBM to generate a process prediction set of high-resolution three-dimensional full-coverage aerosol extinction coefficients; The second generation module is used to fuse the process prediction set with the global atmospheric reanalysis dataset through the wavelet transform method, extract the advantageous information of the two datasets, and obtain an accurate prediction dataset of high-resolution three-dimensional full-coverage aerosol extinction coefficient.

Images