Algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion

Abstract

The application discloses an algorithm for filling time-by-time high-precision space-time seamless particulate matter concentration based on machine learning and multi-source data fusion, which comprises the following steps: S1, acquiring satellite time-by-time AOD data, ground station AOD data, CAMS numerical simulation AOD prediction data, meteorological data, emission data, vegetation index, terrain data and social economic data of a research area; S2, pre-processing the multi-source data of the research area acquired above; S3, using the pre-processed multi-source data and space-time characteristics of a grid pixel to make a training data set; S4, inputting the training data set into a two-section space-time random forest model for training, and when the two-section space-time random forest model meets evaluation indexes, the model training is completed, and space-time seamless particulate matter concentration data is output. The algorithm has the advantages that satellite remote sensing AOD and ground observation station data are more fully utilized, space-time seamless AOD is used to predict particulate matter, and the accuracy of particulate matter concentration estimation is improved.

Description

Technical Field

[0001] This invention relates to the field of atmospheric pollutant concentration spatiotemporal distribution calculation technology, and in particular to an algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion. Background Technology

[0002] Particulate matter is a significant component of air pollutants and the primary air pollutant in most parts of my country. Currently, important particulate matter is classified into two categories: fine particulate matter (PM2.5) and inhalable particulate matter (PM10). Particulate matter has a long residence time in the atmosphere and travels long distances, posing a significant threat to human health and severely impacting the atmospheric environment. Particulate matter is one of the monitoring targets of ground-based meteorological monitoring stations nationwide. How to economically, efficiently, and accurately utilize data from ground-based monitoring stations, meteorological data, and satellite remote sensing data to obtain high spatiotemporal resolution near-surface particulate matter concentration data is currently a research hotspot in environmental big data.

[0003] Remote sensing multi-source data fusion technology is an emerging technique that combines data from multiple sources to supplement remote sensing imagery and improve the spatiotemporal coverage of remote sensing data. Accurately predicting the concentration of near-surface particulate matter and other atmospheric pollutants using satellite remote sensing products is challenging. Multi-source fusion technology is one approach to addressing this problem. Based on machine learning, multi-source fusion assimilates, trains, and fuses data from different sources to increase the spatial coverage of the target data. In recent years, this technology has been increasingly applied to the estimation and monitoring of atmospheric pollutant concentrations.

[0004] Machine learning has been applied to reconstruct the spatiotemporal distribution data of air pollutants such as particulate matter. The method uses data from ground monitoring stations as the dependent variable and data from satellite remote sensing, topography, and meteorology as independent variables, employing machine learning models to reconstruct the spatiotemporal distribution of air pollutants. However, due to inherent limitations of satellite remote sensing data (low temporal coverage and short transit time), the reconstructed spatiotemporal distribution data also contains significant gaps.

[0005] Therefore, how to combine the advantages of machine learning and multi-source data fusion to obtain particulate matter spatiotemporal distribution data with high spatial coverage, high temporal coverage, and accurate data estimation is a key challenge in current research. Summary of the Invention

[0006] The purpose of this invention is to provide an algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion, thereby solving the aforementioned problems existing in the prior art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion includes the following steps:

[0009] S1. Acquire hourly satellite AOD data, ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, emission data, vegetation index, topographic data, and socioeconomic data for the study area;

[0010] S2. Preprocess the multi-source data of the study area obtained above to unify the coordinate system and spatial resolution of the multi-source data;

[0011] S3. Use the preprocessed multi-source data and the spatiotemporal features of grid pixels to create a training dataset;

[0012] S4. Input the training dataset into two spatiotemporal random forest models for training. When both spatiotemporal random forest models meet the evaluation criteria, the model training is complete, and the spatiotemporal seamless particulate matter concentration data is output. Step S4 specifically includes the following:

[0013] S41. The first spatiotemporal random forest model uses spatiotemporal seamless AOD data as the dependent variable and ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, and topographic data as independent variables. It obtains initial spatiotemporal seamless AOD data based on satellite hourly AOD data through training; and then weights the initial spatiotemporal seamless AOD data with satellite hourly AOD data to obtain the final spatiotemporal seamless AOD data.

[0014] AOD 卫星 ~f STRT (AOD sta AOD model ,SP,TEM,WU,WV,BLH,RH,DEM,NDVI,P s ,P t )

[0015] Among them, AOD 卫星 The initial spatiotemporal seamless AOD data obtained during training; AOD sta AOD data for ground stations; AOD model CAMS numerical simulation AOD forecast data; SP represents surface air pressure; TEM represents air temperature; WU and WV represent wind volume and wind speed, respectively; BLH represents boundary layer height; RH represents relative humidity; DEM represents topographic data; NDVI represents vegetation index; P s Spatial features; P t It is a time-related feature.

[0016] S42. The second spatiotemporal random forest model uses station particulate matter concentration data as the dependent variable and final spatiotemporal seamless AOD data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, topographic data and other socioeconomic data as independent variables to obtain spatiotemporal seamless particulate matter concentration data through training.

[0017] PM sta ~f STRT (AOD all ,SP,TEM,WU,WV,BLH,RH,DEM,NDVI,…,P s ,P t )

[0018] Among them, PM sta For site particulate matter concentration data, AOD all This is for the final spatiotemporal seamless AOD data.

[0019] Preferably, the hourly AOD data of the satellite includes AOD data from MAIAC and L3-level AOD data from Sunflower 8 and Sunflower 9.

[0020] Preferably, the AOD data for ground stations comes from the national monitoring stations of the China National Environmental Monitoring Centre.

[0021] Preferably, the meteorological data consists of temperature, humidity, wind direction, wind speed, air pressure, solar radiation, rainfall, and evaporation data from the three datasets: ERA5-Land, ERA5-single-levels, and ERA5-pressure-levles.

[0022] Preferably, the emission data is CAMS emission inventory data, which includes SO2, NOx, PM10, and PM2.5.

[0023] Preferably, step S2 specifically involves cleaning the multi-source data to remove missing and invalid values, and unifying the coordinate system and spatial resolution of the multi-source data through reprojection, resampling, and downscaling methods. Ultimately, all the data are unified to a spatial resolution of 1km and a projected coordinate system of WGS84 geographic coordinate system.

[0024] Preferably, L3 AOD data from Sunflower 8 and Sunflower 9 are downscaled using a generative adversarial network; AOD forecast data from CAMS numerical simulations are downscaled using an autocoded deep residual network.

[0025] Preferably, the spatiotemporal features of the grid pixels include spatial features and temporal features.

[0026] Spatial features include the distances between the pixel and the Earth's ellipsoid at nine locations: top left corner, top center point, top right corner, left center point, center point, right center point, bottom left corner, bottom center point, and bottom right corner. These distances are calculated using the inverse cosine formula.

[0027] The calculation of spatiotemporal features includes day and hour. The day feature is calculated by converting the 365 days of a year into a sine function and making the two years before and after continuous. The hour feature is calculated by converting the 24 hours of a day into a sine function and making the two years before and after continuous.

[0028] Preferably, during model training, a ten-fold cross-validation method is used, in which the training dataset is randomly divided into 10 parts, 9 parts are used for training in turn, and 1 part is used for validation. The coefficient of determination and root mean square error are used as model evaluation metrics.

[0029] Preferred options also include,

[0030] S5. Visualize the filled spatiotemporal seamless particulate matter concentration data to show the spatial distribution and changing trends of particulate matter concentration.

[0031] The beneficial effects of this invention are: 1. It obtains hourly, seamless spatiotemporal particulate matter concentration data through multi-source data fusion technology, achieving higher temporal resolution and applicability to near real-time systems. 2. It has a wider range of applications for multi-source data, utilizing not only satellite AOD, ground monitoring station data, meteorological data, and emission data, but also socioeconomic data sources such as land classification data, vegetation index data, nighttime light, road traffic, and human footprints. 3. It makes fuller use of satellite remote sensing AOD and ground observation station data, using seamless spatiotemporal AOD to predict particulate matter concentration, thus improving the accuracy of particulate matter concentration estimation. Attached Figure Description

[0032] Figure 1 This is a flowchart of the algorithm in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the input and output of two segments of the random forest model in the embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] like Figure 1As shown in this embodiment, an algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentrations based on machine learning and multi-source data fusion is provided. This algorithm processes multi-source data, including satellite remote sensing AOD data, meteorological data, topographic data, emission data, ground monitoring station data, population data, and land classification data. It utilizes two segments of machine learning multi-source data fusion technology to obtain hourly-scale seamless historical and near-real-time particulate matter concentration data, reducing errors in particulate matter prediction filling. The algorithm includes the following parts:

[0036] I. Acquisition of Multi-Source Data

[0037] Acquire hourly satellite AOD data, ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, topographic data, and socioeconomic data for the study area. Specifically:

[0038] 1. Hourly AOD Data from Satellites: The data used here are MAIAC data products and L3-level AOD products from Himawari-8 and 9 satellites. MAIAC AOD data is one of the benchmark data used in this invention to create seamless spatiotemporal AOD data, with a spatial resolution of 1 km. The L3-level AOD data from Himawari-8 and 9 satellites has a spatial resolution of 5 km. These two satellite sets were chosen primarily to improve the temporal resolution, from a daily scale to an hourly scale.

[0039] 2. Ground-based AOD data: This data comes from the national monitoring stations of China's National Environmental Monitoring Center (CNEMC).

[0040] 3. CAMS numerical simulation AOD forecast data: This data was obtained by CAMS model forecasting. The time resolution of this data is hourly, and the spatial resolution is approximately 0.4°×0.4°.

[0041] 4. Meteorological Data: The data used is the fifth-generation atmospheric reanalysis dataset (ERA5) from the European Centre for Medium-Range Weather Forecasts (ECMWF). It primarily utilizes various meteorological data, including temperature, humidity, wind direction, wind speed, air pressure, solar radiation, precipitation, and evaporation, from three datasets: ERA5-Land, ERA5-single-levels, and ERA5-pressure-levles. The spatial resolution of these three data products is hourly, with spatial resolutions of 0.1°×0.1° and 0.25°×0.25°, respectively.

[0042] 5. Emissions Data: This data uses CAMS emission inventory data, including air pollutants such as SO2, NOx, PM10, and PM2.5. The temporal resolution is monthly, and the spatial resolution is 0.1° × 0.1°.

[0043] 6. Other data: This mainly includes geographical and socioeconomic data such as topographic data, population data, land classification data, vegetation index data, and transportation data. The temporal resolution of this data is generally annual, and the spatial resolution is typically within 100 meters.

[0044] II. Multi-source data preprocessing

[0045] The multi-source data of the study area obtained above are preprocessed to unify the coordinate system and spatial resolution of the multi-source data.

[0046] Multi-source data, originating from various satellite remote sensing products and assimilation systems, cannot be directly used for training subsequent machine learning models. Therefore, data cleaning and preprocessing are necessary to ensure the accuracy of all data and to maintain a consistent spatial resolution when inputting them into the machine learning model, thus guaranteeing accurate model learning. The data cleaning stage involves removing missing and invalid values ​​from the multi-source data. Since the projection coordinate systems and spatial resolutions of the multi-source data differ, they need to be unified before being input into the machine learning model for training. Therefore, based on the baseline satellite remote sensing AOD data and commonly used geographic coordinate systems in China, corresponding reprojection and resampling preprocessing procedures are performed on the multi-source auxiliary data.

[0047] Satellite AOD data products are downscaled using a downscaling method to increase the spatial resolution to 1km, thereby improving the accuracy of seamless AOD data. CAMS numerical simulation AOD forecast data is downscaled using a deep residual network based on autocoding, while Himawari-8 / 9 satellite AOD data is downscaled using a generative adversarial network. The AOD data obtained through these two downscaling methods is more accurate than AOD data obtained through direct resampling. Finally, all data are unified to a spatial resolution of 1km and a projected coordinate system of WGS84 geographic coordinates. Since the scope of this invention is China, the data is ultimately cropped to cover the entire Chinese region.

[0048] III. Training Dataset Creation

[0049] The preprocessed multi-source data and the spatiotemporal features of grid pixels are combined to form a training dataset.

[0050] The spatiotemporal characteristics of grid pixels include spatial and temporal features.

[0051] Spatial feature P s The distances between pixels and the Earth's ellipsoid at nine locations—top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right—are calculated using the inverse cosine formula.

[0052] P s =(Lon,Lat,d p1 ,d p2 ,d p3 ,d p4 ,d p5 ,d p6 ,d p7 ,d p8 ,d p9 )

[0053] Where d pi Let i be the distance from the pixel to each reference point pi, where i = 1 to 9. Lon,Lat represents the latitude and longitude of the pixel.

[0054]

[0055] Where A1 and A2 are the radians corresponding to the longitudes of the two points, B1 and B2 are the radians corresponding to the latitudes of the two points, and r is the Earth's radius.

[0056] Spatiotemporal characteristics P t The calculation includes day and hour. The day characteristic is achieved by converting the 365 days of a year into a sine function, so that the last day of each year can be connected to the first day of the year. The sine function represents the time characteristic of that day within the year. The hour characteristic is achieved by converting the 24 hours of a day into a sine function, so that the two days can be connected. The sine function represents the time characteristic of that hour within the day.

[0057] IV. Training the Spatiotemporal Random Forest Model

[0058] The training dataset is input into two spatiotemporal random forest models for training. The model training is complete when both spatiotemporal random forest models meet the evaluation criteria, and the output is seamless spatiotemporal particulate matter concentration data. For example... Figure 2 As shown, it specifically includes the following content:

[0059] 1. The first spatiotemporal random forest model uses spatiotemporal seamless AOD data as the dependent variable and ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, and topographic data as independent variables. It obtains initial spatiotemporal seamless AOD data based on satellite hourly AOD data through training; and then weights the initial spatiotemporal seamless AOD data with satellite hourly AOD data to obtain the final spatiotemporal seamless AOD data.

[0060] AOD 卫星 ~f STRT (AOD sta AOD model ,SP,TEM,WU,WV,BLH,RH,DEM,NDVI,P s ,P t )

[0061] Among them, AOD 卫星 The initial spatiotemporal seamless AOD data obtained during training; AOD sta AOD data for ground stations; AOD model CAMS numerical simulation AOD forecast data; SP represents surface air pressure; TEM represents air temperature; WU and WV represent wind volume and wind speed, respectively; BLH represents boundary layer height; RH represents relative humidity; DEM represents topographic data; NDVI represents vegetation index; P s Spatial features; P t It is a time-related feature.

[0062] 2. The second spatiotemporal random forest model uses the site particulate matter concentration data as the dependent variable and the final spatiotemporal seamless AOD data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, and terrain data as independent variables to obtain spatiotemporal seamless particulate matter concentration data through training.

[0063] PM sta ~f STRT (AOD all ,SP,TEM,WU,WV,BLH,RH,DEM,NDVI,…,P s ,P t )

[0064] Among them, PM sta For site particulate matter concentration data, AOD all This is for the final spatiotemporal seamless AOD data.

[0065] V. Model Result Validation

[0066] During model training, a 10-fold cross-validation method was used. The training dataset was randomly divided into 10 parts, with 9 parts used for training and 1 part used for validation in turn. The coefficient of determination (R²) was used. 2 Ri and root mean square error (RMSE) are used to quantitatively characterize the accuracy of the model. 2 A larger RSE indicates higher accuracy of the particulate matter concentration values ​​obtained by the model, while a larger RMSE indicates lower accuracy of the particulate matter concentration data obtained by the model. In this embodiment, RSE... 2 :0.81-0.89, RMSE:10.26-16.19.

[0067] VI. Visualization of Data Results

[0068] Visualizing the spatiotemporal seamless particulate matter concentration data output by the trained model can, to some extent, demonstrate the spatial distribution and changing trends of particulate matter concentration.

[0069] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0070] This invention provides an algorithm for filling hourly, high-precision, spatiotemporally seamless particulate matter concentration data based on machine learning and multi-source data fusion. It obtains hourly spatiotemporally seamless particulate matter concentration data through multi-source data fusion technology, achieving higher temporal resolution and applicability to near real-time systems. The algorithm utilizes a wider range of multi-source data, including not only satellite AOD, ground monitoring station data, meteorological data, and emissions data, but also socioeconomic data sources such as land classification data, vegetation index data, nighttime light, road traffic, and human footprints. It makes fuller use of satellite remote sensing AOD and ground observation station data, using spatiotemporally seamless AOD to predict particulate matter concentration, thus improving the accuracy of particulate matter concentration estimation.

[0071] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion, characterized in that: Includes the following steps, S1. Acquire hourly satellite AOD data, ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, emission data, vegetation index, topographic data, and socioeconomic data for the study area; S2. Preprocess the multi-source data of the study area obtained above to unify the coordinate system and spatial resolution of the multi-source data; S3. Use the preprocessed multi-source data and the spatiotemporal features of grid pixels to create a training dataset; S4. Input the training dataset into two spatiotemporal random forest models for training. When both spatiotemporal random forest models meet the evaluation criteria, the model training is complete, and the spatiotemporal seamless particulate matter concentration data is output. Step S4 specifically includes the following: S41. The first spatiotemporal random forest model uses spatiotemporal seamless AOD data as the dependent variable and ground station AOD data, CAMS numerical simulation AOD forecast data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, and topographic data as independent variables. It obtains initial spatiotemporal seamless AOD data based on satellite hourly AOD data through training; and then weights the initial spatiotemporal seamless AOD data with satellite hourly AOD data to obtain the final spatiotemporal seamless AOD data. ； in, This is the initial spatiotemporal seamless AOD data obtained during training; AOD data for ground stations; This is CAMS numerical simulation data for AOD forecasting; It is the surface air pressure; For temperature; and These are air volume and air speed, respectively. This represents the boundary layer height. Relative humidity; For terrain data; Vegetation index; Spatial features; It is a time-related feature; This is the training function for the spatiotemporal random forest model; S42. The second spatiotemporal random forest model uses station particulate matter concentration data as the dependent variable and final spatiotemporal seamless AOD data, meteorological data, spatiotemporal characteristics of grid pixels, emission data, vegetation index, terrain data, and socioeconomic data as independent variables. The spatiotemporal seamless particulate matter concentration data is obtained through training. The socioeconomic data includes population data, land classification data, nighttime light data, road traffic data, and human footprint data. ； in, For the particulate matter concentration data of the station, The final spatiotemporally seamless AOD data is used for data, while SED represents socioeconomic data.

2. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion as described in claim 1, characterized in that: The hourly AOD data from the satellites includes AOD data from MAIAC as well as L3-level AOD data from Sunflower 8 and Sunflower 9.

3. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 1, characterized in that: The AOD data from the ground stations comes from the national monitoring stations of the China National Environmental Monitoring Centre.

4. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 1, characterized in that: The meteorological data refers to temperature, humidity, wind direction, wind speed, air pressure, solar radiation, precipitation, and evaporation data from three datasets: ERA5-Land, ERA5-single-levels, and ERA5-pressure-levels.

5. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 1, characterized in that: The emission data is CAMS emission inventory data, which includes SO2, NOx, PM10, and PM2.

5.

6. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 2, characterized in that: Step S2 specifically involves cleaning the multi-source data, removing missing and invalid values, and unifying the coordinate system and spatial resolution of the multi-source data through reprojection, resampling, and downscaling methods. Ultimately, all the data are unified to a spatial resolution of 1km and a projected coordinate system of WGS84 geographic coordinate system.

7. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 6, characterized in that: L3 AOD data from Sunflower-8 and Sunflower-9 were downscaled using generative adversarial networks; AOD forecast data from CAMS numerical simulations were downscaled using autocoded deep residual networks.

8. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 1, characterized in that: The spatiotemporal characteristics of grid pixels include spatial and temporal features. Spatial features include the distances between the pixel and the Earth's ellipsoid at nine locations: top left corner, top center point, top right corner, left center point, center point, right center point, bottom left corner, bottom center point, and bottom right corner. These distances are calculated using the inverse cosine formula. The calculation of spatiotemporal features includes day and hour. The day feature is calculated by converting the 365 days of a year into a sine function and making the two years before and after continuous. The hour feature is calculated by converting the 24 hours of a day into a sine function and making the two years before and after continuous.

9. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to claim 1, characterized in that: During model training, a 10-fold cross-validation method was used, in which the training dataset was randomly divided into 10 parts, and 9 parts were used for training in turn, while 1 part was used for validation. The coefficient of determination and root mean square error were used as the model evaluation metrics.

10. The algorithm for filling hourly high-precision spatiotemporal seamless particulate matter concentration based on machine learning and multi-source data fusion according to any one of claims 1 to 9, characterized in that: It also includes, S5. Visualize the filled spatiotemporal seamless particulate matter concentration data to show the spatial distribution and changing trends of particulate matter concentration.

Images