A PM2.5 chemical component vertical profile inversion system and method driven by multimodal reinforcement learning

Abstract

The present invention discloses a multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system and method, wherein: the system includes a deep reinforcement learning model; the deep reinforcement learning model includes a state space, an action space, a reward function, an actor network, and a critic network; the deep reinforcement learning model is used to calculate the reward value at the current moment based on the difference between the vertical profile prediction value of each PM2.5 chemical component concentration output in the action space and the actual monitored value, combined with the reward function; based on the evaluation results of the strategy evaluation, the strategy is continuously iterated using the gradient descent method until the deep reinforcement learning model converges and the optimal target deep reinforcement learning model is obtained. The system adopts the actor-critic network dynamic optimization strategy and combines it with edge computing to achieve minute-level model updates, solving the problems of high-altitude blind spots and insufficient real-time performance of traditional models.

Description

Technical Field

[0001] The present invention relates to the field of environmental monitoring, and in particular to a PM2.5 chemical component vertical profile inversion system and method driven by multimodal reinforcement learning. Background Art

[0002] With the increasing severity of air pollution, especially the increasingly significant impact of PM2.5 (fine particulate matter) on human health and the environment, accurate monitoring and forecasting of PM2.5 has become a key research topic in the field of environmental monitoring. In recent years, the application of deep learning technology in air quality prediction and inversion has made significant progress. Traditional models, such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), have been widely used to predict the spatiotemporal distribution and infer concentrations of PM2.5. However, these traditional models have several limitations:

[0003] Traditional deep learning models often suffer from data limitations and poor analysis accuracy. The reasons for these data limitations are as follows: traditional inversion models rely on single-source ground-based lidar data, which cannot cover high-altitude blind spots (such as above 6 km), and the temporal and spatial alignment accuracy of satellite data and ground observations is insufficient.

[0004] The poor analysis accuracy stems primarily from two factors: static model flaws and insufficient real-time performance. Existing deep learning models (such as CNN-LSTM) are mostly trained statically, making them difficult to adapt to the dynamic process of pollution diffusion, leading to delayed inversion during sudden pollution events. Furthermore, traditional optimization algorithms (such as NSGA-II) are computationally complex and cannot support minute-level real-time inversion.

[0005] Therefore, how to design an accurate and more adaptable inversion system is a key issue that technicians in this field urgently need to solve. Summary of the Invention

[0006] The purpose of the present invention is to provide a PM2.5 chemical component vertical profile inversion system and method driven by multimodal reinforcement learning, which solves the above-mentioned technical problems pointed out in the prior art.

[0007] The present invention provides a PM2.5 chemical component vertical profile inversion system driven by multimodal reinforcement learning, including a deep reinforcement learning model; the deep reinforcement learning model includes a state space, an action space, a reward function, an actor network, and a critic network;

[0008] The state space is used to perform data conversion on multimodal input data and output a normalized state tensor;

[0009] The Action space is used to transform the multimodal input data into a normalized action tensor to achieve the vertical profile prediction value of each chemical component concentration of PM2.5;

[0010] The Actor network is used to use a multi-layer Transformer encoder to capture the long range across height layers in the vertical grid of the lidar through a self-attention mechanism; the capture function of the long range across height layers of the self-attention mechanism is: ;

[0011] Q: Query matrix, representing the query matrix;

[0012] K: Key matrix, representing the key matrix;

[0013] V: Value matrix, representing the content matrix;

[0014] : is the attention mechanism;

[0015] : Perform dot product on the Query matrix and the Key matrix to calculate the similarity between the Query matrix and the Key matrix;

[0016] : is a scaling factor used to control the order of magnitude of the dot product result;

[0017] : To normalize the dot product result;

[0018] The Critic network is used to input the same state tensor to the GCN-based Critic network to obtain the state value corresponding to the current state; when performing reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of the PM2.5 chemical component concentration and the actual monitoring value, combined with the reward function; the state value output by the Critic network and the actual collected reward are used to calculate the error between the two to obtain the advantage function; when updating the Actor network, the advantage function is used to calculate the gradient of the current Actor network.

[0019] Preferably, as an implementable solution, the deep reinforcement learning model also includes a convergence output module; the convergence output module is used to continuously perform strategy iteration using the gradient descent method until the deep reinforcement learning model converges, obtains the corresponding Actor-Critic network parameters, and outputs the optimal target deep reinforcement learning model.

[0020] Preferably, as an implementable solution, the multimodal terminal includes a ground-based lidar, a remote sensing satellite and a ground monitoring station.

[0021] Preferably, as an implementable solution, the multimodal source data includes lidar optical parameters, satellite remote sensing data, and ground component concentration data.

[0022] The present invention provides a method for inverting PM2.5 chemical component vertical profiles driven by multimodal reinforcement learning, which utilizes the above-mentioned multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system to perform processing, including the following steps:

[0023] The multimodal data input layer obtains the multimodal source data collected by the multimodal terminal, and performs preprocessing and normalization on the multimodal source data; then performs the network parameter processing operation of initializing the Actor-Critic network parameters;

[0024] The data fusion and preprocessing module performs spatiotemporal alignment and denoising. The spatiotemporal alignment process first performs spatial alignment, using Kriging interpolation to match the satellite remote sensing data to the lidar vertical grid. The dynamic time warping (DTW) algorithm is then used to measure the similarity of time series of different lengths and align the time series of ground monitoring stations. The denoising process applies wavelet thresholding to the lidar optical parameters to obtain denoised data.

[0025] The deep reinforcement learning model calculates the reward value at the current moment based on the difference between the vertical profile prediction value of each chemical component concentration of PM2.5 output in the action space and the actual monitored value, combined with the reward function;

[0026] Then, a policy evaluation is performed on the current policy to obtain an evaluation result. Then, based on the evaluation result of the policy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, and the policy iteration is continuously performed using the gradient descent method until the deep reinforcement learning model converges, and the corresponding Actor-Critic network parameters are obtained and the optimal target deep reinforcement learning model is output.

[0027] Preferably, as an implementable solution, the execution of the initialization network parameter processing operation specifically includes: configuring a multi-layer Transformer encoder of the Actor network;

[0028] Configure the structure, initial weights, and hyperparameters of the GCN-based Critic network.

[0029] Preferably, as an implementable solution, the deep reinforcement learning model calculates the reward value at the current moment based on the difference between the vertical profile prediction value of the PM2.5 chemical component concentration output by the action action space and the actual monitored value, combined with the reward function; then the current strategy is evaluated to obtain an evaluation result; then, based on the evaluation result of the strategy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, specifically including:

[0030] The current state tensor of the normalized multimodal data is obtained;

[0031] Transform the multimodal input data and output a normalized action tensor;

[0032] The Actor network performs forward propagation, using a multi-layer Transformer encoder to capture and integrate long-range dependencies across altitude layers through a self-attention mechanism, and outputs predicted values ​​for the vertical profiles of PM2.5 chemical component concentrations.

[0033] And perform forward propagation through the Critic network: input the same state tensor to the GCN-based Critic network to obtain the state value corresponding to the current state;

[0034] When performing reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of each chemical component concentration of PM2.5 and the actual monitored value, combined with the reward function;

[0035] Using the state value output by the Critic network and the actual collected rewards, the error between the two is calculated to obtain the advantage function;

[0036] When updating the Actor network, the advantage function is used to calculate the gradient of the current Actor network.

[0037] Preferably, as an implementable solution, the reward function is: ;

[0038] Among them, α and β are weight coefficients, and Temporal Inconsistency is used to penalize the sudden changes in the prediction results of adjacent time steps.

[0039] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0040] Analysis of the above-mentioned multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system and method provided by the present invention shows that in specific applications, first, data is obtained in a vertical spatial range of 0-6 km through ground-based lidar (optical parameters), remote sensing satellites and ground monitoring stations (PM2.5 component concentrations). This data covers satellite images and near-ground component concentrations, providing multimodal source data input for the model; and preprocessing and normalization processing are performed on the multimodal source data; then, the network parameter processing operation of initializing the Actor-Critic network parameters is performed;

[0041] The data fusion and preprocessing module performs spatiotemporal alignment and denoising. During the spatiotemporal alignment process, spatial alignment is first performed, using Kriging interpolation to match the satellite remote sensing data to the vertical grid of the LiDAR. The Dynamic Time Warping (DTW) algorithm is then used to measure the similarity of time series of different lengths and align the time series of the ground monitoring stations. During the denoising process, wavelet threshold denoising is applied to the LiDAR optical parameters to obtain denoised data. The spatiotemporal alignment of the data using Kriging interpolation and Dynamic Time Warping (DTW) algorithms resolves the problem of data inconsistency in time and space. Wavelet threshold denoising is also applied to the LiDAR optical parameters, effectively reducing noise interference and improving data quality, providing a technical foundation for subsequent data operations.

[0042] The deep reinforcement learning model calculates the current reward based on the difference between the predicted vertical profiles of PM2.5 chemical component concentrations output in the action space and the actual monitored values, combined with a reward function. The current policy is then evaluated to obtain an evaluation result. Based on the evaluation result, the gradient of the actor network is calculated using the policy gradient method. Policy iteration is performed continuously using gradient descent until the deep reinforcement learning model converges. The corresponding actor-critic network parameters are then obtained, and the optimal target deep reinforcement learning model is output. The policy gradient method optimizes the actor network parameters through a "state-action-reward" cycle, enabling the model to dynamically adapt to time-varying characteristics of PM2.5 components, such as diurnal variations and seasonal migration. This deep reinforcement learning model uses an actor-critic network, combined with a reward function and a policy gradient method. Through dynamic optimization through reinforcement learning, the model can adapt to changes in time and space, improving generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the overall architecture of a multimodal reinforcement learning-driven PM2.5 chemical composition vertical profile inversion system;

[0044] Figure 2A schematic diagram of the specific architecture principle of a multimodal reinforcement learning-driven PM2.5 chemical composition vertical profile inversion system;

[0045] Figure 3 Schematic diagram of a deep reinforcement learning model in a multimodal reinforcement learning-driven PM2.5 chemical composition vertical profile inversion system;

[0046] Figure 4 Schematic diagram of the main process of a multimodal reinforcement learning-driven PM2.5 chemical composition vertical profile inversion method;

[0047] Figure 5 Schematic diagram comparing the error results of a multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion method at different altitudes in experimental data.

[0048] Figure 1: Multimodal data input layer 10; Data fusion and preprocessing module 20; Deep reinforcement learning model 30; State space 31; Action space 32; Reward function 33; Actor network 34; Critic network 35. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0051] Example 1

[0052] like Figure 1 as well as Figure 2 As shown, the first embodiment of the present invention provides a PM2.5 chemical component vertical profile inversion system driven by multimodal reinforcement learning, including a multimodal data input layer 10, a data fusion and preprocessing module 20, and a deep reinforcement learning model 30: the multimodal data input layer 10 is used to obtain multimodal source data collected by a multimodal terminal, and perform preprocessing and normalization processing on the multimodal source data; then perform the initialization network parameter processing operation of the Actor-Critic network parameters;

[0053] The data fusion and preprocessing module 20 is used to perform spatiotemporal alignment and denoising. During the spatiotemporal alignment, spatial alignment is first performed by using Kriging to match the satellite remote sensing data to the vertical grid of the lidar. Then, the dynamic time warping (DTW) algorithm is used to measure the similarity of time series of different lengths and align the time series of the ground monitoring stations. During the denoising process, wavelet threshold denoising is applied to the lidar optical parameters to obtain denoised data.

[0054] The deep reinforcement learning model 30 is used to calculate the reward value at the current moment based on the difference between the vertical profile prediction value of the PM2.5 chemical component concentration output by the Action action space and the actual monitoring value, combined with the reward function; then the current strategy is evaluated to obtain an evaluation result; then, based on the evaluation result of the strategy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, and the strategy iteration is continuously performed using the gradient descent method until the deep reinforcement learning model converges, and the corresponding Actor-Critic network parameters are obtained and the optimal target deep reinforcement learning model is output.

[0055] The core architecture of the multimodal reinforcement learning driven PM2.5 chemical composition vertical profile inversion system of the present invention includes the following modules (see Figure 2 as well as Figure 3 ):

[0056] The above system includes a deep reinforcement learning model; the deep reinforcement learning model 30 includes a state space 31, an action space 32, a reward function 33, an actor network 34 and a critic network 35;

[0057] The state space 31 (State Space) is used to perform data conversion on multimodal input data (lidar optical parameters, satellite AOD, ground component concentration) and output a normalized state tensor;

[0058] The Action space 32 is used to transform the multimodal input data and output a normalized action tensor to achieve the vertical profile prediction value of each chemical component concentration of PM2.5;

[0059] The Actor network 34 is used to use a multi-layer Transformer encoder to capture the long-range across height layers in the vertical grid of the lidar through a self-attention mechanism; the capture function of the long-range across height layers of the self-attention mechanism is: ;

[0060] The Critic network 35 is used to input the same state tensor to the GCN-based Critic network to obtain the state value corresponding to the current state; when performing reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of the PM2.5 chemical component concentration and the actual monitoring value, combined with the reward function; the state value output by the Critic network and the actual collected reward are used to calculate the error between the two to obtain the advantage function; when updating the Actor network, the advantage function is used to calculate the gradient of the current Actor network.

[0061] Multimodal Data Input Layer 10: Ground-based LiDAR: Provides vertical profile data of 532 nm aerosol backscattering coefficient, extinction coefficient, and depolarization ratio (temporal resolution: 1 hour, 60 vertical layers). Satellite remote sensing data: Integrates MODIS aerosol optical depth (AOD) and CALIPSO vertical profile data to address high-altitude blind spots. Ground monitoring stations: Obtain real-time monitoring values ​​of near-surface PM2.5 chemical component concentrations (SO₄²⁻, NO₃⁻, NH₄⁺, OM, and BC).

[0062] Data Fusion and Preprocessing Module 20: Spatiotemporal Alignment Algorithm: Kriging is used to match satellite data to the lidar vertical grid, and Dynamic Time Warping (DTW) is used to align ground monitoring station time series. Noise Suppression: Wavelet threshold denoising (Equation 1) is applied to the lidar data to eliminate instrument noise and weather interference.

[0063] ;Formula (1) ;

[0064] in, is the wavelet basis function, is the threshold function.

[0065] The above deep reinforcement learning model 30 (belonging to the framework of the Actor-Critic network):

[0066] State Space 31: Normalized tensor of multimodal input data (lidar optical parameters, satellite AOD, ground component concentration).

[0067] Action Space 32: Predicted vertical profiles of PM2.5 chemical component concentrations.

[0068] Reward Function 33 (Reward Function): ;Formula 2; Among them, α and β are weight coefficients, and Temporal Inconsistency is used to penalize the sudden changes in the prediction results of adjacent time steps.

[0069] Actor Network 34: uses a multi-layer Transformer encoder to capture long-range dependencies across multiple layers through a self-attention mechanism (Formula 3): ;Formula 3;

[0070] Q: Query, the query matrix, represents the input query vector and is responsible for locating the information of interest. K: Key, the key matrix, is used to measure the strength of associations between different locations. V: Value, the content matrix, is used to store the actual semantic or spatial information that needs to be conveyed.

[0071] : The attention mechanism focuses on local information, thereby suppressing useless information and improving the efficiency and accuracy of the model.

[0072] : Perform dot product on the Query matrix and the Key matrix to calculate the similarity between the Query and Key.

[0073] : Scaling factor, used to control the magnitude of the dot product result to prevent gradient vanishing or exploding. This scaling ensures that the gradient of the softmax output is neither too small nor too large, improving stability.

[0074] : Normalize the dot product result, that is, normalize the similarity.

[0075] Critic Network 35: Evaluates state value based on graph convolutional networks (GCN) and dynamically adjusts actor strategies.

[0076] Online learning and optimization module: Real-time data stream processing: Deploy edge computing nodes to support pipeline operations for data reception, preprocessing, and model inference. Progressive update strategy: Adopt the proximal policy optimization (PPO) algorithm (Formula 4) to update model parameters every 10 minutes to avoid policy fluctuations: ;Formula 4;

[0077] in is the advantage function estimated under the old policy, is the importance weight, which indicates the probability ratio of the same action under the new and old strategies; The ratio Restricted to Clipping function within range.

[0078] This system uses satellite data to fill high-altitude blind spots and combines lidar and ground-based monitoring station data to constrain near-surface component concentrations, improving spatial coverage by 40%. A reinforcement learning framework enables real-time model response to pollution events, reducing inversion latency from hours to minutes. Furthermore, by combining edge computing with the PPO algorithm, optimization speed is increased by 50% compared to the traditional NSGA-II.

[0079] Preferably, as an implementable solution, the deep reinforcement learning model also includes a convergence output module; the convergence output module is used to continuously perform strategy iteration using the gradient descent method until the deep reinforcement learning model converges, obtains the corresponding Actor-Critic network parameters, and outputs the optimal target deep reinforcement learning model.

[0080] Preferably, as an implementable solution, the multimodal terminal includes a ground-based lidar, a remote sensing satellite and a ground monitoring station.

[0081] Preferably, as an implementable solution, the multimodal source data includes lidar optical parameters, satellite remote sensing data, and ground component concentration data.

[0082] It should be noted that the lidar optical parameters include vertical profile data of aerosol backscattering coefficient, extinction coefficient, and depolarization ratio; satellite remote sensing data include MODIS AOD data and CALIPSO vertical profile data; MODIS AOD data refers to MODIS aerosol optical depth. Ground component concentration data refers to the concentration of ground-level PM2.5 chemical components. These chemical components primarily include SO₄²⁻, NO₃⁻, NH⁺, OM, and BC.

[0083] Example 2

[0084] See also Figure 4 The second embodiment of the present invention provides a method for inverting vertical profiles of PM2.5 chemical components driven by multimodal reinforcement learning, including the following steps:

[0085] Step S10: The multimodal data input layer obtains the multimodal source data collected by the multimodal terminal, and performs preprocessing and normalization on the multimodal source data; then performs the network parameter processing operation of initializing the Actor-Critic network parameters;

[0086] Multimodal source data must be cleaned and outliers removed separately. For example, for ground-based lidar data, the raw backscatter coefficient (σ_bsc) is subjected to wavelet decomposition (Daubechies-4 basis), with a hard threshold λ set to three times the noise standard deviation. Then, a unified normalization process is applied to all data types, converting data of different dimensions into normalized tensors to ensure that the values ​​are on the same scale when processed by the subsequent networks (Actor and Critic). The network parameter initialization process is performed: the structure, initial weights, and hyperparameters (learning rate, weight decay, gradient threshold, etc.) of the Actor network (multi-layer Transformer encoder) and the Critic network (based on a graph convolutional network, GCN) are configured.

[0087] Step S20: The data fusion and preprocessing module performs spatiotemporal alignment and denoising. During the spatiotemporal alignment, spatial alignment is first performed, using Kriging to match the satellite remote sensing data to the vertical grid of the lidar. Then, the dynamic time warping (DTW) algorithm is used to measure the similarity of time series of different lengths and align the time series of the ground monitoring stations. During the denoising, wavelet threshold denoising is applied to the lidar optical parameters to obtain denoised data.

[0088] In the data fusion and preprocessing module, spatial alignment is first performed using the Kriging interpolation method. Using regionalized variables as the basis and the variogram as the basic tool, linear, unbiased, and optimal estimation is performed on unknown sample points. The vertical variogram of the satellite data is calculated to determine the spatial correlation structure, and the variogram parameters are fitted using the maximum likelihood estimation method. Vertical layers are divided according to the radar's detection range to generate grid coordinates (0-6 km, 60 layers) with a spatial resolution of 1 km × 1 km. The satellite data are then matched to the lidar vertical grid. Dynamic time warping (DTW) is used to measure the similarity of time series of different lengths and align the time series of ground monitoring stations, with the time error controlled within 5 minutes. Wavelet threshold denoising is then applied to the lidar data. The denoising method is described in Equation (1), which eliminates instrument noise and weather interference.

[0089] Step S30: The deep reinforcement learning model calculates the reward value at the current moment based on the difference between the vertical profile prediction value of each chemical component concentration of PM2.5 output in the action space and the actual monitored value, combined with the reward function;

[0090] Then, a policy evaluation is performed on the current policy to obtain an evaluation result. Then, based on the evaluation result of the policy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, and the policy iteration is continuously performed using the gradient descent method until the deep reinforcement learning model converges, and the corresponding Actor-Critic network parameters are obtained and the optimal target deep reinforcement learning model is output.

[0091] The normalized, multimodal data obtained in steps S1 and S2 is combined to form the current state tensor (State), which represents comprehensive information about the current environment at each moment. The forward propagation of the actor network utilizes a multi-layer Transformer encoder, which uses a self-attention mechanism to capture and integrate long-range dependencies across altitude layers in the lidar vertical grid. It outputs predictions (i.e., actions, or action spaces) corresponding to the vertical profiles of PM2.5 chemical component concentrations. The inter-layer information exchange of the Transformer ensures that the dependencies between vertical structures are fully modeled (see Equation 3).

[0092] And perform the forward propagation of the Critic network: input the same state tensor to the GCN-based Critic network to obtain the value estimate corresponding to the current state, which is used for subsequent TD (Temporal Difference) error calculation and strategy evaluation.

[0093] When performing reward evaluation, the reward at the current moment is calculated based on the difference between the action output and the actual monitoring value, combined with the reward function (Formula 2).

[0094] Weight coefficients such as α and β are introduced into the reward function. At the same time, the Temporal Inconsistency term is used to penalize drastic changes in the prediction results of adjacent time steps to ensure the smoothness of the prediction.

[0095] Using the state values ​​output by the Critic network and the actual rewards collected, we calculate the TD error (or advantage function) to measure the quality of the current policy. Based on the policy gradient method, we use the advantage as a weighted term to calculate the gradient of the Actor network.

[0096] The goal is to maximize the expected reward, so the loss between the predicted action and the actual high-altitude vertical profile is back-propagated while taking into account the Temporal Inconsistency penalty.

[0097] When updating the Actor network, an advantage function is needed to calculate the policy gradient.

[0098] To prevent updates to the Critic network from affecting updates to the Actor network, the stop-gradient operator can be used to prevent gradient information from flowing to the Critic network. Before officially using the model, offline pre-training can be performed, using historical data to pre-train the network. To constrain the policy network from deviating from the behavioral policy in the offline data, a regularization term (such as KL divergence) can be introduced during policy updates to limit policy changes. Online fine-tuning is then performed, with real-time data streams input to edge nodes. The PPO algorithm updates the Actor-Critic network parameters every 10 minutes. A drop in the reward value exceeding 10% for three consecutive iterations triggers a rollback to the previous stable version. After the deep reinforcement learning model converges, the target deep reinforcement learning model should be output.

[0099] The actor-critic algorithm combines policy optimization (actor) and value assessment (critic). Specifically: The actor network (policy network) generates the policy. Given a state, it outputs a probability distribution over actions. It updates the policy using policy gradients to maximize the expected reward. The critic network (value network) evaluates the value of the current policy. Given a state, it outputs a state value V, which is used to estimate the temporal difference (TD) error, thereby guiding policy updates.

[0100] Preferably, as an implementable solution, the execution of the initialization network parameter processing operation specifically includes: configuring a multi-layer Transformer encoder of the Actor network;

[0101] Configure the structure, initial weights, and hyperparameters of the GCN-based Critic network.

[0102] Preferably, as an implementable solution, the deep reinforcement learning model calculates the reward value at the current moment based on the difference between the vertical profile prediction value of the PM2.5 chemical component concentration output by the action action space and the actual monitored value, combined with the reward function; then the current strategy is evaluated to obtain an evaluation result; then, based on the evaluation result of the strategy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, specifically including:

[0103] Step S31: The current state tensor of the normalized multimodal data obtained in step S1 / step S2;

[0104] Step S32: convert the multimodal input data to output a normalized action tensor;

[0105] Step S33: The Actor network performs forward propagation, using a multi-layer Transformer encoder to capture and integrate long-range dependencies across altitude layers through a self-attention mechanism, and outputs the predicted values ​​of the vertical profiles of the PM2.5 chemical component concentrations.

[0106] Step S34: and perform forward propagation through the Critic network: input the same state tensor to the GCN-based Critic network to obtain the state value corresponding to the current state, which is used for subsequent TD (Temporal Difference) error calculation and strategy evaluation.

[0107] Step S35: When performing the reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of each chemical component concentration of PM2.5 and the actual monitored value, combined with the reward function;

[0108] Weight coefficients such as α and β are introduced into the reward function, and the Temporal Inconsistency term is used to penalize drastic changes in the prediction results of adjacent time steps to ensure prediction smoothness.

[0109] Using the state value output by the Critic network and the actual collected rewards, the error between the two is calculated to obtain the advantage function;

[0110] Step S36: When updating the Actor network, use the advantage function to calculate the gradient of the current Actor network.

[0111] Preferably, as an implementable solution, the reward function 30 (Reward Function) is: ;

[0112] Among them, α and β are weight coefficients, and Temporal Inconsistency is to punish the sudden change of the prediction results of adjacent time steps. RMSE is the root mean square error, where represents the predicted value (i.e. the vertical profile predicted value), where represents the observed value (i.e., the actual monitored value);

[0113] It should be noted that the researchers conducted the following experimental tests based on the above-mentioned embodiments of the present application:

[0114] 1-Execute data preprocessing process:

[0115] LiDAR data denoising:

[0116] The raw backscattering coefficient (σ_bsc) was subjected to wavelet decomposition (Daubechies-4 basis), and the hard threshold λ was set to 3 times the standard deviation of the noise.

[0117] Satellite data interpolation:

[0118] Kriging interpolation was used to map the MODIS AOD data to the lidar vertical grid (0–6 km, 60 layers) with a spatial resolution of 1 km × 1 km.

[0119] Time Alignment:

[0120] The ground monitoring station data and lidar data were aligned to a unified timestamp (time error < 5 minutes) through dynamic time warping (DTW).

[0121] 2- First perform the offline model training steps:

[0122] Offline pre-training:

[0123] Input: One year of historical data (with a time resolution of 1 hour), including pollution events (such as sandstorms) and non-pollution periods.

[0124] Objective: Minimize RMSE and KL divergence (Equation 5) to ensure that the predicted distribution conforms to physical laws: (5)

[0125] Online fine-tuning:

[0126] Real-time data streams are input to edge nodes, and the PPO algorithm updates the Actor-Critic network parameters every 10 minutes.

[0127] If the reward value drops by more than 10% for three consecutive iterations, the model will be rolled back to the previous stable version.

[0128] 3- Output test results

[0129] 1. Experimental Design

[0130] Selected datasets: 2022 sandstorm event in the Beijing-Tianjin-Hebei region (March 15-20) and routine monitoring data.

[0131] Comparison baselines: the original patented CNN-ATT-BiLSTM-NSGA model and the traditional chemical transport model (WRF-Chem).

[0132] 2. Performance indicators

[0133] Accuracy: CORR (correlation coefficient), RMSE (μg / m³).

[0134] Real-time: Inversion delay (time from data input to result output).

[0135] Result Analysis

[0136] Ground layer accuracy (see Table 1):

[0137] Upper air layer coverage ( Figure 5 ): The fusion of satellite data above 6 km reduces the OM concentration inversion error by 30% (from 4.2 μg / m³ to 2.9 μg / m³). Figure 5 A comparison of error results at different altitude levels is also shown; Figure 5 Also shown: = Comparison of high-altitude OM concentration inversion results before and after satellite data fusion (aerial survey verification).

[0138] Real-time response (see Table 2): During a sandstorm event, the model updated parameters within 10 minutes, with an inversion delay of <2 minutes and a 40% reduction in false alarm rate compared to the baseline.

[0139] Table 2: Real-time response comparison

[0140]

[0141] In summary, the multimodal reinforcement learning-driven PM2.5 chemical composition vertical profile inversion system and method proposed in this paper is based on multimodal data fusion and deep reinforcement learning. This system and method provides real-time inversion of PM2.5 chemical composition vertical profiles (including sulfate, nitrate, ammonium salts, organic matter, and black carbon) based on multimodal data fusion and deep reinforcement learning. By integrating ground-based lidar, satellite remote sensing, and ground-based monitoring station data with a reinforcement learning framework, this system achieves high-resolution and high-precision dynamic monitoring of the vertical distribution of PM2.5 chemical compositions, making it suitable for emergency response to pollution incidents and long-term air quality assessment.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system, characterized by: Including multimodal data input layer, data fusion and preprocessing module, deep reinforcement learning model: The multimodal data input layer is used to obtain the multimodal source data collected by the multimodal terminal, and perform preprocessing and normalization processing on the multimodal source data; then perform the initialization network parameter processing operation of the Actor-Critic network parameters; The data fusion and preprocessing module is used to perform spatiotemporal alignment and denoising. During the spatiotemporal alignment, spatial alignment is first performed, using Kriging interpolation to match the satellite remote sensing data to the vertical grid of the lidar. Then, a dynamic time warping algorithm is used to measure the similarity of time series of different lengths and align the time series of ground monitoring stations. During the denoising process, wavelet threshold denoising is applied to the lidar optical parameters to obtain denoised data. The deep reinforcement learning model is used to calculate the reward value at the current moment based on the difference between the vertical profile prediction value of each chemical component concentration of PM2.5 output in the action space and the actual monitored value, combined with the reward function; Then, the current strategy is evaluated to obtain the evaluation result; Then, according to the evaluation results of the policy evaluation, the gradient of the Actor network is calculated based on the policy gradient method, including: obtaining the current state tensor of the normalized multimodal data; transforming the multimodal input data to output the normalized action tensor; the Actor network performs forward propagation, using a multi-layer Transformer encoder to capture and integrate long-range dependencies across height layers through the self-attention mechanism, and outputs the predicted value of the vertical profile of the concentration of each chemical component of PM2.5; and performs forward propagation through the Critic network: input the same state tensor to the GCN-based Critic network to obtain the state value corresponding to the current state ; When performing reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of each chemical component concentration of PM2.5 and the actual monitored value, combined with the reward function; the state value output by the Critic network and the actual collected reward are used to calculate the error between the two to obtain the advantage function; when updating the Actor network, the advantage function is used to calculate the gradient of the current Actor network; the gradient descent method is used to continuously iterate the strategy until the deep reinforcement learning model converges, and the corresponding Actor-Critic network parameters are obtained and the optimal target deep reinforcement learning model is output; The reward function is: ； Among them, α and β are weight coefficients, Temporal_Inconsistency is to punish the sudden change of the prediction results of adjacent time steps. is the root mean square error between the predicted value and the actual monitored value of the vertical profile.

2. A multimodal reinforcement learning driven PM2.5 chemical component vertical profile inversion system according to claim 1, characterized in that: Including a deep reinforcement learning model; the deep reinforcement learning model includes a state space, an action space, a reward function, an actor network and a critic network; The state space is used to perform data conversion on multimodal input data and output a normalized state tensor; The Action space is used to transform the multimodal input data into a normalized action tensor to achieve the vertical profile prediction value of each chemical component concentration of PM2.5; The Actor network is used to use a multi-layer Transformer encoder to capture the long range across height layers in the vertical grid of the lidar through a self-attention mechanism; the capture function of the long range across height layers of the self-attention mechanism is: ； Q: Query matrix, representing the query matrix; K: Key matrix, representing the key matrix; V: Value matrix, representing the content matrix; : It is the self-attention mechanism; : Perform dot product on the Query matrix and the Key matrix to calculate the similarity between the Query matrix and the Key matrix; : is a scaling factor used to control the order of magnitude of the dot product result; : To normalize the dot product result; The critic network is used to input the same state tensor to the GCN-based critic network to obtain the state value corresponding to the current state; when performing reward evaluation, the reward value at the current moment is calculated based on the difference between the output vertical profile prediction value of each chemical component concentration of PM2.5 and the actual monitored value, combined with the reward function; The advantage function is obtained by calculating the error between the state value output by the Critic network and the actual reward collected. When updating the Actor network, the advantage function is used to calculate the gradient of the current Actor network.

3. The multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system according to claim 2, characterized in that: The deep reinforcement learning model also includes a convergence output module; the convergence output module is used to continuously perform strategy iteration using the gradient descent method until the deep reinforcement learning model converges, obtain the corresponding Actor-Critic network parameters, and output the optimal target deep reinforcement learning model.

4. The multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system according to claim 1, characterized in that: The multimodal terminal includes a ground-based laser radar, a remote sensing satellite and a ground monitoring station.

5. The multimodal reinforcement learning-driven PM2.5 chemical component vertical profile inversion system according to claim 3, characterized in that: The multimodal source data includes lidar optical parameters, satellite remote sensing data, and ground component concentration data.

Images