An air pollution concentration prediction method based on multi-modal feature fusion

Abstract

This invention discloses an air pollution concentration prediction method based on multimodal feature fusion. The method includes acquiring multimodal data from various monitoring stations within a target area and performing temporal alignment and missing value processing. A multimodal air pollution concentration prediction model is constructed, where the image branch extracts spatial features based on a convolutional neural network combined with a self-attention mechanism, and the numerical branch extracts spatiotemporal features through a temporal convolutional network, a long short-term memory network, and a graph attention network. Physical prior weights that fuse spatial distance and wind direction information are introduced into the graph structure. During training, a contrastive learning constraint is introduced to achieve semantic alignment of different modal features. The fused features are input into a regression module, which outputs the pollutant concentration prediction results for each monitoring station. This invention effectively improves the collaborative expression capability of multimodal features and the accuracy of spatial modeling, and still exhibits good predictive performance and generalization ability under conditions of missing data or no historical observations.

Description

Technical Field

[0001] This invention belongs to the field of environmental monitoring technology, specifically relating to a method for predicting air pollution concentration based on multimodal feature fusion. Background Technology

[0002] Predicting air pollutant concentrations is a key technology in environmental science and pollution control. With the accelerating pace of urbanization, the impacts of fine particulate matter (PM2.5), inhalable particulate matter (PM10), and various gaseous pollutants on human health and the ecological environment are becoming increasingly prominent. Achieving accurate and efficient air quality prediction has become a core concern for environmental management departments and research institutions. The formation and evolution of air pollutant concentrations exhibit significant spatiotemporal correlations and nonlinear characteristics. Accurate prediction not only relies on historical pollution observation data but also requires comprehensive consideration of multi-source heterogeneous information such as meteorological conditions, geographical distribution, and anthropogenic emissions. Therefore, it is a typical high-complexity spatiotemporal modeling problem.

[0003] Traditional methods for predicting air pollutant concentrations mainly include numerical prediction models based on physical mechanisms, regression models based on statistical analysis, and machine learning methods based on shallow structures. These methods have advantages such as clear mechanisms and strong interpretability, but they generally suffer from insufficient expressive power and limited spatiotemporal dependency modeling capabilities when dealing with complex environmental data that is high-dimensional, multi-source, and highly nonlinear, making it difficult to meet the requirements for high-precision prediction.

[0004] In recent years, with the development of deep learning technology, air pollution prediction methods based on multimodal fusion have gradually become a research hotspot. For example, in existing technologies, CN120783905A proposes a multimodal air pollutant prediction method that integrates monitoring station data, emission source data, and meteorological field data, and inputs the fused features into a deep learning model for prediction, thus improving prediction performance to some extent. However, this type of method still has the following shortcomings in practical applications: (1) Existing technologies mainly integrate multi-source data through feature splicing or implicit learning, without explicitly introducing the physical mechanism of atmospheric pollutant diffusion into the model structure, such as the influence of factors like wind direction and spatial distance on pollutant transport. This results in the model's limited ability to characterize the spatial propagation pattern of pollutants, thus affecting the prediction accuracy. (2) Existing methods do not make full use of the dynamic spatial relationship between monitoring stations and lack an adaptive feature aggregation mechanism based on neighborhood relationship, making it difficult to accurately capture the local diffusion and transmission characteristics of pollutants in space.

[0005] Therefore, how to introduce a spatial modeling mechanism that conforms to physical laws in the process of multimodal air pollutant prediction, and how to construct a feature modeling method that can effectively characterize the spatial dependence between monitoring stations, so as to improve the accuracy and reliability of air pollutant concentration prediction, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an air pollution concentration prediction method based on multimodal feature fusion.

[0007] The objective of this invention can be achieved through the following technical solutions: This invention provides a method for predicting air pollution concentration based on multimodal feature fusion, comprising the following steps: Acquire multimodal input data of the target area to be predicted, and perform time alignment and missing value processing on the multimodal input data; the multimodal input data includes image data, pollutant time series data, meteorological time series data, and geographical location information of each monitoring station in the target area; A multimodal air pollution concentration prediction model is constructed, comprising an image feature extraction branch, a numerical spatiotemporal feature extraction branch, and a regression module, wherein: The image feature extraction branch is used to extract spatial feature representations from the image data; The numerical spatiotemporal feature extraction branch is used to encode the pollutant time series data, meteorological time series data, and geographical location information to obtain the spatiotemporal feature representation of each monitoring station; The multimodal air pollution concentration prediction model is trained end-to-end by constructing a joint loss function to obtain the trained multimodal air pollution concentration prediction model. The multimodal input data of the target area to be predicted is input into the trained multimodal air pollution concentration prediction model. The output image feature representation is fused with the spatiotemporal feature representation corresponding to the monitoring station to obtain the fused feature representation. The fused feature representation is then input into the regression module to output the predicted air pollutant concentration values ​​of each monitoring station in the target area at the target time.

[0008] Furthermore, the monitoring stations are air quality monitoring nodes distributed within or around the target area, used to collect pollutant observation data at the corresponding locations; The image data refers to remote sensing image data reflecting the atmospheric environment status of the monitoring nodes, including satellite remote sensing images; The pollutant time series data refers to the pollutant observation data collected by the monitoring station within a continuous time range. The pollutant observation data includes fine particulate matter concentration data, inhalable particulate matter concentration data, and gaseous pollutant concentration data. The meteorological time series data are the meteorological data corresponding to the monitoring node, including temperature, humidity, wind speed, wind direction, and air pressure; The geographic location information refers to data representing the spatial location of each monitoring station, including latitude and longitude coordinates.

[0009] Furthermore, the image feature extraction branch is a ResNet-18-based convolutional neural network, and a self-calibrating self-attention module (SCSA) is integrated after the intermediate and terminal layers of ResNet; the SCSA is used to adaptively recalibrate the convolutional features in terms of spatial and channel dimensions.

[0010] Furthermore, the numerical spatiotemporal feature extraction branch includes a temporal convolutional network (TCN), a long short-term memory network (LSTM), and a graph attention network (GAT).

[0011] Furthermore, the processing procedure of the numerical spatiotemporal feature extraction branch specifically includes: The pollutant time-series data from each monitoring station were coded, for the first... The pollutant time series data from each monitoring station are used to extract time-dependent features through a temporal convolutional network (TCN). The TCN employs a convolutional structure that includes causal convolution, dilated convolution, and residual connections. The meteorological time series data is encoded, and the meteorological time series is input into a Long Short-Term Memory (LSTM) network to obtain meteorological feature representations; The geographic location information is encoded, and the latitude and longitude coordinates of each monitoring station are input into a multilayer perceptron (MLP) for mapping to obtain a location feature representation. Each monitoring station is treated as a node, and a spatial graph structure is constructed based on the spatial distance relationship between the monitoring stations. When the distance between nodes is less than a preset threshold, a connection is established between the corresponding nodes to determine the node. The set of neighboring nodes ; The pollutant time characteristics, meteorological characteristics and location characteristics corresponding to each monitoring station are spliced ​​together as the initial feature representation of the corresponding node. For the target monitoring station, zero-filling is used to complete the data when pollutant time series data is missing. The initial feature representations of the nodes are input into the Graph Attention Network (GAT) for spatial feature aggregation. The calculation of the attention coefficients between nodes is expressed as follows: in, and Representing nodes respectively With nodes Feature representation, It is a linear transformation matrix. For attention weight parameters, For nodes The set of neighboring nodes; Represents a node With nodes The unnormalized attention coefficients between them; Represents a node With nodes The normalized attention coefficient; This is the adjustment coefficient for the physical prior weights; For nodes With nodes Physical prior weights; Based on the attention coefficient, the features of neighboring nodes are weighted and aggregated to obtain the node update features, which are expressed as follows: in, For nodes Updated node features For activation functions; The As a node Spatiotemporal characteristics of the corresponding monitoring stations.

[0012] Furthermore, the physical prior weights are defined by the following formula: in, Represents a node With nodes Spatial distance between them; Represents a node j Relative to node i The angle between the direction of the wind and the wind direction; Preset weighting coefficients; To prevent constants with a denominator of zero.

[0013] Furthermore, the regression module is a multilayer perceptron structure, used to map and regress the fused image feature representation and numerical spatiotemporal feature representation, and output the predicted air pollutant concentration values ​​of each monitoring station at the target time.

[0014] Furthermore, the multimodal air pollution concentration prediction model is trained end-to-end by constructing a joint loss function to obtain the trained multimodal air pollution concentration prediction model, specifically including: Construct a training dataset by organizing historical multimodal input data in chronological order, based on each target time point. t The image data corresponding to that moment, as well as the pollutant time series data and meteorological time series data within a preset time window prior to that moment, are extracted and combined with the geographical location information of each monitoring station to construct a sample data pair. ,in, Indicates the first i Multimodal input data of one sample Indicates the first i The air pollutant concentration observation value of each sample at the next moment, wherein the air pollutant concentration observation value is a vector composed of the fine particulate matter concentration, inhalable particulate matter concentration and main gaseous pollutant concentration observed by the monitoring station at the target time. The training dataset is divided into a training set, a validation set, and a test set for model training, parameter tuning, and performance evaluation. The multimodal input data is preprocessed, including time alignment, missing value imputation, and time resolution unification. Missing values ​​in the time series are imputed using interpolation methods, expressed as follows: in, and Represents the observation values ​​at adjacent known times; Indicates time The interpolation result; The preprocessed multimodal data is input into the multimodal air pollution concentration prediction model. Image feature representation and spatiotemporal feature representation are extracted through the image feature extraction branch and the numerical spatiotemporal feature extraction branch, respectively. Feature fusion is then performed to obtain the predicted value of air pollutant concentration. Based on the predicted and observed values ​​of air pollutant concentrations, a joint loss function is calculated. Based on the joint loss function, the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The model parameters are then updated using an optimization algorithm until the joint loss function converges, thus obtaining the trained multimodal air pollution concentration prediction model.

[0015] Furthermore, the joint loss function includes prediction loss and contrast loss, expressed as: in, For the joint loss function; To predict losses; To compare the losses; The prediction loss is calculated using the following formula: in, Indicates the training batch size; These are predicted values ​​for air pollutant concentrations.

[0016] Furthermore, the contrast loss is formulated as follows: in, Indicates the first i The image features of the sample and the first j Cosine similarity of the numerical spatiotemporal features of each sample in the projection space; Indicates the first i The image features of the sample and the first The cosine similarity of the numerical spatiotemporal features of each sample in the projection space; the projection space is obtained by linear or nonlinear mapping of multimodal features; This is a positive sample mask, with a value of 1 indicating a positive sample. With sample If it is a positive sample, then it is 0; This is a negative sample mask; a value of 1 indicates a negative sample. With sample i A positive sample is a negative sample if it is negative, otherwise it is zero. A positive sample is one in which the image features and the numerical spatiotemporal features originate from the same monitoring station within the target time and its preceding time window. A negative sample is one in which the image features and the numerical spatiotemporal features originate from different monitoring stations within the target time and its preceding time window. A stable term to prevent numerical underflow.

[0017] Compared with the prior art, the present invention has the following advantages: (1) In view of the problem that the existing technology does not explicitly introduce the physical mechanism of atmospheric pollutant diffusion into the model, which makes it difficult to accurately depict the transmission process of pollutants in space and the physical consistency of the prediction results is poor, the present invention introduces physical prior weights based on spatial distance, wind direction relationship and distance attenuation mechanism into the graph attention network, and uses the physical prior weights as attention bias terms to participate in the calculation of attention coefficients between nodes, thereby embedding the physical law of pollutant diffusion into the model structure, realizing the constraint and guidance of the spatial propagation process of pollutants, effectively improving the model's ability to model complex atmospheric diffusion processes, and thus significantly improving the accuracy and physical rationality of air pollutant concentration prediction.

[0018] (2) In view of the problem that the existing technology does not make full use of the spatial correlation between monitoring stations and is difficult to characterize the local diffusion and regional transmission characteristics of pollutants, the present invention constructs a graph structure based on the spatial distance of monitoring stations and combines a graph attention network to adaptively weight and aggregate the features of neighboring nodes, so that the model can dynamically learn the influence intensity between different stations, realize the fine modeling of spatial dependence, thereby improving the ability to express the diffusion path and local aggregation phenomenon of pollutants in the spatial range, enhancing the model's ability to characterize complex spatial distribution features, and improving the stability and reliability of prediction results.

[0019] (3) In view of the problems that multimodal data fusion process is prone to inconsistency of feature semantics and insufficient utilization of intermodal information due to large distribution differences and direct splicing, this invention introduces a contrastive learning mechanism to construct cross-modal contrastive loss in the joint loss function, constrains the representation of image features and numerical spatiotemporal features in the projection space, makes cross-modal features from the same monitoring station closer in the representation space, while keeping the features of different stations distinguishable, thereby realizing semantic alignment and collaborative optimization of multimodal features, improving the model's ability to fuse multi-source information and feature expression consistency, and thus improving the model's generalization performance and prediction accuracy.

[0020] (4) In view of the problem that traditional methods rely on complete historical observation data of target monitoring stations and are difficult to make effective predictions when new stations are built or data is missing, this invention preprocesses the missing time series data and combines graph attention network to spatially aggregate the features of neighboring stations. When the historical data of the target station is insufficient or missing, the spatiotemporal information of the surrounding stations is used to generate the feature representation of the target station, thereby realizing the prediction of air pollutant concentration in scenarios with no historical observation or missing data, and significantly improving the applicability and robustness of the model in complex practical application environments.

[0021] (5) To address the problem of insufficient utilization of spatial information in image data in air pollution prediction, this invention constructs an image feature extraction branch based on a convolutional neural network and introduces a self-calibrating self-attention module into the network to adaptively recalibrate the spatial and channel dimensions of image features, thereby enhancing the model's ability to perceive key areas and important features in remote sensing images, improving the information contribution of image modalities in multimodal fusion, and further enhancing the overall prediction performance. Attached Figure Description

[0022] Figure 1 This is a flowchart of the air pollution concentration prediction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal air pollution concentration prediction model according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] Example 1: This embodiment provides a method for predicting air pollution concentration based on multimodal feature fusion, such as... Figure 1 As shown, it includes the following steps: Step S1: Obtain multimodal input data of the target region to be predicted, and perform time alignment and missing value processing on the multimodal input data; In this embodiment, the multimodal input data includes image data, pollutant time-series data, meteorological time-series data, and geographical location information of each monitoring station within the target area. Monitoring stations are air quality monitoring nodes distributed within or around the target area, used to continuously collect pollutant concentration information at their corresponding spatial locations. Image data utilizes remote sensing images that characterize the atmospheric environment, such as satellite remote sensing images, which can reflect information such as aerosol distribution, cloud cover, and changes in the surface environment at a macroscopic scale, thus providing spatial background information for pollutant diffusion. Pollutant time-series data consists of pollutant observations collected by each monitoring station over a continuous time period, specifically including concentration data of fine particulate matter (PM2.5), inhalable particulate matter (PM10), and gaseous pollutants (such as NO2, SO2, CO, and O3). This type of data can depict the dynamic changes of pollutants over time. Meteorological time-series data corresponds one-to-one with the monitoring stations and includes elements such as temperature, humidity, wind speed, wind direction, and air pressure, used to reflect external environmental conditions affecting the generation, diffusion, and transport of pollutants. Geographic location information is used to describe the spatial relationship between each monitoring station, and is usually represented by latitude and longitude coordinates, providing a basis for subsequent spatial modeling.

[0025] In the data preprocessing process, the data from different sources are first time aligned. Since image data, pollutant monitoring data, and meteorological data usually differ in acquisition frequency and timestamps, a unified time resolution (e.g., hourly or daily) and a resampling strategy are used to map various types of data onto a unified time axis, thereby ensuring that different modal data have consistent temporal semantics at the same time.

[0026] For missing data in the time series, interpolation methods are used to complete the data, ensuring the continuity and integrity of the input data. Specifically, for any missing time t in the time series, its corresponding value is estimated using observations from adjacent known times, expressed as: in, and Represents the observation values ​​at adjacent known times; Indicates time The interpolation result; This interpolation method is based on the assumption that time series changes relatively smoothly over a short period of time. It can restore the continuous trend of data changes without introducing additional model complexity, thereby reducing the adverse effects of missing values ​​on model training and improving the stability of subsequent feature extraction.

[0027] In practical applications, for cases with long periods of continuous missing data or complete missing values, auxiliary completion can be achieved by combining data from nearby monitoring stations, or a zero-filling strategy can be used for initialization, providing conditions for subsequent feature compensation based on spatial relationships. Through the above time alignment and missing value processing, data from different modalities are unified in the time dimension, ensuring the integrity and consistency of data input, providing a reliable data foundation for subsequent multimodal feature extraction and fusion, thereby improving the overall training effect and prediction performance of the model.

[0028] Step S2: Construct a multimodal air pollution concentration prediction model; such as Figure 2 As shown, the multimodal air pollution concentration prediction model includes an image feature extraction branch, a numerical spatiotemporal feature extraction branch, and a regression module. By using a multi-branch structure to perform targeted modeling of different modal data, it is beneficial to fully explore the feature expression capabilities of various types of data and achieve effective fusion in subsequent stages, thereby improving the overall prediction performance.

[0029] The image feature extraction branch is used to extract spatial feature representations from image data. This branch is a ResNet-18-based convolutional neural network, and a self-calibrating self-attention module (SCSA) is integrated after the intermediate and terminal layers of the ResNet. The SCSA module is used to adaptively recalibrate the convolutional features in terms of spatial and channel dimensions. Its calculation process is as follows: in, This is the output feature map of the ResNet convolutional layer. The SCSA module calibrates the feature map through spatial attention-first computation and a self-attention-based channel attention mechanism to obtain... Subsequently, the visual spatial features extracted by the CNN and SCSA modules are linearly mapped to output the feature vector of the image modality. By weighting the feature maps using a spatial attention mechanism, the model can focus on areas in the image related to air pollution, such as high aerosol areas or pollution accumulation areas, thereby enhancing spatial discrimination capabilities. Subsequently, a channel attention mechanism based on self-attention redistributes the importance of different channels, strengthening key semantic information while suppressing redundant information. This "spatial first, channel second" processing order better aligns with the hierarchical rules of visual information extraction, helping to improve the effectiveness and robustness of feature representation. After processing by a convolutional network and a self-calibrating attention module, high-dimensional features are compressed into fixed-dimensional vectors through linear mapping, resulting in image modality feature vectors, which facilitate fusion with other modalities.

[0030] The numerical spatiotemporal feature extraction branch is used to encode pollutant time series data, meteorological time series data, and geographical location information to obtain spatiotemporal feature representations of each monitoring station. The numerical spatiotemporal feature extraction branch includes Temporal Convolutional Network (TCN), Long Short-Term Memory Network (LSTM), and Graph Attention Network (GAT).

[0031] The processing steps of the numerical spatiotemporal feature extraction branch specifically include: The pollutant time-series data from each monitoring station were coded, for the first... The pollutant time series data from each monitoring station were used to extract time-dependent features through a Temporal Convolutional Network (TCN). The TCN employs a convolutional structure including causal convolution, dilated convolution, and residual connections. The TCN is implemented through a series of convolutional blocks with causal convolution, weight normalization, and residual connections. The computation of its core residual block can be expressed as follows: in The learned residual mapping consists of two layers of convolutional networks with dilated convolutions.

[0032] Meteorological time series data are encoded and input into a Long Short-Term Memory (LSTM) network to obtain meteorological feature representations. LSTM effectively captures the long-term temporal dependence of meteorological factors through its internal forget gate, input gate, and output gate structure. Its core computational process is as follows: Finally, take the hidden state of the last time step of the LSTM. As a feature vector of meteorological modes.

[0033] Geographic location information is encoded by inputting the latitude and longitude coordinates of each monitoring station into a multilayer perceptron (MLP) for mapping, thereby obtaining a location feature representation. Each monitoring station is treated as a node, and a spatial graph structure is constructed based on the spatial distance relationship between the monitoring stations. When the distance between nodes is less than a preset threshold, a connection is established between the corresponding nodes to determine the node. The set of neighboring nodes ; The pollutant time characteristics, meteorological characteristics and location characteristics corresponding to each monitoring station are spliced ​​together as the initial feature representation of the corresponding node. For the target monitoring station, zero-filling is used to complete the data when pollutant time series data is missing. The initial feature representations of the nodes are input into the graph attention network GAT for spatial feature aggregation. The calculation of the attention coefficients between nodes is expressed as follows: in, and Representing nodes respectively With nodes Feature representation, It is a linear transformation matrix. For attention weight parameters, For nodes The set of neighboring nodes; Represents a node With nodes The unnormalized attention coefficients between them; Represents a node With nodes The normalized attention coefficient; This is the adjustment coefficient for the physical prior weights; For nodes With nodes Physical prior weights; The physical prior weights are calculated using the following formula: in, Represents a node With nodes Spatial distance between them; Represents a node j Relative to node i The angle between the direction of the wind and the wind direction; Preset weighting coefficients; To prevent constants with a denominator of zero.

[0034] This physical prior weight, based on the pollutant diffusion mechanism, models the influence intensity between nodes and participates in the spatial feature aggregation process as a bias term in the attention mechanism, thereby improving the physical interpretability and prediction accuracy of the model.

[0035] First item This is a distance decay term based on a Gaussian function, used to describe the gradual attenuation of pollutants as distance increases during spatial propagation. By modeling the exponential decay on the square of the distance, closer nodes can be given greater weight, while the influence of farther nodes decreases rapidly, thus conforming to the physical characteristic of "more significant influence at closer distances" in actual pollution diffusion.

[0036] Second item This is used to characterize the impact of wind direction on pollutant transport. Since pollutant diffusion in the atmosphere has a significant directionality, propagation along the prevailing wind direction is more pronounced. Therefore, a cosine function is used to describe the angular relationship between the direction of the nodal connection and the wind direction. When smaller, When the value is close to 1, the corresponding node is located downwind. In this case, the value is larger, indicating that node j has a stronger influence on node i; when... When the value is close to π, it indicates that the node is in the upwind direction. This value is small, thus reducing its influence.

[0037] Third item This is an inverse distance-based enhancement term used to further amplify the role of nearest-neighbor nodes. During pollutant diffusion, extremely close sites often exhibit stronger correlations, and relying solely on Gaussian attenuation may be insufficient to highlight this strong local correlation. Therefore, this term is introduced to provide additional enhancement for nearby nodes. When the distance is small, this term has a larger value, thus increasing the weight of neighboring nodes in feature aggregation; as the distance increases, this term gradually weakens, complementing the Gaussian decay term.

[0038] By combining the above three components, the physical prior weights simultaneously consider three key factors: spatial distance attenuation, wind direction guidance, and local neighborhood reinforcement. This makes the association modeling between nodes more consistent with the actual atmospheric pollution diffusion process. This design not only overcomes the shortcomings of traditional graph structures where adjacency relationships are fixed or rely solely on data-driven learning, but also provides stable prior constraints even with sparse or noisy data, thereby improving the model's robustness and generalization ability. Introducing this weight into the graph attention network, the attention coefficients not only reflect feature similarity but also incorporate physical mechanism information, making the spatial feature aggregation results more reasonable and contributing to improving the accuracy and reliability of air pollutant concentration prediction.

[0039] The node update features are obtained by weighted aggregation of neighborhood node features based on attention coefficients, and are expressed as follows: in, For nodes Updated node features For activation functions; Will As a node Spatiotemporal characteristics of the corresponding monitoring stations.

[0040] The regression module is a multilayer perceptron structure used to map and regress the fused image feature representation and numerical spatiotemporal feature representation, and output the predicted air pollutant concentration values ​​of each monitoring station at the target time.

[0041] Step S3: The multimodal air pollution concentration prediction model is trained end-to-end by constructing a joint loss function to obtain the trained multimodal air pollution concentration prediction model, specifically including: Construct a training dataset by organizing historical multimodal input data in chronological order, based on each target time point. t The image data corresponding to that moment, as well as the pollutant time series data and meteorological time series data within a preset time window prior to that moment, are extracted and combined with the geographical location information of each monitoring station to construct a sample data pair. ,in, Indicates the first i Multimodal input data of one sample Indicates the first i The air pollutant concentration observation value of each sample at the next moment. The air pollutant concentration observation value is a vector composed of the fine particulate matter concentration, inhalable particulate matter concentration and main gaseous pollutant concentration observed by the monitoring station at the target time. The training dataset is divided into training, validation, and test sets for model training, parameter tuning, and performance evaluation. Preprocessing of multimodal input data includes time alignment, missing value imputation, and time resolution unification. Missing values ​​in the time series are imputed using interpolation methods, expressed as follows: in, and Represents the observation values ​​at adjacent known times; Indicates time The interpolation result; The preprocessed multimodal data is input into the multimodal air pollution concentration prediction model. Image feature representation and spatiotemporal feature representation are extracted through the image feature extraction branch and the numerical spatiotemporal feature extraction branch, respectively. The features are then fused to obtain the predicted value of air pollutant concentration. Based on the predicted and observed values ​​of air pollutant concentrations, a joint loss function is calculated. Based on the joint loss function, the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The model parameters are then updated using an optimization algorithm until the joint loss function converges, thus obtaining the trained multimodal air pollution concentration prediction model.

[0042] The joint loss function includes prediction loss and contrastive loss, expressed as: in, For the joint loss function; To predict losses; To compare the losses; The formula for predicting loss is: in, Indicates the training batch size; These are predicted values ​​for air pollutant concentrations.

[0043] The formula for comparative loss is: in, Indicates the first i The image features of the sample and the first j Cosine similarity of the numerical spatiotemporal features of each sample in the projection space; Indicates the first i The image features of the sample and the first The cosine similarity of the numerical spatiotemporal features of each sample in the projection space; the projection space is obtained by linear or nonlinear mapping of multimodal features; This is a positive sample mask, with a value of 1 indicating a positive sample. With sample If it is a positive sample, then it is 0; This is a negative sample mask; a value of 1 indicates a negative sample. With sample i A positive sample is a negative sample; otherwise, it is 0. A positive sample is one in which the image features and the numerical spatiotemporal features originate from the same monitoring station within the target time and its preceding time window. A negative sample is one in which the image features and the numerical spatiotemporal features originate from different monitoring stations within the target time and its preceding time window. A stable term to prevent numerical underflow.

[0044] By introducing this contrastive loss, the model optimizes the prediction error while further improving the consistency and discriminativeness of cross-modal features, enabling image information and numerical spatiotemporal information to form a close association in the same semantic space. This enhances the model's ability to represent complex environmental conditions and improves the accuracy and generalization ability of air pollutant concentration prediction.

[0045] Step S4: Input the multimodal input data of the target area to be predicted into the trained multimodal air pollution concentration prediction model, and fuse the output image feature representation with the spatiotemporal feature representation corresponding to the monitoring station to obtain the fused feature representation; wherein, the fused pointer is spliced.

[0046] Step S5: Input the fused feature representation into the regression module, and output the predicted air pollutant concentration values ​​of each monitoring station in the target area at the target time.

[0047] Example 2: This embodiment provides an air pollution concentration prediction system based on multimodal feature fusion, including a data acquisition and preprocessing module, a multimodal feature extraction module, a feature fusion module, and a pollutant concentration prediction module. The modules are connected and interact via data streams to jointly complete the task of predicting air pollutant concentrations.

[0048] The data acquisition and preprocessing module is used to acquire multimodal input data from various monitoring stations within the target area and perform unified processing on the multi-source data. The acquired data includes remote sensing image data, pollutant time-series data, meteorological time-series data, and the geographical location information of the monitoring stations. Since different data sources differ in sampling frequency and timestamps, this module performs time alignment processing on all types of data, mapping them uniformly to the same time axis to ensure consistency across different modalities in the time dimension. Simultaneously, missing values ​​in the pollutant time-series and meteorological data are interpolated or filled to improve data integrity and reduce interference from outlier data on model training.

[0049] The multimodal feature extraction module is used to encode different types of data in a targeted manner, extracting image spatial features and numerical spatiotemporal features respectively. For image data, a feature extraction structure based on convolutional neural networks is adopted, and a self-calibrating self-attention mechanism is introduced into the network to weight and adjust the feature map in the spatial and channel dimensions, thereby enhancing the model's ability to perceive key areas and important semantic information. For pollutant time series data, time-dependent features are extracted through a temporal convolutional structure, enabling the model to capture short-term fluctuations and long-term trends in pollutant concentration changes. For meteorological time series data, long-term dependencies are modeled through recurrent neural networks, thereby reflecting the continuous impact of meteorological factors on the pollutant diffusion process. For geographic location information, spatial coordinates are embedded into a high-dimensional feature space through nonlinear mapping, enabling the model to learn complex spatial distribution characteristics.

[0050] The feature fusion module is used for joint modeling of features from different modalities. In this module, each monitoring station is treated as a node in a graph structure, and connections are built based on the spatial distance between stations to form a spatial topology. During node feature construction, pollutant temporal features, meteorological features, and location features are fused as the initial representation of the node, and a graph attention mechanism is used to weighted aggregate information from neighboring nodes. Physical prior weights based on spatial distance and wind direction information are introduced during attention calculation, ensuring that information transmission between nodes relies not only on data-driven feature similarity but also on the constraints of the physical laws governing pollutant diffusion. This makes the feature fusion process more consistent with actual environmental change mechanisms.

[0051] The pollutant concentration prediction module outputs air pollutant concentration prediction results based on the fused feature representation. This module adopts a multilayer perceptron structure, and performs regression modeling on the fused features through multilayer nonlinear mapping to map high-dimensional features into specific pollutant concentration values, including fine particulate matter, inhalable particulate matter, and various gaseous pollutant concentrations.

[0052] Through the synergistic effect of the above modules, the system can achieve unified processing and deep fusion of multi-source heterogeneous data, and model the concentration of air pollutants by combining temporal dynamic characteristics and spatial diffusion laws.

[0053] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-modal feature fusion-based air pollution concentration prediction method, characterized in that, Includes the following steps: Acquire multimodal input data of the target area to be predicted, and perform time alignment and missing value processing on the multimodal input data; the multimodal input data includes image data, pollutant time series data, meteorological time series data, and geographical location information of each monitoring station in the target area; A multimodal air pollution concentration prediction model is constructed, comprising an image feature extraction branch, a numerical spatiotemporal feature extraction branch, and a regression module, wherein: The image feature extraction branch is used to extract spatial feature representations from the image data; The numerical spatiotemporal feature extraction branch is used to encode the pollutant time series data, meteorological time series data, and geographical location information to obtain the spatiotemporal feature representation of each monitoring station; The multimodal air pollution concentration prediction model is trained end-to-end by constructing a joint loss function to obtain the trained multimodal air pollution concentration prediction model. The multimodal input data of the target area to be predicted is input into the trained multimodal air pollution concentration prediction model. The output image feature representation is fused with the spatiotemporal feature representation corresponding to the monitoring station to obtain the fused feature representation. The fused feature representation is then input into the regression module to output the predicted air pollutant concentration values ​​of each monitoring station in the target area at the target time.

2. The air pollution concentration prediction method based on multi-modal feature fusion according to claim 1, characterized in that, The monitoring stations are air quality monitoring nodes distributed within or around the target area, used to collect pollutant observation data at the corresponding locations; The image data refers to remote sensing image data reflecting the atmospheric environment status of the monitoring nodes, including satellite remote sensing images; The pollutant time series data refers to the pollutant observation data collected by the monitoring station within a continuous time range. The pollutant observation data includes fine particulate matter concentration data, inhalable particulate matter concentration data, and gaseous pollutant concentration data. The meteorological time series data are meteorological data corresponding to the monitoring nodes, including temperature, humidity, wind speed, wind direction, and air pressure; The geographic location information refers to data representing the spatial location of each monitoring station, including latitude and longitude coordinates. 3.The air pollution concentration prediction method based on multi-modal feature fusion according to claim 1, characterized in that, The image feature extraction branch is a convolutional neural network based on ResNet-18, and a self-calibrating self-attention module (SCSA) is integrated after the intermediate and terminal layers of ResNet; the SCSA module is used to adaptively recalibrate the convolutional features in terms of spatial and channel dimensions.

4. The air pollution concentration prediction method based on multi-modal feature fusion according to claim 1, characterized in that, The numerical spatiotemporal feature extraction branch includes a temporal convolutional network (TCN), a long short-term memory network (LSTM), and a graph attention network (GAT).

5. The air pollution concentration prediction method based on multi-modal feature fusion according to claim 4, characterized in that, The processing procedure of the numerical spatiotemporal feature extraction branch specifically includes: The time series data of pollutants of each monitoring site are coded, and for the time series of pollutants of the first monitoring site, time-dependent features are extracted by a time convolution network (TCN) which adopts a convolution structure including causal convolution, hollow convolution and residual connection. The time-dependent features are extracted by a time convolution network (TCN) which adopts a convolution structure including causal convolution, hollow convolution and residual connection. The meteorological time series data is encoded, and the meteorological time series is input into a Long Short-Term Memory (LSTM) network to obtain meteorological feature representations; The geographic location information is encoded, and the latitude and longitude coordinates of each monitoring station are input into a multilayer perceptron (MLP) for mapping to obtain a location feature representation. Each monitoring station is treated as a node, and a spatial graph structure is constructed based on the spatial distance relationship between the monitoring stations. When the distance between nodes is less than a preset threshold, a connection is established between the corresponding nodes to determine the node. The set of neighboring nodes ; The pollutant time characteristics, meteorological characteristics and location characteristics corresponding to each monitoring station are spliced ​​together as the initial feature representation of the corresponding node. For the target monitoring station, zero-filling is used to complete the data when pollutant time series data is missing. The initial feature representation of the nodes is input into the Graph Attention Network (GAT) for spatial feature aggregation. The calculation of the attention coefficients between nodes is expressed as follows: in, and Representing nodes respectively With nodes Feature representation, It is a linear transformation matrix. For attention weight parameters, For nodes The set of neighboring nodes; Represents a node With nodes The unnormalized attention coefficients between them; Represents a node With nodes The normalized attention coefficient; This is the adjustment coefficient for the physical prior weights; For nodes With nodes Physical prior weights; Based on the attention coefficient, the features of neighboring nodes are weighted and aggregated to obtain the node update features, which are expressed as follows: in, For nodes Updated node features For activation functions; The As a node Spatiotemporal characteristics of the corresponding monitoring stations.

6. The air pollution concentration prediction method based on multimodal feature fusion according to claim 5, characterized in that, The physical prior weights are calculated using the following formula: in, Represents a node With nodes Spatial distance between them; Represents a node j Relative to node i The angle between the direction of the wind and the wind direction; Preset weighting coefficients; To prevent constants with a denominator of zero.

7. The air pollution concentration prediction method based on multimodal feature fusion according to claim 1, characterized in that, The regression module is a multilayer perceptron structure, used to map and regress the fused image feature representation and numerical spatiotemporal feature representation, and output the predicted air pollutant concentration values ​​of each monitoring station at the target time.

8. The air pollution concentration prediction method based on multimodal feature fusion according to claim 1, characterized in that, The multimodal air pollution concentration prediction model is trained end-to-end by constructing a joint loss function to obtain the trained multimodal air pollution concentration prediction model, specifically including: Construct a training dataset by organizing historical multimodal input data in chronological order, based on each target time point. t The image data corresponding to that moment, as well as the pollutant time series data and meteorological time series data within a preset time window prior to that moment, are extracted and combined with the geographical location information of each monitoring station to construct a sample data pair. ,in, Indicates the first i Multimodal input data of one sample Indicates the first i The air pollutant concentration observation value of each sample at the next moment, wherein the air pollutant concentration observation value is a vector composed of the fine particulate matter concentration, inhalable particulate matter concentration and main gaseous pollutant concentration observed by the monitoring station at the target time. The training dataset is divided into a training set, a validation set, and a test set for model training, parameter tuning, and performance evaluation. The multimodal input data is preprocessed, including time alignment, missing value imputation, and time resolution unification. Missing values ​​in the time series are imputed using interpolation methods, expressed as follows: in, and Represents the observation values ​​at adjacent known times; Indicates time The interpolation results; The preprocessed multimodal data is input into the multimodal air pollution concentration prediction model. Image feature representation and spatiotemporal feature representation are extracted through the image feature extraction branch and the numerical spatiotemporal feature extraction branch, respectively. Feature fusion is then performed to obtain the predicted value of air pollutant concentration. Based on the predicted and observed values ​​of air pollutant concentrations, a joint loss function is calculated. Based on the joint loss function, the gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm. The model parameters are then updated using an optimization algorithm until the joint loss function converges, thus obtaining the trained multimodal air pollution concentration prediction model.

9. The air pollution concentration prediction method based on multimodal feature fusion according to claim 1, characterized in that, The joint loss function includes prediction loss and contrast loss, expressed as: in, For the joint loss function; To predict losses; To compare the losses; The prediction loss is calculated using the following formula: in, Indicates the training batch size; These are predicted values ​​for air pollutant concentrations.

10. The air pollution concentration prediction method based on multimodal feature fusion according to claim 9, characterized in that, The comparison loss is formulated as follows: in, Indicates the first i The image features of the sample and the first j Cosine similarity of the numerical spatiotemporal features of each sample in the projection space; Indicates the first i The image features of the sample and the first The cosine similarity of the numerical spatiotemporal features of each sample in the projection space; the projection space is obtained by linear or nonlinear mapping of multimodal features; This is a positive sample mask, with a value of 1 indicating a positive sample. With sample If it is a positive sample, then it is 0; This is a negative sample mask; a value of 1 indicates a negative sample. With sample i A positive sample is a negative sample if it is negative, otherwise it is zero. A positive sample is one in which the image features and the numerical spatiotemporal features originate from the same monitoring station within the target time and its preceding time window. A negative sample is one in which the image features and the numerical spatiotemporal features originate from different monitoring stations within the target time and its preceding time window. A stable term to prevent numerical underflow.

Images