Full-chain data governance and quality control method for environment management

Abstract

The invention relates to a full-chain data governance and quality control method for environment management, and belongs to the technical field of artificial intelligence. The method comprises the following steps: collecting and marking four types of heterogeneous monitoring data of atmosphere, water quality, weather and remote sensing, and constructing a data set; designing an adaptive normalization function according to the physical characteristics of each data source, and generating a structured feature vector; an environment full-chain monitoring data classification model is constructed, the model comprises a multi-scale adaptive graph convolution unit, a cross-source interactive attention unit and an adaptive gating unit, and a weighted interval binary cross entropy loss function is adopted for training; and normalizing new data, inputting the normalized data into the model, and outputting an environment state probability. According to the method, physical characteristics of data can be reserved, intra-source and cross-source association can be captured, the model robustness and classification precision are improved, closed-loop management and control of data quality are realized, and support is provided for environment management decisions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a full-chain data governance and quality control method for environmental management. Background Technology

[0002] With the continuous advancement of environmental monitoring data acquisition technology, various types of environmental data sources, such as air quality, water quality, meteorological data, and remote sensing data, are increasingly becoming important bases for environmental management and decision-making. These data sources not only have complex spatiotemporal distributions but also contain rich physical and chemical information, thus requiring efficient processing methods to extract effective features and reveal the intrinsic relationships between data.

[0003] Existing technologies suffer from the following drawbacks: Existing global normalization methods tend to ignore the physical relationships and characteristics between data sources, resulting in the fused model failing to effectively capture the sensitive change patterns of different data sources, thus affecting model performance; conventional graph convolutional networks typically rely on fixed graph structures, making them unable to adapt to complex dynamic relationships between data sources and difficult to flexibly model complex internal connections within data sources, thus reducing the model's expressive power; existing environmental state classification methods usually only focus on a single data source, lacking modeling of cross-source interaction effects between multiple data sources, making it difficult to comprehensively consider the combined impact of environmental monitoring data and lacking effective mechanisms for handling data anomalies; conventional loss functions often ignore the cost of misclassification between different categories, failing to make appropriate decisions for classification tasks of different environmental states, resulting in low classification accuracy, especially in cases of class imbalance or where the cost of misclassification for certain environmental states is high. Summary of the Invention

[0004] To achieve the above objectives, the present invention employs the following technical solution: This invention provides a full-chain data governance and quality control method for environmental management, comprising the following steps: S1. Collect historical monitoring data covering four heterogeneous data sources: air, water quality, meteorology, and remote sensing; label the environmental status; and construct a dataset. The environmental status is divided into excellent, good, lightly polluted, moderately polluted, and heavily polluted. S2. For each heterogeneous data source, construct an adaptive normalization function based on its physical characteristics and numerical distribution to map the original data to a unified numerical range, while preserving the internal relative relationships and physical meanings of each source data, to obtain the normalized feature vectors of the four heterogeneous data sources, and then obtain the structured feature vectors. S3. Construct a classification model for environmental monitoring data across the entire chain. Input the structured feature vector into the model and build a dynamic adaptive feature map for each data source. Use learnable multi-scale similarity calculation and fusion to generate an adjacency matrix. Apply graph convolution to aggregate neighborhood information to obtain an enhanced feature matrix. The enhanced feature matrices of the four data sources are modeled using an interactive attention mechanism and the physical correlation between multi-source features is fused to obtain the final fused feature vector. Use an adaptive gating unit to dynamically weight the final fused feature vector based on the quality of the original data before inputting it into a multilayer perceptron for classification. The loss function used is the interval binary cross-entropy loss function. S4. By iteratively adjusting all trainable parameters of the model through optimization algorithms and minimizing the loss function, a well-trained environmental full-chain monitoring data classification model is obtained. S5. After processing the newly collected data stream, input it into the trained environmental full-chain monitoring data classification model, and output the probability that the current environmental condition belongs to each preset state category.

[0005] Furthermore, in S2, a multi-source monitoring data input vector is constructed, including an air quality monitoring vector. Water quality monitoring vector Meteorological monitoring vectors Remote sensing monitoring vector The air quality monitoring vector include , , , , and Concentration values ​​of six pollutants; the water quality monitoring vector. This includes four water quality indicators: pH, dissolved oxygen, chemical oxygen demand, and ammonia nitrogen; the meteorological monitoring vector... This includes five meteorological parameters: temperature, humidity, wind speed, wind direction, and air pressure; the remote sensing monitoring vector... It includes four environmental parameters retrieved from remote sensing: normalized difference vegetation index, surface temperature, aerosol optical thickness, and improved normalized difference water index.

[0006] Furthermore, the atmospheric quality monitoring vector is processed based on a piecewise S-shaped normalization function to obtain a normalized atmospheric feature vector. This piecewise S-shaped normalization function adaptively maps the pollutant concentration to the standard threshold based on the dynamic relationship between pollutant concentration and the standard threshold, amplifying key changes in pollutant concentration near the standard threshold. The water quality monitoring vector is normalized using an arctangent transform based on the ideal value to obtain a normalized water quality feature vector, giving the data higher sensitivity near the ideal value. For meteorological and remote sensing monitoring vectors, normalization is performed using a linear scaling method based on physical upper and lower limits, according to the physical meaning and typical variation range of their parameters, to obtain normalized meteorological and remote sensing feature vectors. The normalized features from the four heterogeneous data sources are concatenated to obtain a structured feature vector. .

[0007] Furthermore, the input structured feature vector is split into independent feature sub-vectors according to its physical source; the feature sub-vectors of each data source are mapped to three different feature spaces through learnable linear projection; and the original feature sub-vectors of each data source are mapped to different feature spaces through three different learnable linear transformations to obtain feature representations at multiple scales, thus obtaining the representation matrix.

[0008] Furthermore, based on the representation matrix, the cosine similarity of any two feature indices within the data source at each scale is calculated. The similarity is then adjusted using a temperature coefficient and normalized using the Softmax function to obtain a normalized similarity matrix. The normalized similarity matrix is ​​then weighted and fused using learnable weights to obtain an adaptive adjacency matrix. Finally, the adaptive adjacency matrix is ​​used to perform graph convolution on the feature sub-vectors. By aggregating information from neighboring nodes, the features of each node are updated and enhanced to obtain an enhanced feature matrix.

[0009] Furthermore, the enhanced feature matrices of the four data sources are flattened into feature vectors along the feature dimension; an interactive attention module is designed, which calculates the attention weights of any data source feature affected by all other data source features, and iteratively enhances the fused features of different data sources through a gating mechanism and residual connection, and concatenates the iteratively enhanced feature vectors of each data source to obtain the final fused feature vector.

[0010] Furthermore, the final fused feature vector is then modulated by an adaptive gating vector and input into a multilayer perceptron for classification to obtain the probability that the sample belongs to each environmental state category.

[0011] Furthermore, the adaptive gating vector is generated by the confidence scores of each data source through a preset small multilayer perceptron.

[0012] Furthermore, a weighted interval binary cross-entropy loss function is adopted, which introduces penalty weights based on environmental standard level and historical occurrence frequency for each category to distinguish the severity of misjudgment for different categories.

[0013] The advantages of this invention are: This invention employs an adaptive normalization and structuring processing method based on multi-source heterogeneous data. For four different types of data sources—atmospheric, water quality, meteorological, and remote sensing data—it proposes personalized normalization methods that preserve the internal relationships and physical characteristics of each data source, avoiding information loss and data inconsistency problems caused by conventional normalization methods. A multi-scale adaptive graph convolutional network is used for intra-source feature extraction. Based on the complex index correlations within the data sources, the adaptive graph structure captures the unique change patterns of each data source, improving the ability to express intra-source features. An interactive attention mechanism and adaptive gating units are introduced in environmental state determination, effectively fusing the cross-source nonlinear coupling relationships between multiple data sources. Dynamic weighting further enhances the model's robustness to local data anomalies, avoiding the oversensitivity of conventional methods to abnormal data. A weighted interval binary cross-entropy loss function is used for multi-label environmental state classification tasks, solving the class imbalance problem. Furthermore, a penalty weight for misclassification of different categories based on environmental standard levels is introduced, further enhancing the model's accuracy and discriminative ability in classifying different environmental states. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0015] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 Flowchart for constructing structured feature vectors; Figure 3 This is a graph showing the real-time dynamic evaluation results of the method of the present invention; Figure 4 This diagram illustrates the contribution of each component to the final classification performance. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 In this embodiment, as Figure 1 As shown, this invention provides a full-chain data governance and quality control method for environmental management, the specific steps of which include: S1. Full-chain data collection and training dataset construction Historical monitoring data from four heterogeneous data sources—atmosphere, water quality, meteorology, and remote sensing—were collected. The data collection process first required determining representative geographical areas and time spans to ensure that the dataset could comprehensively cover environmental conditions under different seasons, meteorological conditions, and pollution levels.

[0018] Air quality monitoring data is collected through a network of ambient air quality monitoring stations deployed in the target area. Water quality monitoring data is collected through automatic water quality monitoring stations set up in key rivers, lakes, and reservoirs. Meteorological monitoring data is collected in conjunction with local meteorological observation stations. Remote sensing monitoring data is collected by receiving and processing multi-source satellite remote sensing images. All data undergoes rigorous time synchronization and spatial alignment preprocessing after collection to ensure that the four types of data at the same timestamp correspond to the same spatial location or region, thereby forming multi-source data pairs with spatiotemporal consistency.

[0019] After completing the raw data collection and alignment, each synchronous data sample needs to be labeled with its environmental status to construct the labels required for supervised training. The labeling categories are defined according to national environmental quality standards and ecological health assessment guidelines. For example, the overall environmental status is divided into five categories: "Excellent," "Good," "Lightly Polluted," "Moderately Polluted," and "Heavily Polluted." The labeling process typically combines standard thresholds for indicators from various data sources with expert knowledge for comprehensive judgment. For instance, it combines atmospheric pollutant concentration classifications, water quality standards, and remote sensing ecological index ranges to label the overall environmental status at each time point with multiple or single labels.

[0020] The labeled dataset is divided into a training set, a validation set, and a test set to complete the construction of the training dataset.

[0021] S2, Adaptive Normalization and Structuring The environmental monitoring data comes from four heterogeneous data sources: atmosphere, water quality, meteorology, and remote sensing. Each data source has different physical dimensions, numerical ranges, and statistical distribution characteristics. Conventional global normalization methods can obscure the structural relationships within each source and ignore the physical constraints between indicators, making it difficult for subsequent fusion models to capture the unique sensitive change patterns of each data source.

[0022] This invention constructs an adaptive normalization function based on the physical characteristics and numerical distribution of each heterogeneous data source, mapping the original data to a unified numerical range while preserving the internal relative relationships and physical meaning of each source data. Figure 2 As shown, the specific steps are as follows: 1) Constructing multi-source monitoring data input vectors It collects raw monitoring data from four heterogeneous data sources: atmosphere, water quality, meteorology, and remote sensing. Specifically, it defines... This represents an air quality monitoring vector, containing concentration values ​​for six pollutants, namely... , , , , and This is used to organize atmospheric source data into a model-processable format; This represents a water quality monitoring vector, which includes four water quality index values: pH, dissolved oxygen, chemical oxygen demand, and ammonia nitrogen. It is used to organize water quality source data into a form that the model can process. This represents a meteorological monitoring vector containing five meteorological parameter values: temperature, humidity, wind speed, wind direction, and air pressure. It is used to organize meteorological source data into a form that the model can process. This represents a remote sensing monitoring vector containing four environmental parameters retrieved from remote sensing: NDVI (Normalized Difference Vegetation Index), LST (Land Surface Temperature), AOD (Aerosol Optical Depth), and MNDWI (Modified Normalized Difference Water Index). These parameters are used to organize remote sensing source data into a model-processable format.

[0023] In one implementation, , , , The unit is micrograms per cubic meter (µg / m³) ), The unit is micrograms per cubic meter (µg / m³) ), The unit is milligrams per cubic meter (mg / m³) pH is dimensionless, and dissolved oxygen is measured in milligrams per liter (mg / L). Chemical oxygen demand (COD) is measured in milligrams per liter (mg / L). Ammonia nitrogen is measured in milligrams per liter (mg / L). The unit of temperature is Celsius. Humidity is expressed as a percentage. The unit of wind speed is meters per second (m / s). The unit for wind direction is degrees. The unit of air pressure is hectopascal (hPa). NDVI is dimensionless, with a theoretical range of [-1, 1]. LST is measured in degrees Celsius. ), AOD is dimensionless, MNDWI is dimensionless, and the theoretical range is [-1,1].

[0024] 2) Piecewise S-shaped normalization of air quality monitoring vectors The air quality monitoring vector is processed using a piecewise S-shaped normalization function. This function adaptively maps the pollutant concentration based on the dynamic relationship between pollutant concentration and standard threshold, amplifying key changes in pollutant concentration near the standard threshold. It is expressed as follows: , In the formula, Indicates the first Normalized values ​​of various pollutants, output range in The interval represents the pollutant concentration level after adaptive mapping; the closer the value is to 1, the higher the degree of pollution. Indicates an index of pollutant types. , respectively corresponding , , , , and ; The slope adjustment factor for atmospheric data is a positive real hyperparameter that controls the steepness of the S-shaped function near the center point. An example value is 5.0. The larger the value, the stronger the amplification effect on small changes in concentration near the threshold. Indicates the first The raw concentration values ​​of these pollutants are part of the air quality monitoring vector. The One eigenvalue; Represents the natural exponential function; Indicates the first The dynamic weighted center of each pollutant, used to adaptively adjust the mapping center point of the normalization function, is obtained by calculating the weighted average of four graded thresholds, reflecting the standard interval in which the current pollution level falls, and is expressed as follows: ; Indicates the first The first pollutant corresponding to the Each air quality standard classification threshold These correspond to concentration thresholds for excellent, good, lightly polluted, and moderately polluted levels, respectively, and are used to provide a normalized physical reference standard. For example, using... ( Taking (e.g.) as an example, its four tiered thresholds are preset as follows: (Excellent) Corresponding , (Good) Correspondence , (Light pollution) Corresponding , (Moderate pollution) Corresponding to ; Indicates the first The pollutants in the first The adaptive weights at each threshold are calculated using a Gaussian kernel function, representing how close the current pollutant concentration is to the threshold. A larger weight indicates a greater impact of that threshold on the current sample. The calculation method is expressed as follows: ; This represents the bandwidth parameter of the Gaussian kernel function, which controls the rate at which the weights decay with distance. It is a hyperparameter, with an example value of 2.0. The larger the value, the slower the weight decay.

[0025] It should be noted that the impact of air pollutant concentrations on environmental health is not linear, but rather there are key inflection points near concentration standard thresholds. For example, when PM2.5 concentrations reach a certain level... 2.5 When the concentration rises from near the "good" threshold (e.g., 75 μg / m³) to the "lightly polluted" threshold, its impact on human health and the environment increases significantly. The piecewise sigmoid normalization function has the characteristic of a large gradient near the center point and a gentle gradient at both ends. This can be achieved by dynamically weighting the center... Setting it to a value related to a standard threshold can adaptively amplify small changes in concentration near the threshold (i.e., more dramatic changes in output value), while compressing concentration changes far below or far above the threshold range (i.e., smoother changes in output value).

[0026] 3) Arctangent normalization of water quality monitoring vectors Considering that some indicators of the water quality monitoring vector exhibit nonlinear biochemical effects, an arctangent transform based on the ideal value is used to normalize the water quality monitoring vector, thereby increasing the sensitivity of the data near the ideal value, as expressed below: , In the formula, Indicates the first Normalized values ​​of water quality indicators, output range mapped to The interval represents the water quality index level after adaptive mapping; This represents a water quality index. These correspond to pH value, dissolved oxygen, chemical oxygen demand and ammonia nitrogen, respectively; The scale factor representing water quality data is a positive real hyperparameter used to adjust the slope of the arctangent function near the center point. An example value is 2.5. The larger the value, the steeper the function is near the ideal value, and the higher the sensitivity to deviation. Indicates the first The original measured values ​​of each water quality indicator are water quality monitoring vectors. The One eigenvalue; Indicates the first Ideal or standard reference values ​​for water quality indicators, such as setting the ideal value of pH to 7.0, are used to center the indicator values ​​so that the normalized results are symmetrical around the ideal value. This represents the arctangent function, which maps the input to... Interval.

[0027] It should be noted that water quality indicators usually have an ideal ecological health value or standard value. The further the indicator value deviates from the ideal value, the more nonlinearly the potential harm to the aquatic ecosystem may increase. The arctangent function has the characteristics of bounded output range and approximately linear near zero, and tends to saturate when far from zero. It will not produce huge output fluctuations due to extreme outliers.

[0028] 4) Adaptive normalization of meteorological monitoring vectors and remote sensing monitoring vectors For meteorological monitoring vectors and remote sensing monitoring vectors, a normalization method based on linear scaling using physical upper and lower limits is adopted to normalize their values ​​according to the physical meaning and typical variation range of their parameters. This aims to standardize their values ​​to a dynamic range similar to that of atmospheric and water quality data. Specifically, the following definitions are provided. This represents a normalized meteorological feature vector containing 5 features, whose values ​​are obtained by linear scaling based on physical upper and lower limits. This represents a normalized remote sensing feature vector containing four features whose values ​​have been scaled or normalized based on physical extent.

[0029] In practical implementation, for each parameter in the meteorological monitoring vector, reasonable physical lower limit and physical upper limit values ​​are preset according to its known physical meaning and typical observation range. For example, if the temperature range of a certain place may be between -20°C and 45°C, then during normalization, the original measured value of each parameter is normalized linearly and the result is mapped to the [0,1] interval. For wind direction as an angle value, trigonometric transformation is required before normalization.

[0030] In practical implementation, for each environmental inversion parameter in the remote sensing monitoring vector, the lower and upper limits are determined based on its physical definition and effective range. For example, if the theoretical range of NDVI is [-1,1], then by normalization and linear scaling, the result is mapped to the [0,1] interval.

[0031] 5) Construct structured feature vectors The normalized features from the four data sources are concatenated to form a unified high-dimensional structured feature vector, which is defined as follows. Represents the structured feature vector, i.e. It has 19 dimensions and integrates information from all heterogeneous data sources, among which... This represents the normalized atmospheric eigenvector; Represents the normalized water quality feature vector; Represents the normalized meteorological feature vector; This represents the normalized remote sensing feature vector.

[0032] S2. Construct a classification model for environmental monitoring data across the entire supply chain. S201. In-source feature extraction based on multi-scale adaptive graph convolution There are complex relationships between monitoring indicators within each data source. Conventional fully connected layers are difficult to explicitly model such relationships, while conventional graph convolutional networks rely on fixed predefined graph structures and cannot be applied to scenarios where relationships are unknown and adaptive learning from data is required.

[0033] This invention constructs a dynamically adaptive feature map for each data source, generates an adjacency matrix through learnable multi-scale similarity calculation and fusion, and applies graph convolution to aggregate neighborhood information, thereby enhancing the representation ability within each source and capturing its unique patterns. The specific steps are as follows: 1) Extracting sub-vectors from source data The input structured feature vector is split into independent feature sub-vectors according to its physical source, and a set of learnable parameters for constructing the adaptive graph structure of each source is initialized. Specifically, the parameters are defined as follows: The atmospheric quality feature vector is derived from the structured feature vector. Extracted from the front The dimension of the atmospheric quality feature subvector is a vector composed of elements. ; The water quality feature subvector is derived from the structured feature vector. Immediately following Extracted later A vector composed of elements, the dimension of the water quality feature subvector. ; The meteorological feature subvector is derived from the structured feature vector. Immediately following Extracted later The dimension of the meteorological feature subvector is a vector composed of elements. ; Representing remote sensing feature vectors is derived from structured feature vectors. Immediately following Extracted later The dimension of the remote sensing feature subvector is a vector composed of elements. .

[0034] 2) Calculate multi-scale feature representation By using learnable linear projection, the feature vectors of each data source are mapped to three different feature spaces. Through three different learnable linear transformations, the original feature vectors of each data source are mapped to different feature spaces to obtain multi-scale feature representations, expressed as follows: , In the formula, Indicates data source In the The representation matrix at the scale has dimensions of , The Behavior Vector , Indicates data source The The first characteristic index in the th The representation vector at the scale has a dimension of This is used to capture the feature information of the indicator at a specific scale; This represents the preset hidden layer dimension, which is a hyperparameter. An example value is 16. Indicates data source The dimension of the feature vectors; express A column vector of all 1s; Represents the ReLU activation function, which is a modified linear unit activation function that introduces nonlinearity to enhance the model's expressive power. Indicates the data source identifier. These correspond to four data sources: atmosphere, water quality, meteorology, and remote sensing. Indicates the index of feature indicators; Indicates scale index. These correspond to local, meso-level, and global scales, respectively. Indicates data source The eigenvectors, i.e. , dimension ; Indicates targeting the data source ,scale The projected weight vector is a trainable parameter used to map the original features to a scale-specific feature space with dimensions of . , express Transpose of; Indicates targeting the data source ,scale The bias vector is a trainable parameter with dimension . This is used to add an offset after feature projection. express The transpose of .

[0035] It should be noted that by using learnable linear projection to map features to spaces of different scales (local, meso, and global), the aim is to capture potential relationship patterns between feature indicators from different granularities or perspectives. For example, at the local scale, the model may focus on the detailed patterns of short-term synergistic changes between pollutants, while at the global scale, the model may capture the long-term, overall correlation structure between all indicators.

[0036] 3) Construct a multi-scale similarity matrix Based on the representation vectors across multiple scales, the cosine similarity of any two feature indicators within the data source is calculated at each scale. This is then adjusted using a temperature coefficient and normalized using the Softmax function to obtain the normalized similarity matrix at that scale, denoted as follows: , In the formula, Indicates data source In the The normalized similarity matrix at the scale has dimensions of . Its element value is If the sum of each row is 1, then the following are considered as candidates for constructing the graph structure. express The Middle Line number The elements of the column represent the first element. The first indicator and the first The first indicator in the Similarity weights at different scales; This represents the cosine similarity function, used to calculate the cosine of the angle between two vectors, ranging from... , used to measure feature similarity; Indicates data source The temperature coefficient, used to adjust the sharpness of the distribution in similarity calculation, is a trainable parameter; Indicates data source The The first characteristic index in the th The representation vector at the scale.

[0037] 4) Integrate multi-scale similarity to generate an adaptive adjacency matrix By weighting the normalized similarity matrix with learnable weights, the model adaptively learns and integrates relational information at different scales, thereby generating a more comprehensive and robust adaptive adjacency matrix, expressed as: , In the formula, Indicates data source An adaptive adjacency matrix with dimensions of This characterizes the comprehensive similarity relationship among indicators within the source; Indicates data source In the Importance weights at different scales, satisfying , is used to quantify the contribution of different scales to the adaptive adjacency matrix, and is a trainable parameter.

[0038] 5) Perform adaptive graph convolution operation By performing graph convolution on the feature vectors using an adaptive adjacency matrix, the features of each node are updated and enhanced by aggregating information from neighboring nodes, resulting in an enhanced feature matrix, represented as: , In the formula, Indicates data source The enhanced feature matrix is ​​the data source. The output feature matrix after graph convolution enhancement represents a more abstract and powerful environmental state feature that integrates complex correlations between in-source indicators. This corresponds to the four enhanced feature matrices output by the graph convolution operation. , dimension ; This represents the dimension of the output features after the graph convolution operation. It is a preset hyperparameter, with an example value of 32. express The degree matrix is ​​a diagonal matrix. The i-th diagonal element , indicating the first In layman's terms, the degree of a node indicates that the higher the degree of an indicator, the more "central" or "active" it is in the relational network within the source, and its characteristics will be influenced by more neighbor indicators during the information aggregation process. Represents the adaptive adjacency matrix The Middle Line number The elements of the column represent the first element. The node (the first) Indicators) and the first The node (the first) The overall connection strength or similarity weight between the indicators; This represents the weight vector of the graph convolutional layer, with dimension . , used for linear feature transformation, are trainable parameters. The dimension of the item is .

[0039] It should be noted that, Item representation pairs Symmetric normalization is used to address the numerical instability caused by uneven node degree distribution in graph convolution. Without normalization, the feature values ​​of nodes with high degree will become very large after feature aggregation, while those of nodes with low degree will change very little, leading to gradient explosion or vanishing. Symmetric normalization ensures that the total amount of information received by each node from its neighbors is normalized during information aggregation, regardless of the node's own degree, thus stabilizing the training process and usually resulting in better performance.

[0040] It should also be noted that in the graph structure of this invention, each node represents a specific environmental monitoring indicator. For example, in the graph corresponding to the air quality data source, there are 6 nodes, each representing... , , , , and These are the normalized concentration characteristics of the six pollutants.

[0041] It should also be noted that the core idea of ​​graph convolution is to utilize graph structures (adaptive adjacency matrices). The relationships between nodes defined by the algorithm update and enhance the features of each node by aggregating information from neighboring nodes. Nodes are different monitoring indicators within the same data source, so that the feature representation of each indicator not only includes its own information, but also incorporates information from other highly correlated indicators, thereby modeling and strengthening the physical or statistical correlation between indicators within the source and improving the expressive power of features.

[0042] S202, Comprehensive Assessment of Environmental Status The comprehensive assessment of environmental conditions depends not only on the correlation of indicators within each data source, but also on the complex nonlinear cross-source interaction effects between different data sources such as air, water quality, meteorology, and remote sensing. Conventional feature splicing followed by fully connected networks is difficult to explicitly model such cross-source nonlinear coupling relationships, and lacks robustness to local sensor failures or instantaneous data anomalies that may occur during data acquisition.

[0043] This invention constructs a dual-path classification model based on an interactive attention mechanism and an adaptive gating unit. The interactive attention mechanism models and fuses the physical relationships between multi-source features, while the adaptive gating unit dynamically weights the final fused feature vector based on the quality of the original data to improve the model's robustness to local data anomalies. The specific steps are as follows: 1) Cross-source interactive attention feature fusion Four enhanced feature matrices from atmospheric, water quality, meteorological, and remote sensing sources are flattened along their feature dimensions into feature vectors, and a cross-source feature set is constructed. An interactive attention module is designed, which first calculates the attention weight of any data source feature on the influence of all other data source features. This weight integrates prior knowledge of physical associations based on environmental science and data-driven evidence based on the consistency of trends in the original data. Then, through a gating mechanism and residual connection, deep fusion and iterative enhancement of features from different data sources are achieved. Finally, the enhanced feature vectors from each data source are concatenated to obtain the final fused feature vector, represented as: In the formula, The final fused feature vector not only includes the internal features of each data source after graph convolution enhancement, but also models and fuses the nonlinear coupling relationship between features from different data sources through interactive attention mechanism, representing the overall state of the environmental system after integrating multi-source, cross-scale, and interactively related information at the current moment; Indicates the first The fusion characteristics of the data sources in the second stage, namely, This indicates the fusion characteristics of atmospheric data sources in the second stage; This indicates the fusion characteristics of water quality data sources in the second stage; This indicates the fusion characteristics of meteorological data sources in the second stage; This represents the fusion characteristics of the remote sensing data source in the second stage. Through residual connection and layer normalization processing, it retains its original information and fuses interactive information, as shown below. ; This represents a vector concatenation operation; Indicates the first The fusion features of each data source in the first stage are obtained by aggregating the features of other data sources through attention weighting, and then modulating them with a gating signal generated based on its own features. This is represented as follows: ; Indicates the first The enhanced feature vectors of each data source are generated by the data source. Enhanced feature matrix Flattened, the dimensions are: ; Indicates the first The enhanced feature vectors of each data source are generated by the data source. Enhanced feature matrix Flattened, the dimensions are: ; Presentation layer normalization operation; The attention weights are the attention weights normalized by the Softmax function, satisfying the following conditions: , representing the The data source for the first The combined impact of each data source is calculated as follows: ; Indicates the first The data source for the first The raw attention score from the first data source is used to quantify the first... The characteristics of the data source are helpful in explaining the first... The potential contribution of the current state of each data source is calculated as follows: ; This indicates the index of the query data source. ; Indicates the index of the key-value data source. ; This represents the weight matrix used to generate the query vector, with dimensions [missing information]. , are trainable parameters; This represents the weight matrix used to generate the key vectors, with dimensions of . , are trainable parameters; This represents the weight matrix used to generate the gate vector, with dimensions of . , are trainable parameters; This represents the weight matrix used to generate the value vector, with dimensions of . , are trainable parameters; express Transpose of the item; Indicates the first The data source and the first The prior factors of physical associations between data sources are scalars presupposed based on environmental science knowledge, used to guide attention to learn interaction relationships that conform to physical laws, for example... (Atmosphere-Meteorology) can be set to 0.5; The dimension of the key vector is a hyperparameter, with an example value of 48. Indicates the first The data source and the first The raw data trend consistency measure of two data sources is data-driven evidence of interaction. Its value increases when the trends of change in the two data sources are consistent. The calculation method is expressed as follows: ; Indicates the first The first data source The rate of change of a normalized feature within a short time window, preferably set to the past two time steps; No. The first data source The rate of change of a normalized feature within a short time window, preferably set to the past two time steps; This represents the sigmoid activation function. This represents the element-wise multiplication operator.

[0044] 2) Adaptive gated multi-label environmental state classification The final fused feature vector is input into a multilayer perceptron for classification. To address the multi-label classification task of environmental states and the need for robustness to data quality, an adaptive gating unit based on the confidence level of the original data is introduced before the classification layer. This unit calculates the instantaneous quality confidence score of each data source using the original data and generates a gating vector with the same dimension as the fused feature vector through a pre-defined small multilayer perceptron. This dynamically modulates the features input to the classifier. Then, the main classification multilayer perceptron outputs the probability that a sample belongs to each environmental state category, expressed as: , This represents the multi-label probability vector predicted by the model, calculated using an element-wise sigmoid activation function. , Indicates that the sample belongs to the first The probability of a class environment state; This indicates the total number of environmental status categories, which can be set according to environmental standards, such as: excellent, good, lightly polluted, moderately polluted, and heavily polluted. Indicates category index, ; This represents the gated modulation feature vector, which is the classification feature vector after gated modulation. Based on the instantaneous data quality confidence of each data source at the current moment, the fused features are dynamically and finely reweighted. The calculation method is expressed as follows: ; The category association adjustment matrix is ​​a trainable parameter with dimension . This is used to model the soft associations between different categories of environmental states; The main classifier, a multilayer perceptron, in one implementation consists of three fully connected layers, with intermediate layers using... The activation function introduces non-linearity, and the last layer does not use an activation function; This represents the adaptive gating vector, whose dimension is the same as that of the fused feature vector. Similarly, the confidence scores from each data source are generated using a pre-defined small multilayer perceptron, which is used to dynamically weight the fused features, and is represented as follows: ; This indicates a predefined small multilayer perceptron, used to generate gated vectors. In one implementation, it consists of two fully connected layers, and finally uses... The activation function limits the output to The interval is used to ensure that it serves as a gating weight; Indicates the first Instantaneous confidence scores for each data source, ranging from The range represents the data source; a higher value indicates more reliable data quality in the current batch, and a lower probability of anomalies or noise. The calculation method is expressed as follows: ;Right now, Instantaneous confidence score of atmospheric data sources, Instantaneous confidence score of water quality data source, Instantaneous confidence score of meteorological data sources, Instantaneous confidence score of remote sensing data source; Indicates the first The sensitivity coefficient of each data source is a trainable parameter used to adjust the sensitivity of the confidence score to data mutations. This represents the function that takes the maximum value. Indicates data source The One original monitoring indicator value, express The difference between adjacent time points, express The historical average absolute value on the training set is used for dimensionless transformation; Indicates data source The number of indicators; This represents a very small constant used to prevent the denominator from being zero. Examples of its values ​​are shown below. .

[0045] In one embodiment, such as Figure 3As shown in the figure, the confidence level trend over time visually demonstrates the real-time dynamic evaluation results of the multi-source monitoring data quality by the adaptive gating unit in this invention. The four curves in the figure represent the instantaneous confidence level score fluctuations of the atmospheric, water quality, meteorological, and remote sensing data sources over 24 hours, respectively. The red dashed line indicates the quality alarm threshold of 0.8. It can be seen that the confidence level of each data source is not constant, but exhibits reasonable fluctuations over time.

[0046] S203. Calculate the interval binary cross-entropy loss function. For multi-label environmental state classification tasks, the conventional binary cross-entropy loss function is difficult to handle the differences in the importance of different categories and the deterministic requirements of model prediction. For example, the environmental cost of misclassifying "severe pollution" as "excellent" is much higher than the opposite.

[0047] This invention employs a weighted, interval-based binary cross-entropy loss function. By introducing penalty weights based on environmental standard levels and historical frequency for each category, it differentiates the severity of misclassification across different categories. Simultaneously, by incorporating a margin parameter, it encourages the model to stretch the predicted probabilities of positive and negative samples towards 1 and 0 respectively, thereby enhancing the certainty of predictions and improving the model's discriminative ability and robustness. This is expressed as: , In the formula, This represents the interval binary cross-entropy loss function, which is used to train a classification model for full-chain monitoring data in the environment. Indicates the sample at the 1st Real labels in each category 1 indicates that the environment state exists, and 0 indicates that it does not exist; Indicates the first The penalty weights for each category are used to adjust the importance of different categories in the loss calculation. A larger value indicates a heavier penalty for misclassification of that category. The calculation method is expressed as follows: ; This represents the adjustment hyperparameter of the penalty weight, used to control the magnitude of the weight increment; an example value is 0.5. Indicates the first training set The prior frequencies of each category are statistically analyzed across all samples in the entire training set before model training begins, and are represented as follows: This value is a fixed statistic used to automatically balance the effects of class imbalance in the loss function; Indicates the first The environmental standard levels corresponding to each category are preset by humans. For example, heavy pollution is set to 4, moderate pollution to 3, good to 2, and excellent to 1. This represents the edge margin parameter, a positive hyperparameter used to stretch the predicted probability away from the decision boundary by 0.5, with an example value of 0.1. This represents extremely small numerical boundaries, used to prevent numerical calculation errors caused by zero input to the logarithmic function. Examples of possible values ​​are: ; This represents a logarithmic function, with the default base being the natural constant. This represents the function that takes the minimum value.

[0048] S3, Environmental Full-Chain Monitoring Data Classification Model Training The process of iteratively adjusting all trainable parameters of a model through optimization algorithms to minimize the loss function and improve the model's generalization performance.

[0049] At the start of training, the model parameters are first initialized, for example, by using the Xavier initialization method to initialize all trainable parameters. Then, in each training iteration, multi-source data samples and their corresponding multi-label environmental state ground values ​​are extracted from the training set in batches and input into the model. The forward propagation process of the model sequentially performs adaptive normalization and structuring, source feature extraction based on multi-scale adaptive graph convolution, and comprehensive environmental state determination based on interactive attention and adaptive gating. Finally, the output is the predicted probability vector of the sample belonging to each environmental state category. Then, based on the interval binary cross-entropy loss function defined in S2, the loss function value between the model's predicted value and the true label is calculated. This loss function value not only measures the prediction error, but also distinguishes the environmental cost of different misjudgments through the category penalty weight, and encourages the model to make more certain predictions through the interval parameter.

[0050] The core of training lies in backpropagation and parameter update. The gradient of the loss function with respect to all trainable parameters is calculated through the backpropagation algorithm, and then the parameters are updated according to the gradient direction and the preset learning rate using an adaptive optimization algorithm (such as the Adam optimizer).

[0051] The entire training process typically involves multiple iterations, with each iteration fully traversing the entire training set. During training, a separate validation set is used simultaneously to monitor and evaluate model performance, calculating metrics such as the loss function value and classification accuracy on the validation set.

[0052] The criteria for stopping iterations are usually set based on the following criteria: First, early stopping, when the validation set loss no longer decreases or begins to increase within a predetermined number of consecutive rounds, training is stopped to prevent overfitting; second, the preset maximum number of training rounds is reached; and third, the performance metrics on the validation set (such as the macro average F1 score) reach a preset threshold.

[0053] After training stops, the snapshot of the model parameters that performs best on the validation set is selected as the final trained model, and a final performance evaluation is performed on an independent test set to report the model's true generalization ability.

[0054] In one embodiment, such as Figure 4 As shown, this paper analyzes the contribution of each core component in the model of this invention to the final classification performance, especially the improvement effect on different environmental state categories. The experiment sets up a progressive model configuration comparison: the baseline model uses only a simple fully connected network; the second configuration adds an intra-source feature extraction module based on multi-scale adaptive graph convolution; the third configuration further adds a cross-source feature fusion module based on interactive attention mechanism; and the fourth configuration is the complete model of this invention, including adaptive gating units. The experimental metric is the macro-average F1 score, which combines precision and recall and can better evaluate the model's ability to identify various categories (especially categories where samples may be imbalanced). The experimental results are presented as grouped bar charts, with the horizontal axis representing the five environmental states: "Excellent," "Good," "Lightly Polluted," "Moderately Polluted," and "Heavily Polluted," and the vertical axis representing the "F1 score." The experimental results show that as model components are continuously added, the height of the bars in each group increases in a stepwise manner, with the baseline model's bars being at the lowest position in each category. After adding adaptive graph convolution, all bars showed a significant increase, especially in the "lightly polluted" and more severe pollution categories. This indicates that the module effectively captured the complex correlations between indicators within each data source, enhancing the model's ability to represent pollution states. Adding an interactive attention mechanism further increased the bar height, demonstrating that modeling cross-source nonlinear interactions is crucial for comprehensively determining environmental states. The complete model reached its highest bar height, with the most prominent advantage in the "heavily polluted" category. This verifies that the adaptive gating unit, through confidence weighting, improves the model's robustness to data quality fluctuations, enabling more reliable judgments when facing complex pollution states that may be accompanied by data anomalies.

[0055] S4, End-to-End Data Governance and Quality Control Based on the trained environmental full-chain monitoring data classification model, we can achieve full-chain data governance and quality control for environmental management.

[0056] First, the model is deployed in a real-world environmental monitoring data stream processing system to process multi-source heterogeneous monitoring data (atmosphere, water quality, meteorology, and remote sensing) collected in real-time or near real-time online. For newly input data streams, the data is first preprocessed for time synchronization and spatial alignment as described in S1. Then, the raw data is processed according to the method defined in the "adaptive normalization and structuring" step in S2. This includes segmented S-shaped normalization of atmospheric pollutant concentrations, arctangent normalization of water quality indicators, and linear normalization of meteorological and remote sensing parameters based on physical range. The data is then spliced ​​together to form a unified structured feature vector. Then, the structured feature vectors are input into the trained classification model. The model sequentially performs multi-scale adaptive graph convolutional source feature extraction and comprehensive environmental state determination, and finally outputs the probability that the current environmental state belongs to each preset state category. This determination result can directly serve environmental quality assessment, pollution early warning and decision support.

[0057] It's important to note that the model's internal intermediate processes and outputs provide in-depth insights for data governance and quality control. First, the adaptive normalization step itself can identify and smooth extreme outliers, such as compressing concentration values ​​far exceeding the normal range using a sigmoid function. Second, in the multi-scale adaptive graph convolution module, the model dynamically generates an adaptive adjacency matrix for each data source, revealing the implicit correlations between monitoring indicators; if the correlation weight of a certain indicator with all other indicators remains abnormally low, it may indicate that the sensor data is abnormal or malfunctioning. Furthermore, the attention weights generated by the interactive attention mechanism reflect the interaction strength between different data sources; if the attention contribution of a certain data source (such as a meteorological source) to all other sources remains abnormally high, it may indicate that the current environmental state is dominated by that source or that there is data anomaly. Finally, the instantaneous confidence scores of each data source calculated by the adaptive gating unit directly quantify the real-time reliability of each source's data; excessively low scores trigger data quality alarms.

[0058] Based on the multi-dimensional information provided by the model (classification results, association matrix, attention weights, and confidence scores), the system can construct a closed-loop quality control and governance process. When the model outputs an abnormal environmental state (such as a sudden severe pollution), the attention weights and confidence scores can be combined to trace the source back to the most likely problematic data source or specific indicator. Simultaneously, the system can set a confidence score threshold. When the confidence score of a data source falls below the threshold, the batch of data is automatically marked as "pending verification," and quality control procedures are triggered, such as notifying maintenance personnel to check the corresponding sensors, compare data from nearby sites, or initiate data repair and interpolation processes.

[0059] By continuously running this model, not only can intelligent perception and classification of environmental conditions be achieved, but also quality monitoring, anomaly diagnosis, and credibility assessment of the entire chain of monitoring data can be realized. This improves the reliability, consistency, and value of the entire environmental monitoring data system, ultimately supporting more accurate and robust environmental management decisions.

[0060] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for end-to-end data governance and quality control for environmental management, characterized in that, Includes the following steps: S1. Collect historical monitoring data covering four heterogeneous data sources: atmosphere, water quality, meteorology, and remote sensing; label the environmental status; and construct a dataset. S2. For each heterogeneous data source, construct an adaptive normalization function based on its physical characteristics and numerical distribution to map the original data to a unified numerical range, thereby obtaining the normalized feature vectors of the four heterogeneous data sources, and then obtaining the structured feature vectors. S3. Construct a classification model for environmental full-chain monitoring data. Input the structured feature vector into the classification model for environmental full-chain monitoring data. Construct a dynamic adaptive feature map for each data source. Use learnable multi-scale similarity calculation and fusion to generate an adjacency matrix. Apply graph convolution to aggregate neighborhood information to obtain an enhanced feature matrix. The enhanced feature matrix is ​​modeled using an interactive attention mechanism and the physical correlations between multi-source features are fused to obtain the final fused feature vector. The final fused feature vector is then dynamically weighted based on the quality of the original data using an adaptive gating unit and input into a multilayer perceptron for classification. The loss function adopted is the interval binary cross-entropy loss function. S4. By iteratively adjusting all trainable parameters of the model through optimization algorithms and minimizing the loss function, a well-trained environmental full-chain monitoring data classification model is obtained. S5. After processing the newly collected data stream, input it into the trained environmental full-chain monitoring data classification model, and output the probability that the current environmental condition belongs to each preset state category.

2. The end-to-end data governance and quality control method for environmental management according to claim 1, characterized in that, In S2, a multi-source monitoring data input vector is constructed, including an air quality monitoring vector. Water quality monitoring vector Meteorological monitoring vectors Remote sensing monitoring vector The air quality monitoring vector include , , , , and Concentration values ​​of six pollutants; the water quality monitoring vector. This includes four water quality indicators: pH, dissolved oxygen, chemical oxygen demand, and ammonia nitrogen; the meteorological monitoring vector... This includes five meteorological parameters: temperature, humidity, wind speed, wind direction, and air pressure; the remote sensing monitoring vector... It includes four environmental parameters retrieved from remote sensing: normalized difference vegetation index, land surface temperature, aerosol optical thickness, and improved normalized difference water index.

3. The method for end-to-end data governance and quality control for environmental management according to claim 2, characterized in that, The atmospheric quality monitoring vector is processed by a piecewise S-shaped normalization function to obtain a normalized atmospheric feature vector; the piecewise S-shaped normalization function is used to perform adaptive mapping based on the dynamic relationship between pollutant concentration and standard threshold; the water quality monitoring vector is normalized by an arctangent transform based on ideal values ​​to obtain a normalized water quality feature vector. For meteorological monitoring vectors and remote sensing monitoring vectors, a normalization method based on physical upper and lower limits of linear scaling is used to obtain normalized meteorological feature vectors and normalized remote sensing feature vectors. The normalized features from the four heterogeneous data sources are concatenated to obtain structured feature vectors. .

4. The end-to-end data governance and quality control method for environmental management according to claim 3, characterized in that, The input structured feature vector is split into independent feature sub-vectors according to its physical source; the feature sub-vectors of each data source are mapped to three different feature spaces through learnable linear projection; and the original feature sub-vectors of each data source are mapped to different feature spaces through three different learnable linear transformations to obtain feature representations at multiple scales, thus obtaining the representation matrix.

5. The end-to-end data governance and quality control method for environmental management according to claim 4, characterized in that, Based on the representation matrix, the cosine similarity of any two feature indices within the data source at each scale is calculated. The similarity is then adjusted using a temperature coefficient and normalized using the Softmax function to obtain a normalized similarity matrix. The normalized similarity matrix is ​​then weighted and fused using learnable weights to obtain an adaptive adjacency matrix. The adaptive adjacency matrix is ​​then used to perform graph convolution on the feature sub-vectors. By aggregating information from neighboring nodes, the features of each node are updated and enhanced to obtain an enhanced feature matrix.

6. The end-to-end data governance and quality control method for environmental management according to claim 5, characterized in that, The enhanced feature matrices of the four data sources are flattened into feature vectors along the feature dimensions. An interactive attention module is designed, which calculates the attention weights of any feature from any data source as a result of the influence of features from all other data sources. The fused features from different data sources are iteratively enhanced through a gating mechanism and residual connections. The feature vectors of each data source after iterative enhancement are concatenated to obtain the final fused feature vector.

7. The end-to-end data governance and quality control method for environmental management according to claim 6, characterized in that, The final fused feature vector is then modulated by adaptive gating vector and input into a multilayer perceptron for classification to obtain the probability that the sample belongs to each environmental state category.

8. The end-to-end data governance and quality control method for environmental management according to claim 7, characterized in that, The adaptive gating vector is generated by using a preset small multilayer perceptron based on the confidence scores of each data source.

9. The end-to-end data governance and quality control method for environmental management according to claim 8, characterized in that, We employ a weighted interval binary cross-entropy loss function, and introduce penalty weights based on environmental standard levels and historical occurrence frequencies for each category to distinguish the severity of misjudgments for different categories.

10. The end-to-end data governance and quality control method for environmental management according to claim 9, characterized in that, The environmental status is classified as excellent, good, slightly polluted, moderately polluted, and heavily polluted.

Images