Aerosol monitoring method and system based on clustering processing

Abstract

The invention discloses an aerosol monitoring method and system based on clustering processing. The method comprises the steps of data acquisition, preprocessing, aerosol clustering and aerosol monitoring. The invention belongs to the field of data processing, and particularly relates to an aerosol monitoring method and system based on clustering processing, according to the scheme, a monitoring sub-model is customized for each monitoring station, specific pollution characteristics of the monitoring station are reflected, global regularization is carried out, and local monitoring is limited to deviate from global monitoring; the weight is dynamically adjusted based on the regional importance, the equipment quality and the acquisition time difference to ensure that the monitoring result reflects the latest pollution condition, so that the accuracy and reliability of the monitoring result are improved; the long-term pollution characteristics are reflected by introducing historical trend weights, result distortion caused by local deviation is avoided through global reference positions and correction, the long-term pollution trend can be reflected, short-term changes can be adapted, and then the aerosol monitoring accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an aerosol monitoring method and system based on clustering processing. Background Art

[0002] Aerosol monitoring methods cover the entire process from data collection to analysis and application. By continuously optimizing monitoring technology and data processing methods, we can better understand the distribution and sources of aerosols and their impact on the environment and health, and provide scientific support for improving air quality and addressing climate change. However, general aerosol monitoring methods are unable to capture the differences between different monitoring points, and the global and local monitoring of aerosols are not properly balanced, which leads to poor accuracy of aerosol monitoring results. General aerosol monitoring methods are unable to rely solely on current aerosol monitoring data, making it difficult to capture long-term changes, and the monitoring effect is poor due to local monitoring deviations. Summary of the invention

[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides an aerosol monitoring method and system based on clustering processing. In view of the fact that general aerosol monitoring methods are unable to capture the differences between different monitoring points, and the global monitoring and local monitoring of aerosol monitoring are improperly balanced, which leads to poor accuracy of aerosol monitoring results, this solution tailors a monitoring sub-model for each monitoring site to reflect its unique pollution characteristics, and performs global regularization to limit the deviation of local monitoring from global monitoring. The weights are dynamically adjusted based on regional importance, equipment quality and acquisition time difference to ensure that the monitoring results reflect the latest pollution conditions, thereby improving the accuracy and reliability of the monitoring results; in view of the fact that general aerosol monitoring methods are unable to rely solely on current aerosol monitoring data, it is difficult to capture long-term changes, and the monitoring effect is poor due to local monitoring deviations, this solution introduces historical trend weights to reflect long-term pollution characteristics, and uses global reference positions to correct and avoid distortion of results caused by local deviations. It can not only reflect long-term pollution trends, but also adapt to short-term changes, thereby improving the accuracy of aerosol monitoring.

[0004] The technical solution adopted by the present invention is as follows: The aerosol monitoring method based on clustering processing provided by the present invention comprises the following steps: Step S1: data collection; Step S2: preprocessing; Step S3: aerosol clustering; Step S4: Aerosol monitoring.

[0005] Furthermore, in step S1, the data collection is to collect historical aerosol monitoring data and real-time aerosol monitoring data; the historical aerosol monitoring data and real-time aerosol monitoring data both include aerosol characteristic data, meteorological data and environmental data; the historical aerosol monitoring data also includes monitoring assessment levels; the monitoring assessment levels are used as data labels; data labels are only selected and used as cluster labels and do not participate in clustering operations.

[0006] Furthermore, in step S2, the preprocessing is to perform data cleaning, data conversion and standardization on the collected aerosol monitoring data; the data cleaning is to process missing values ​​and duplicate values; the data conversion is to convert the data into a vector form; and the standardization is to standardize the data based on the maximum and minimum normalization method.

[0007] Furthermore, in step S3, the aerosol clustering specifically includes the following steps: Step S31: sub-model training; at each monitoring site, a clustering sub-model is trained based on fuzzy K-means clustering to identify the local aerosol monitoring mode. Each monitoring device trains a local clustering model based on its locally collected aerosol data; define the quasi-objective function , expressed as: ; Where N is the total number of aerosol monitoring data points collected by the monitoring station; K is the number of cluster centers; is the membership degree of the i1th aerosol monitoring data point to the kth cluster center; is the i1th aerosol monitoring data point; is the kth cluster center; is the square of the L2 norm; m is the fuzzy factor; Step S32: Optimization of objective function: When each monitoring device updates the local sub-model, it must ensure that it does not deviate from the global model to avoid excessive personalization, so the objective function is optimized, which is expressed as: ; ; ;in, is the average location of the cluster center of the ith monitoring station; is the regularization hyperparameter; is the weight of the i-th monitoring point; V is the average position of the global cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; is the historical trend weight; is the optimized objective function; is the quasi-objective function; is a distance metric; Step S33: weight calculation; time decay is introduced, and the weight calculation formula is expressed as: ; Among them, α is the weight adjustment parameter; and Represent the urban area indicator function and the credibility indicator function respectively. If the i-th monitoring point is an urban area, then , if the accuracy of the equipment at the i-th monitoring point meets the standard, then , otherwise it is equal to 0; and is the weight adjustment factor; is the balancing factor; C1 is the total number of categories; and are the data distribution of the kth monitoring point and the gth monitoring point in the cth category respectively; γ is the smoothing parameter; is the time decay factor; is the maximum time difference between historical aerosol monitoring data and real-time aerosol monitoring data; Step S34: Update the cluster center; calculate the membership degree of each aerosol monitoring data point belonging to different clusters, expressed as: ; For each monitoring point, update the cluster center according to the membership weight, and introduce the historical trend weight , expressed as: ; ;in, is the jth cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; Step S35: cluster center correction; aggregate the local model to obtain the global reference position , expressed as: ; Each monitoring point fine-tunes the updated cluster center based on the global reference position, expressed as: ;in, is the learning rate; Step S36: judgment; a maximum number of iterations is set in advance, and the iteration is completed when the global objective function converges or reaches the maximum number of iterations; the global objective function is obtained by weighted summation of the objective functions of the sub-models based on the monitoring point weights; otherwise, the cluster center is re-updated and corrected, and the aerosol monitoring data points are allocated based on the membership.

[0008] Furthermore, in step S4, the aerosol monitoring is based on the clustering results of step S3 for the historical aerosol monitoring data and the real-time aerosol monitoring data, and the label with the largest number of historical data in the clustering results of each monitoring point is used as the cluster label, and the cluster label corresponding to the real-time aerosol monitoring data, that is, the monitoring assessment level is used as the monitoring result; when the monitoring result is moderate pollution or severe pollution, early warning processing is performed.

[0009] The aerosol monitoring system based on clustering processing provided by the present invention comprises a data acquisition module, a preprocessing module, an aerosol clustering module and an aerosol monitoring module; The data acquisition module collects historical aerosol monitoring data and real-time aerosol monitoring data, and sends the data to the preprocessing module; The preprocessing module performs data cleaning, data conversion and standardization on the collected aerosol monitoring data; and sends the data to the aerosol clustering module; The aerosol clustering module combines the dynamic weight adjustment of real-time and historical data, and gradually optimizes the local clustering model and global consistency of each monitoring point through membership calculation, cluster center update and global correction, so as to realize clustering processing of aerosol monitoring data; and sends the data to the aerosol monitoring module; The aerosol monitoring module realizes monitoring and evaluation of real-time aerosol monitoring data based on clustering results.

[0010] The beneficial effects achieved by the present invention using the above scheme are as follows: (1) In view of the fact that general aerosol monitoring methods are unable to capture the differences between different monitoring points and the improper balance between global and local monitoring of aerosol monitoring, which leads to poor accuracy of aerosol monitoring results, this scheme tailors a monitoring sub-model for each monitoring site to reflect its unique pollution characteristics, and performs global regularization to limit the deviation of local monitoring from global monitoring. The weights are dynamically adjusted based on regional importance, equipment quality, and acquisition time difference to ensure that the monitoring results reflect the latest pollution conditions, thereby improving the accuracy and reliability of the monitoring results.

[0011] (2) In view of the problem that general aerosol monitoring methods cannot rely solely on current aerosol monitoring data, are difficult to capture long-term changes, and have poor monitoring results due to local monitoring bias, this scheme introduces historical trend weights to reflect long-term pollution characteristics and uses a global reference position to correct and avoid distortion of results caused by local bias. This scheme can not only reflect long-term pollution trends but also adapt to short-term changes, thereby improving the accuracy of aerosol monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of the process of the aerosol monitoring method based on clustering processing provided by the present invention; Figure 2 A schematic diagram of an aerosol monitoring system based on clustering processing provided by the present invention; Figure 3 It is a schematic diagram of the process of step S3.

[0013] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0015] In the description of the present invention, it is necessary to understand that terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “top”, “bottom”, “inside” and “outside” indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred system or element must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0016] Example 1, see Figure 1 The present invention provides an aerosol monitoring method based on clustering processing, which comprises the following steps: Step S1: data collection: collecting historical aerosol monitoring data and real-time aerosol monitoring data; Step S2: preprocessing: data cleaning, data conversion and standardization of the collected aerosol monitoring data; Step S3: Aerosol clustering; combining the dynamic weight adjustment of real-time and historical data, through membership calculation, cluster center update and global correction, the local clustering model and global consistency of each monitoring point are gradually optimized to achieve clustering processing of aerosol monitoring data; Step S4: Aerosol monitoring: monitoring and evaluating the real-time aerosol monitoring data based on the clustering results.

[0017] Example 2, see Figure 1This embodiment is based on the above embodiment. In step S1, both the historical aerosol monitoring data and the real-time aerosol monitoring data include aerosol characteristic data, meteorological data and environmental data; the historical aerosol monitoring data also include monitoring assessment levels; the monitoring assessment levels are used as data labels; the data labels are only selected and used as cluster labels and do not participate in clustering operations; the monitoring assessment levels include normal, light pollution, moderate pollution and severe pollution; the aerosol characteristic data include PM2.5, PM10, total suspended particulate matter and particle size distribution; the meteorological data include temperature, humidity, wind speed, wind direction, air pressure, rainfall and solar radiation; the environmental data include industrial emission data and traffic flow.

[0018] Example 3, see Figure 1 This embodiment is based on the above embodiment. In step S2, data cleaning is to process missing values ​​and duplicate values; data conversion is to convert data into vector form; and standardization is to standardize data based on the maximum and minimum normalization method.

[0019] Example 4, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S3, aerosol clustering includes: Step S31: sub-model training; at each monitoring site, a clustering sub-model is trained based on fuzzy K-means clustering to identify the local aerosol monitoring mode. Each monitoring device trains a local clustering model based on its locally collected aerosol data; define the quasi-objective function , expressed as: ; Where N is the total number of aerosol monitoring data points collected by the monitoring station; K is the number of cluster centers; is the membership degree of the i1th aerosol monitoring data point to the kth cluster center; is the i1th aerosol monitoring data point; is the kth cluster center; is the square of the L2 norm; m is the fuzzy factor; Step S32: Optimization of objective function: When each monitoring device updates the local sub-model, it must ensure that it does not deviate from the global model to avoid excessive personalization, so the objective function is optimized, which is expressed as: ; ; ;in, is the average location of the cluster center of the ith monitoring station; is the regularization hyperparameter; is the weight of the i-th monitoring point; V is the average position of the global cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; is the historical trend weight; is the optimized objective function; is the quasi-objective function; is a distance metric; Step S33: weight calculation; there are significant differences in aerosol monitoring results between different regions. The aerosol concentration in urban areas is higher than that in suburban areas. Therefore, the weight of urban areas is higher, and the accuracy of the equipment used determines the weight of the monitoring point. Time decay is introduced, and the weight calculation formula is expressed as: ; Among them, α is the weight adjustment parameter; and Represent the urban area indicator function and the credibility indicator function respectively. If the i-th monitoring point is an urban area, then , if the accuracy of the equipment at the i-th monitoring point meets the standard, then , otherwise it is equal to 0; and is the weight adjustment factor; is the balancing factor; C1 is the total number of categories; and are the data distribution of the kth monitoring point and the gth monitoring point in the cth category respectively; γ is the smoothing parameter; is the time decay factor; It is the maximum time difference between historical aerosol monitoring data and real-time aerosol monitoring data.

[0020] By performing the above operations, in order to address the problem that general aerosol monitoring methods are unable to capture the differences between different monitoring points, and the global and local monitoring of aerosol monitoring are improperly balanced, which leads to poor accuracy of aerosol monitoring results, this scheme tailors a monitoring sub-model for each monitoring site to reflect its unique pollution characteristics, and performs global regularization to limit the deviation of local monitoring from global monitoring. The weights are dynamically adjusted based on regional importance, equipment quality, and acquisition time difference to ensure that the monitoring results reflect the latest pollution conditions, thereby improving the accuracy and reliability of the monitoring results.

[0021] Example 5, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S3, aerosol clustering further includes: Step S34: Update the cluster center; calculate the membership degree of each aerosol monitoring data point belonging to different clusters, expressed as: ; For each monitoring point, update the cluster center according to the membership weight, and introduce the historical trend weight , expressed as: ; ;in, is the jth cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; Step S35: cluster center correction; aggregate the local model to obtain the global reference position , expressed as: ; Each monitoring point fine-tunes the updated cluster center based on the global reference position, expressed as: ;in, is the learning rate; Step S36: judgment; a maximum number of iterations is set in advance, and the iteration is completed when the global objective function converges or reaches the maximum number of iterations; the global objective function is obtained by weighted summation of the objective functions of the sub-models based on the monitoring point weights; otherwise, the cluster center is re-updated and corrected, and the aerosol monitoring data points are allocated based on the membership.

[0022] By performing the above operations, the general aerosol monitoring method cannot rely solely on the current aerosol monitoring data, it is difficult to capture long-term changes, and the monitoring effect is poor due to local monitoring deviations. This scheme introduces historical trend weights to reflect the long-term pollution characteristics, and uses the global reference position to correct and avoid distortion of results caused by local deviations. It can not only reflect long-term pollution trends, but also adapt to short-term changes, thereby improving the accuracy of aerosol monitoring.

[0023] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S4, aerosol monitoring is based on the clustering results of step S3 for historical aerosol monitoring data and real-time aerosol monitoring data. The label with the largest number of historical data in the clustering results of each monitoring point is used as the cluster label, and the cluster label corresponding to the real-time aerosol monitoring data, that is, the monitoring assessment level, is used as the monitoring result; when the monitoring result is moderate pollution or severe pollution, early warning processing is performed.

[0024] Embodiment 7, see Figure 2 , this embodiment is based on the above embodiment, and the aerosol monitoring system based on clustering processing provided by the present invention includes a data acquisition module, a preprocessing module, an aerosol clustering module and an aerosol monitoring module; The data acquisition module collects historical aerosol monitoring data and real-time aerosol monitoring data, and sends the data to the preprocessing module; The preprocessing module performs data cleaning, data conversion and standardization on the collected aerosol monitoring data; and sends the data to the aerosol clustering module; The aerosol clustering module combines the dynamic weight adjustment of real-time and historical data, and gradually optimizes the local clustering model and global consistency of each monitoring point through membership calculation, cluster center update and global correction, so as to realize clustering processing of aerosol monitoring data; and sends the data to the aerosol monitoring module; The aerosol monitoring module realizes monitoring and evaluation of real-time aerosol monitoring data based on clustering results.

[0025] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0026] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

[0027] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. An aerosol monitoring method based on clustering processing, characterized in that: The method comprises the following steps: Step S1: data collection: collecting historical aerosol monitoring data and real-time aerosol monitoring data; Step S2: preprocessing; Step S3: aerosol clustering; Step S4: aerosol monitoring: monitoring and evaluating the real-time aerosol monitoring data based on the clustering results; Step S3 includes step S32: objective function optimization; when each monitoring device updates the local sub-model, it must ensure that it does not deviate from the global model to avoid excessive personalization, so the objective function is optimized, which is expressed as: ; ; ;in, is the average location of the cluster center of the ith monitoring station; is the regularization hyperparameter; is the weight of the i-th monitoring point; V is the average position of the global cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; is the historical trend weight; is the optimized objective function; is the quasi-objective function; is a distance metric.

2. The aerosol monitoring method based on clustering processing according to claim 1, characterized in that: In step S3, the aerosol clustering further includes the following steps: Step S31: sub-model training; at each monitoring site, a clustering sub-model is trained based on fuzzy K-means clustering to identify the local aerosol monitoring mode. Each monitoring device trains a local clustering model based on its locally collected aerosol data; define the quasi-objective function , expressed as: ; Where N is the total number of aerosol monitoring data points collected by the monitoring station; K is the number of cluster centers; is the membership degree of the i1th aerosol monitoring data point to the kth cluster center; is the i1th aerosol monitoring data point; is the kth cluster center; is the square of the L2 norm; m is the fuzzy factor; Step S33: weight calculation; time decay is introduced, and the weight calculation formula is expressed as: ; Among them, α is the weight adjustment parameter; and Represent the urban area indicator function and the credibility indicator function respectively. If the i-th monitoring point is an urban area, then , if the accuracy of the equipment at the i-th monitoring point meets the standard, then , otherwise it is equal to 0; and is the weight adjustment factor; is the balancing factor; C1 is the total number of categories; and are the data distribution of the kth monitoring point and the gth monitoring point in the cth category respectively; γ is the smoothing parameter; is the time decay factor; is the maximum time difference between historical aerosol monitoring data and real-time aerosol monitoring data; Step S34: Update the cluster center; calculate the membership degree of each aerosol monitoring data point belonging to different clusters, expressed as: ; For each monitoring point, update the cluster center according to the membership weight, and introduce the historical trend weight , expressed as: ; ;in, is the jth cluster center; and are the average monitoring assessment levels of the historical aerosol monitoring data of the ith and jth monitoring points, respectively; I is the total number of monitoring points; Step S35: cluster center correction; Step S36: judgment; a maximum number of iterations is set in advance, and the iteration is completed when the global objective function converges or reaches the maximum number of iterations; the global objective function is obtained by weighted summation of the objective functions of the sub-models based on the monitoring point weights; otherwise, the cluster center is re-updated and corrected, and the aerosol monitoring data points are allocated based on the membership.

3. The aerosol monitoring method based on clustering processing according to claim 2, characterized in that: In step S35, the cluster center correction is specifically: aggregating the local model to obtain the global reference position , expressed as: ; Each monitoring point fine-tunes the updated cluster center based on the global reference position, expressed as: ;in, is the learning rate.

4. The aerosol monitoring method based on clustering processing according to claim 1, characterized in that: In step S1, the historical aerosol monitoring data and the real-time aerosol monitoring data both include aerosol characteristic data, meteorological data and environmental data; the historical aerosol monitoring data also includes a monitoring assessment level; the monitoring assessment level is used as a data label; the data label is only selected and used as a cluster label and does not participate in the clustering operation.

5. The aerosol monitoring method based on clustering processing according to claim 1, characterized in that: In step S4, the aerosol monitoring is based on the clustering results of step S3 for the historical aerosol monitoring data and the real-time aerosol monitoring data. The label with the largest number of historical data in the clustering results of each monitoring point is used as the cluster label, and the cluster label corresponding to the real-time aerosol monitoring data, that is, the monitoring assessment level is used as the monitoring result; when the monitoring result is moderate pollution or severe pollution, early warning processing is performed.

6. The aerosol monitoring method based on clustering processing according to claim 1, characterized in that: In step S2, the preprocessing is to clean, convert and standardize the collected aerosol monitoring data; the data cleaning is to process missing values ​​and duplicate values; the data conversion is to convert the data into a vector form; and the standardization is to standardize the data based on the maximum and minimum normalization method.

7. An aerosol monitoring system based on clustering processing, used to implement the aerosol monitoring method based on clustering processing as described in any one of claims 1 to 6, characterized in that: It includes data acquisition module, preprocessing module, aerosol clustering module and aerosol monitoring module; The data acquisition module collects historical aerosol monitoring data and real-time aerosol monitoring data, and sends the data to the preprocessing module; The preprocessing module performs data cleaning, data conversion and standardization on the collected aerosol monitoring data; and sends the data to the aerosol clustering module; The aerosol clustering module combines the dynamic weight adjustment of real-time and historical data, and gradually optimizes the local clustering model and global consistency of each monitoring point through membership calculation, cluster center update and global correction, so as to realize clustering processing of aerosol monitoring data; And send the data to the aerosol monitoring module; The aerosol monitoring module realizes monitoring and evaluation of real-time aerosol monitoring data based on clustering results.

Images