A method and system for multi-source heterogeneous processing of environmental monitoring data

Abstract

The present disclosure provides a method and system for multi-source heterogeneous processing of environmental monitoring data, wherein the processing method includes: obtaining monitoring data related to environmental monitoring, wherein the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter; cleaning the data of each monitoring parameter respectively; standardizing the cleaned data of each monitoring parameter respectively to obtain standard data of each monitoring parameter; encapsulating the standard data of each monitoring parameter respectively to form multiple data packets; generating a data storage system adapted to the characteristic information of the monitoring data; converting the standard data in multiple data packets according to the storage format set by the data storage system, and storing them in the data storage system; establishing multiple data indexes in the data storage system. The above technical solution can improve the processing efficiency and accuracy of environmental monitoring data.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a method and system for performing multi-source heterogeneous processing on environmental monitoring data. Background Art

[0002] At present, with the rapid development of the economy and society, the types of pollutants caused by industrial sources and mobile sources (such as wastewater, exhaust gas emissions, etc.) are gradually increasing, resulting in great changes in the surrounding environment. In order to understand the changes and pollution of the environment, environmental parameters can be monitored.

[0003] However, the existing environmental parameter monitoring schemes have the following problems: first, the number of monitored objects is huge, the monitoring tools and means are diverse, and the frequency of data collection is inconsistent; second, the format of the data obtained from monitoring is not unified, which makes it impossible to effectively supervise and process the monitoring data.

[0004] In this context, how to provide a technical solution to improve the processing efficiency and accuracy of environmental monitoring data has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a method and system for performing multi-source heterogeneous processing on environmental monitoring data, which can improve the processing efficiency and accuracy of environmental monitoring data.

[0006] The present disclosure provides a method for processing multi-source heterogeneous environmental monitoring data, including:

[0007] Acquire monitoring data related to environmental monitoring, wherein the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter;

[0008] Clean the data of each monitoring parameter separately;

[0009] The data of each monitoring parameter after cleaning are respectively subjected to standardization processing to obtain standard data of each monitoring parameter; wherein the standardization processing includes: standardizing the numerical value and data format of the data of the monitoring parameter;

[0010] The standard data of each monitoring parameter is respectively encapsulated to form a plurality of data packets, wherein one data packet corresponds to one monitoring parameter;

[0011] Generating a data storage system adapted to characteristic information of the monitoring data, wherein the characteristic information of the monitoring data includes: the amount of the monitoring data, the types of monitoring parameters included in the monitoring data, and the storage requirements of the data of each monitoring parameter;

[0012] Converting the standard data in the plurality of data packets according to the storage format set by the data storage system and storing the converted data in the data storage system;

[0013] A plurality of data indexes are established in the data storage system, wherein one data index corresponds to one data packet.

[0014] Optionally, the acquiring of monitoring data related to environmental monitoring includes:

[0015] According to the monitoring parameters, determining at least one data collection point adapted to the monitoring parameters;

[0016] A data acquisition device for acquiring data of the monitoring parameters is provided at the data acquisition point;

[0017] According to the monitoring data of the monitoring parameters, configure the collection frequency and data transmission mode of the data collection device;

[0018] The initial monitoring data related to environmental monitoring is acquired through the data acquisition device, and the data format of the real-time initial monitoring data is converted to obtain the monitoring data.

[0019] Optionally, the cleaning of the data of each monitoring parameter includes:

[0020] The cleaning process of the data of each monitoring parameter comprises:

[0021] Identify missing values ​​in the data of each monitoring parameter, and process the missing values ​​based on the missing ratio and weight of the missing values; including at least one of the following: when it is determined that the missing ratio of the missing values ​​is lower than a preset ratio threshold and the weight is less than a preset weight threshold, delete the missing values; when it is determined that the missing ratio of the missing values ​​is higher than the preset ratio threshold and the weight is higher than the preset weight threshold, use interpolation processing to fill the missing values;

[0022] Identify abnormal values ​​in the data of each monitoring parameter, and process the abnormal values ​​based on the relationship between the abnormal values ​​and the numerical values ​​in the data; including at least one of the following: when it is determined that the absolute value of the difference between the abnormal value and the numerical value of the data is greater than a preset difference threshold, delete the abnormal value; when it is determined that the absolute value of the difference between the abnormal value and the numerical value of the data is less than a preset difference threshold, select multiple numerical values ​​adjacent to the abnormal value and replace the abnormal value with the average value of the multiple numerical values; retain the abnormal value and mark it.

[0023] Optionally, the standard data of each monitoring parameter is respectively encapsulated to form a plurality of data packets, including:

[0024] For any monitoring parameter, all standard data of the monitoring parameter are split according to a preset time period to form multiple time entries; and all standard data of the monitoring parameter are split according to the geographical location of each data collection point to form multiple geographical location data;

[0025] For any monitoring parameter, a multi-entry general search tree is used to encapsulate multiple time entries into entries and insert them into the encapsulation module; and a multi-entry spatial partition generalized search tree is used to encapsulate multiple geographic location data into multiple geographic sub-objects and insert them into the encapsulation module to generate a data packet;

[0026] Wherein, for any data packet, the encapsulation module is suitable for encapsulating the data of each data collection point as a main object, encapsulating the data in each time period as a secondary object, and encapsulating each pollutant index data as a sub-object.

[0027] Optionally, generating a data storage system adapted to the characteristic information of the monitoring data includes:

[0028] According to the amount of the monitoring data, based on the characteristics of a non-blocking adjustable hash table with fast deletion and memory awareness functions, a database system with high concurrency, fast deletion and dynamically adjustable data structure is selected from a plurality of different types of database systems;

[0029] According to the types of monitoring parameters included in the monitoring data, multiple hash tables are configured in the database system, wherein one hash table corresponds to one monitoring parameter;

[0030] According to the storage requirements of the data of each monitoring parameter, combined with the non-blocking characteristics of a non-blocking adjustable hash table with fast deletion and memory awareness functions, the corresponding hash table is divided into multiple small fragments and respectively set on different nodes in the database system, and the nodes are used to store the monitoring data.

[0031] Optionally, after converting the standard data in the plurality of data packets according to the storage format set by the data storage system and storing them in the data storage system respectively, the processing method further comprises:

[0032] Managing standard data corresponding to each data packet stored in the data storage system; including at least one of the following:

[0033] According to the weight value and update frequency of each monitoring parameter, formulate the frequency and type of data backup, as well as the data recovery strategy;

[0034] Performing encryption operations on each data packet stored in the data storage system;

[0035] Performing encryption operations when transmitting each data packet stored in the data storage system;

[0036] Set access permissions, generate audit logs, and refuse to perform corresponding operations and return corresponding warning information when it is detected that the visitor's request exceeds the corresponding data management permissions;

[0037] Evaluate the standard data corresponding to each data package according to the pre-defined data quality indicators and within the set time limit;

[0038] Record the data update information of the standard data corresponding to each data packet.

[0039] Optionally, establishing a plurality of data indexes in the data storage system includes:

[0040] Classify the standard data in each data packet according to the monitoring parameters and uses corresponding to the standard data in each data packet;

[0041] According to the type of standard data in each data packet, an adaptive index based on adaptive hotspot perception is used as the data index, and an index field for representing the standard data is set;

[0042] Monitor the access frequency of each index field, and according to the hotspot awareness mechanism of the adaptive index of the adaptive hotspot awareness, cache the data packets corresponding to the index fields with higher access frequencies into the memory, or increase the priority of the data packets corresponding to the index fields with higher access times.

[0043] Optionally, the processing method further includes:

[0044] According to the standard data in each data packet stored in the data storage system, a pre-trained prediction model is used to generate a corresponding environmental prediction result;

[0045] And, comparing the environmental prediction result with the actual result, and visualizing the comparison result.

[0046] Optionally, the processing method further includes: generating an environmental monitoring report based on standard data in each data packet stored in the data storage system and based on a preset statistical data generation template.

[0047] Accordingly, the embodiment of the present disclosure further provides a system for performing multi-source heterogeneous processing on environmental monitoring data, including:

[0048] an acquisition unit, configured to acquire monitoring data related to environmental monitoring, wherein the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter;

[0049] The processing unit is configured to perform cleaning processing on the data of each monitoring parameter respectively; and perform standardization processing on the cleaned data of each monitoring parameter respectively to obtain standard data of each monitoring parameter; wherein the standardization processing includes: standardizing the value and data format of the data of the monitoring parameter;

[0050] The encapsulation unit is configured to respectively encapsulate the standard data of each monitoring parameter to form a plurality of data packets; wherein one data packet corresponds to one monitoring parameter;

[0051] A generating unit configured to generate a data storage system adapted to characteristic information of the monitoring data, wherein the characteristic information of the monitoring data includes: the amount of the monitoring data, the type of monitoring parameters included in the monitoring data, and the storage requirements of the data of each monitoring parameter;

[0052] A storage unit is configured to convert the standard data in the plurality of data packets according to a storage format set by the data storage system and store the converted data in the data storage system;

[0053] The construction unit is configured to establish multiple data indexes in the data storage system, wherein one data index corresponds to one data packet.

[0054] Compared with the prior art, the technical solution of the embodiment of the present disclosure has the following advantages:

[0055] By adopting the multi-source heterogeneous processing method for environmental monitoring data provided by the embodiment of the present disclosure, on the one hand, by cleaning and standardizing the acquired data of each monitoring parameter, the format consistency of the data of each monitoring parameter can be improved; on the other hand, by respectively encapsulating the standard data of each monitoring parameter, a plurality of data packets can be formed, and the standard data in the plurality of data packets are converted according to the storage format set by the data storage system and then stored in the data storage system, thereby improving the orderliness and manageability of the data. At the same time, by establishing a data index corresponding to the data packet in the data storage system, the efficiency of data query and retrieval can be improved, thereby improving the processing efficiency and accuracy of the environmental monitoring data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments of the present disclosure or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1A flowchart of a method for multi-source heterogeneous processing of environmental monitoring data in some embodiments of the present disclosure is shown;

[0058] Figure 2 A flow chart of obtaining monitoring data related to environmental monitoring in an embodiment of the present disclosure is shown;

[0059] Figure 3 A flow chart of a method for packaging standard data in an embodiment of the present disclosure is shown;

[0060] Figure 4 A flow chart of a method for generating a data storage system in an embodiment of the present disclosure is shown;

[0061] Figure 5 A flow chart showing another method for generating a data index in an embodiment of the present disclosure is shown;

[0062] Figure 6 A schematic diagram of the structure of a system for performing multi-source heterogeneous processing on environmental monitoring data in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] As described in the background art, existing environmental parameter monitoring solutions cannot effectively supervise and process monitoring data.

[0064] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for multi-source heterogeneous processing of environmental monitoring data. On the one hand, by cleaning and standardizing the acquired data of each monitoring parameter, the format consistency of the data of each monitoring parameter can be improved; on the other hand, by respectively encapsulating the standard data of each monitoring parameter, a plurality of data packets can be formed, and the standard data in the plurality of data packets are converted according to the storage format set by the data storage system and stored in the data storage system, thereby improving the orderliness and manageability of the data. At the same time, by establishing a data index corresponding to the data packet in the data storage system, the efficiency of data query and retrieval can be improved, thereby improving the processing efficiency and accuracy of the environmental monitoring data.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the present invention is described below with reference to the accompanying drawings.

[0066] See also Figure 1 A flowchart of a method for multi-source heterogeneous processing of environmental monitoring data in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the following steps can be performed:

[0067] S11, obtaining monitoring data related to environmental monitoring.

[0068] Specifically, the monitoring data can reflect the current environmental conditions. By obtaining monitoring data related to environmental monitoring, we can understand environmental changes and pollution.

[0069] In some examples, the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter, where: monitoring parameters may refer to specific monitoring types (e.g., noise, air, water, soil, atmosphere); monitoring parameter data may refer to specific values ​​of parameters detected under the current monitoring type (e.g., if the current monitoring parameter is air, the monitoring parameter may include gas concentrations of gaseous pollutants PM2.5, PM10, SO2, NO2, CO, O3, and VOCS); and data collection points may refer to the geographical locations of the monitoring parameters, where one monitoring parameter corresponds to at least one data collection point.

[0070] In some embodiments, the data of the monitoring parameters can be evaluated in detail to understand their characteristics, collection methods and monitoring standards. For example, noise may need to consider its frequency and intensity, while air needs to focus on PM2.5, PM10, SO2, NOx and other indicators.

[0071] S12, cleaning the data of each monitoring parameter respectively.

[0072] Specifically, the acquired monitoring data may be missing or abnormal, which may cause the subsequent process to be unable to process the data, or may result in erroneous predictive analysis results based on the existing monitoring data. Therefore, cleaning processing can be performed to provide a high-quality data foundation for subsequent data analysis and application.

[0073] S13, respectively standardizing the data of each monitoring parameter after cleaning to obtain standard data of each monitoring parameter.

[0074] Specifically, through standardization processing, the consistency between data can be improved, making it easier to compare data of the same monitoring parameters at different times, states and spatial locations, thereby achieving data consistency and comparability.

[0075] S14, respectively encapsulating the standard data of each monitoring parameter to form multiple data packets.

[0076] Specifically, by encapsulating the standard data of each monitoring parameter separately, a data packet corresponding to the monitoring parameter can be formed (that is, one data packet corresponds to one monitoring parameter), so that the standard data of the monitoring parameter can be obtained by accessing and querying the data packet, which is convenient for upper-level application calls.

[0077] S15, generating a data storage system adapted to the characteristic information of the monitoring data.

[0078] Specifically, the data storage system is used to store monitoring data. By generating a corresponding data storage system based on the characteristic information of the monitoring data, the data storage system can be adapted to the storage and analysis requirements of all monitoring data.

[0079] S16, converting the standard data in the plurality of data packets according to the storage format set by the data storage system, and storing the converted data in the data storage system.

[0080] Specifically, by converting the standard data in multiple data packets according to the storage format set by the data storage system and then storing them, the format consistency of the standard data in the data storage system can be improved, and it is easier to query and manage. It can also avoid downtime caused by abnormal standard data formats in data packets, thereby improving data reliability and stability.

[0081] S17, establishing multiple data indexes in the data storage system.

[0082] Among them, one data index corresponds to one data packet.

[0083] Specifically, by establishing data indexes in a data storage system, the efficiency of data query and retrieval can be improved.

[0084] In some embodiments, the data index may be designed based on multiple dimensions such as time, geographic location, data type, etc. of the data packet corresponding to the data index.

[0085] In short, the embodiments of the present disclosure can improve the processing efficiency and accuracy of environmental monitoring data by cleaning, standardizing, packaging, storing in a set storage format, and establishing data indexes for monitoring data related to environmental monitoring.

[0086] In some embodiments of the present disclosure, when acquiring monitoring data related to environmental monitoring, data acquisition equipment that is compatible with the monitoring parameters can be configured based on the monitoring parameters to be acquired to acquire data of each monitoring parameter.

[0087] For some examples, see Figure 2 The flowchart of obtaining monitoring data related to environmental monitoring in the embodiment of the present disclosure shown is mainly implemented in the following steps:

[0088] S21, determining, according to the monitoring parameter, at least one data collection point that matches the monitoring parameter.

[0089] Specifically, the types of monitoring parameters are different and the data collection points are selected in different ways.

[0090] In some examples, the monitoring parameters are different according to the requirements of environmental monitoring. Accordingly, the specific data source types that need to be monitored are also different. For example, the data source may include at least one of noise pollution sources, air pollution sources, water pollution sources, soil pollution sources, and atmospheric pollution sources.

[0091] In some embodiments, after determining the type of monitoring parameter, at least one data collection point may be selected.

[0092] For example, if the monitoring parameter to be collected is industrial source data, data collection points can be set up in various industrial areas, factories, production lines, etc. The specific locations can be determined based on the amount of pollutant emissions and the degree of environmental impact. It is necessary to ensure that the collection points cover all important emission sources.

[0093] For example, if the monitoring parameters to be collected are mobile source data, data collection points can be set up at roads with heavy traffic, bus stations, airports, etc. to monitor mobile source pollutants such as automobile exhaust and aircraft emissions. The collection point layout should be scientific and reasonable to fully reflect the pollution situation in the area.

[0094] In short, the selection of data collection points is strongly related to the monitoring parameters to be collected.

[0095] S22, setting a data acquisition device for acquiring data of the monitoring parameters at the data acquisition point.

[0096] Specifically, when a data collection point is determined, a data collection device may be set at the spatial location of the data collection point to obtain data of the monitoring parameter.

[0097] In some embodiments, according to the characteristics of different data sources, a suitable sensor type can be selected as a data acquisition device.

[0098] For example, noise monitoring can use a noise meter, air quality monitoring can use gas sensors and laser scattering PM2.5 sensors, and water quality monitoring can use a multi-parameter water quality monitor.

[0099] In some embodiments, when selecting a data acquisition device, it is also necessary to evaluate the performance of the data acquisition device, including sensitivity, accuracy, stability, and service life, to ensure that it meets the monitoring requirements.

[0100] In some embodiments, there may be multiple data collection points, so data collection devices may be set at different data collection points to obtain monitoring data of the same monitoring parameter at different spatial locations at the same time.

[0101] S23, configuring the data acquisition frequency and data transmission mode of the data acquisition device according to the monitoring data of the monitoring parameter.

[0102] Specifically, the format, collection time, and collection difficulty of monitoring data for different monitoring parameters are different. Therefore, even when using data collection equipment to collect monitoring parameters with similar collection characteristics, the collection parameters (collection frequency and data transmission method) of the data collection equipment are different, so as to adapt to the collection process of different monitoring parameters.

[0103] In some embodiments, the collection frequency of the data collection device is reasonably set according to the change characteristics of the monitoring data of the monitoring parameters and the monitoring requirements.

[0104] For example, for air quality monitoring, data can be collected once every minute; and for water quality monitoring, data can be collected once every hour.

[0105] In some embodiments, an appropriate data transmission method may be selected, such as wired transmission, wireless transmission, satellite transmission, etc., to ensure that data can be transmitted to the data center for processing in a timely and stable manner.

[0106] S24, acquiring initial monitoring data related to environmental monitoring through the data acquisition device, and converting the data format of the real-time initial monitoring data to obtain the monitoring data.

[0107] Specifically, after configuring the parameters of the data acquisition device (including the setting location, acquisition frequency, and data transmission method), the data acquisition operation can be performed through the data acquisition device to obtain initial monitoring data.

[0108] Furthermore, considering the influence of other factors, after the initial monitoring data is obtained, the data format of the initial monitoring data may be converted.

[0109] In some embodiments, the data format conversion process of the initial monitoring data may specifically include:

[0110] Initial data processing includes: data denoising and filtering processing, where noise processing can refer to noise processing of the collected initial monitoring data to remove irrelevant information such as equipment errors and environmental interference to ensure the authenticity and accuracy of the monitoring data. Data filtering can refer to the use of data filtering algorithms, such as low-pass filters, high-pass filters, etc., to perform preliminary filtering on the initial monitoring data to remove invalid monitoring data and abnormal monitoring data.

[0111] Initial formatting includes: unifying data formats and data labeling. Unifying data formats means converting the data after initial processing into a unified format to facilitate subsequent data cleaning and organization. For example, all data can be unified into JSON format or CSV format. Data labeling can refer to labeling different types of data and adding necessary metadata, such as timestamps, geographic locations, data types, etc., to facilitate subsequent data management and analysis.

[0112] Through the above data collection and processing steps, it can be ensured that the data collection and preliminary processing of all environmental monitoring data sources can be carried out smoothly, and the consistency and integrity of the monitoring data can be improved, laying a solid foundation for subsequent data cleaning, organization and analysis.

[0113] In some embodiments of the present disclosure, when the monitoring data of each monitoring parameter is obtained, a cleaning process can be performed to identify missing values ​​and abnormal values ​​in the data of each monitoring parameter. Missing values ​​refer to missing data, missing parts of fields, etc.; abnormal values ​​refer to values ​​of other data points in the data of the monitoring parameter that are significantly deviated, and abnormal values ​​usually far exceed the range of other data points, or do not match the distribution pattern of other data points.

[0114] In some embodiments, the cleaning process may specifically include: identifying missing values ​​in the data of each monitoring parameter, and processing the missing values ​​based on the missing ratio and weight of the missing values.

[0115] Specifically, missing values ​​can be identified by performing a completeness check on the data.

[0116] In some embodiments of the present disclosure, when it is determined that the missing ratio of the missing values ​​is lower than a preset ratio threshold and the weight is less than a preset weight threshold, the missing values ​​are deleted; when it is determined that the missing ratio of the missing values ​​is higher than the preset ratio threshold and the weight is higher than the preset weight threshold, interpolation processing is used to fill the missing values.

[0117] Specifically, when the missing ratio of missing values ​​is lower than the preset ratio threshold and the weight is less than the preset weight threshold, it means that these missing values ​​are not important for the analysis and management of monitoring data, so these missing values ​​can be deleted.

[0118] When the missing ratio of missing values ​​is higher than the preset ratio threshold and the weight is higher than the preset weight threshold, it means that this part of missing values ​​is very important for the analysis and management of monitoring data. Therefore, interpolation processing (such as linear interpolation, spline interpolation, etc.) can be used to fill the missing values.

[0119] In some other embodiments, randomly missing data may be filled using statistics such as mean, median, mode, etc.

[0120] Through the above operations, the integrity of the monitoring data of each monitoring parameter can be improved.

[0121] In some embodiments, the cleaning process may specifically include: identifying abnormal values ​​in the monitoring data of each monitoring parameter, and processing the abnormal values ​​based on the relationship between the abnormal values ​​and the numerical values ​​in the monitoring data.

[0122] Specifically, statistical analysis methods (such as box plot analysis, Z-score method, etc.) can be used to identify outliers in the data. For specific data sources, outlier determination can also be performed in combination with domain knowledge.

[0123] In some embodiments of the present disclosure, outliers in the data of each monitoring parameter are identified, and based on the relationship between the outliers and the numerical values ​​in the data, the outliers are processed; including at least one of the following: when it is determined that the absolute value of the difference between the outlier and the numerical value of the data is greater than a preset difference threshold, the outlier is deleted; when it is determined that the absolute value of the difference between the outlier and the numerical value of the data is less than the preset difference threshold, multiple numerical values ​​adjacent to the outlier are selected, and the outlier is replaced by the average of the multiple numerical values; the outlier is retained and marked.

[0124] Specifically, if the absolute value of the difference between the outlier and the data value is greater than the preset difference threshold, it means that this part of the outlier is far away from other data points, so this part of the outlier can be deleted.

[0125] If the absolute value of the difference between the outlier and the data value is less than the preset difference threshold, it means that this part of the outliers may be data points of the monitoring data (i.e., suspicious outliers). At this time, multiple values ​​adjacent to the outliers can be selected and the outliers can be replaced with the average of the multiple values.

[0126] Alternatively, outliers can be retained and marked for special treatment in subsequent analyses.

[0127] In some embodiments of the present disclosure, after data cleaning processing is performed, standardization processing may also be performed.

[0128] In some embodiments, the monitoring data of different monitoring parameters can be numerically standardized to facilitate unified analysis. Among them, the standardization processing method can include Z-score standardization, Min-Max standardization, etc. In addition, the data format of different monitoring parameters can be unified, such as unified date format, unit, etc., to ensure the consistency and comparability of the data.

[0129] By performing cleaning and standardization processing, the integrity and format consistency of the monitoring parameter data can be improved and the difficulty of subsequent processing can be reduced.

[0130] By adopting the above example, the standard data of each monitoring parameter can be obtained, and then the standard data of each monitoring parameter can be encapsulated respectively.

[0131] In some embodiments of the present disclosure, different monitoring parameters can be classified according to the data corresponding to the monitoring parameters (such as air quality monitoring, water quality monitoring, noise monitoring, etc.) for separate packaging.

[0132] As an example, see Figure 3 A flowchart of a method for packaging standard data in an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the following steps can be performed:

[0133] S31, for any monitoring parameter, split all standard data of the monitoring parameter according to a preset time period to form multiple time entries.

[0134] Specifically, for any monitoring parameter, the monitoring parameters within a certain period of time can be obtained. When these parameters are obtained, the standard data can be split according to the preset time period to form multiple time entries. Each entry corresponds to the data of a time period, including all pollutant indicators.

[0135] In some embodiments, the preset time period can be configured as hours, days, months, etc.

[0136] In some embodiments, the specific type of the monitoring parameter may be determined based on standard data of the monitoring parameter.

[0137] For example, if the monitored standard data include PM2.5, PM10, SO2, NOx, CO, etc., it means that the monitored parameter is air quality; for another example, if the monitored standard data include pH value, dissolved oxygen, turbidity, temperature, heavy metal content, etc., it means that the monitored parameter is water quality; for another example, if the monitored standard data include noise level, frequency, etc., it means that the monitored parameter is noise; for another example, if the monitored standard data include soil pH value, humidity, organic matter content, etc., it means that the monitored parameter is soil; furthermore, if the monitored standard data include temperature, humidity, wind speed, air pressure, etc., it means that the monitored parameter is the atmosphere.

[0138] S32, for any monitoring parameter, split all standard data of the monitoring parameter according to the geographical location of each data collection point to form a plurality of geographical location data.

[0139] Specifically, for any monitoring parameter, the geographical location of the data collection point may be different, so the standard data of the same monitoring parameter may be obtained by data collection devices placed at different geographical locations. Therefore, the data collection operation can be split according to the geographical location of each data collection point to form multiple geographical location data, where one geographical location data includes all time periods and pollutant indicators.

[0140] In some embodiments, the geographic location may be configured as a city, region, or the like.

[0141] S33, for any monitoring parameter, a multi-entry general search tree is used to encapsulate multiple time entries into entries and insert them into the encapsulation module; and a multi-entry spatial partitioned generalized search tree is used to encapsulate multiple geographic location data into multiple geographic sub-objects and insert them into the encapsulation module to generate a data packet.

[0142] In some embodiments, when performing the insert operation, a plurality of time entries may be encapsulated as entries according to time periods through a multi-entry generalized search tree (MGiST), and inserted into the encapsulation module.

[0143] In some embodiments, when performing an insert operation, the data may be split into multiple geographic sub-objects according to geographic location through a Multi-entry Space Partitioned Generalized Search Tree (MSP-GiST), and inserted into the encapsulation module, thereby generating a data packet.

[0144] Among them, during the data insertion process, MGiST and MSP-GiST can decompose complex objects into multiple sub-objects, so that any monitoring parameter data can be divided according to time periods and regional characteristics, and then the environmental monitoring data can be efficiently encapsulated and managed.

[0145] In some embodiments of the present disclosure, for any data packet, the encapsulation module is suitable for encapsulating data of each data collection point as a main object, encapsulating data within each time period as a secondary object, and encapsulating each pollutant index data as a sub-object.

[0146] Specifically, after setting the splitting strategy, you can set the encapsulation data structure of the database.

[0147] For example, for any monitoring parameter, the data of each data collection point can be encapsulated as a main object, the data in each time period can be encapsulated as a secondary object, and the data of each pollutant indicator can be encapsulated as a final sub-object, thereby realizing hierarchical encapsulation of data.

[0148] In some optional examples, after the encapsulation operation is completed and multiple data packets are formed, the encapsulation module can also be verified and optimized.

[0149] Specifically, verify whether the MGeST and MSP-GiST modules correctly split and encapsulate data to ensure the integrity and consistency of each encapsulated object; and monitor the performance of the encapsulation and insertion process to identify bottlenecks in order to improve the efficiency of encapsulation and insertion by optimizing the splitting strategy and pluggable parameters (i.e., preset time period and geographical location).

[0150] As an example, the packaging process of air quality monitoring data is used for explanation.

[0151] First, the monitoring parameter is identified as air quality, and the information of the air quality monitoring data is determined. For example, the information of the air quality monitoring data includes various pollutant indicators and data collection point information.

[0152] Secondly, define the packaging structure. Main object: data of each data collection point; secondary object: data of each time period; sub-object: pollutant index data.

[0153] Next, the encapsulation function of the encapsulation module is designed. The MGeST module is used to split the daily data by hour and encapsulate it into multiple entries; and the MSP-GiST module is used to split the data of each data collection point by geographical location and encapsulate it into multiple sub-objects.

[0154] Secondly, encapsulate data processing. Receive daily raw data, split it into items by hour using the encapsulation module, and encapsulate the data of each data collection point into sub-objects according to geographical location.

[0155] Finally, verification and optimization. For example, verify the integrity and consistency of packaged data, ensure that data is correctly split and stored, and optimize the packaged module to improve the efficiency of data insertion and retrieval.

[0156] Therefore, through the above steps, using MGiST and MSP-GiST technologies, environmental monitoring data can be efficiently encapsulated and managed, flexible splitting and index optimization of complex objects can be achieved, and the overall performance and data management capabilities of the system can be improved.

[0157] After a plurality of data packets are formed, the data packets may be stored via a data storage system for subsequent analysis operations.

[0158] In some embodiments of the present disclosure, the characteristic information of the monitoring data includes: the amount of the monitoring data, the type of monitoring parameters included in the monitoring data, and the storage requirements of the data of each monitoring parameter. In this case, see Figure 4, the method for generating a data storage system may include:

[0159] S41, according to the amount of the monitoring data, based on the characteristics of the non-blocking adjustable hash table with fast deletion and memory awareness functions, select a database system with high concurrency, fast deletion and dynamic adjustment of data structure from multiple different types of database systems.

[0160] Specifically, in order to improve the accuracy of monitoring, considering that the amount of monitoring data is large and complicated, when selecting a database system, you can choose from multiple different types of database systems that meet the requirements of high concurrency, fast deletion and dynamic adjustment of data structure, so as to meet the monitoring data storage and management needs.

[0161] In some embodiments, the plurality of different types of database systems may include at least one of the following:

[0162] Relational database system: suitable for structured data storage, such as MySQL, PostgreSQL, etc., with powerful query and transaction processing capabilities.

[0163] NoSQL database system: suitable for unstructured or semi-structured data storage, such as MongoDB, Cassandra, etc., with high scalability and flexible data model.

[0164] Time series database system: suitable for the storage and analysis of time series data, such as InfluxDB, TimescaleDB, etc., which specializes in efficient storage and query of time series data.

[0165] Specifically, considering the amount of monitoring data, you can choose a database system that supports high concurrency, fast deletion, and dynamic resizing, such as a NoSQL database (such as MongoDB) or an in-memory database (such as Redis), based on the characteristics of a non-blocking adjustable hash table (DLHT) with fast deletion and memory awareness, to make full use of the advantages of DLHT.

[0166] S42: configuring a plurality of hash tables in the database system according to the types of monitoring parameters included in the monitoring data, wherein one hash table corresponds to one monitoring parameter.

[0167] Specifically, after determining the type of the database system, the data storage structure of the database system can be set.

[0168] For example, for a relational database system, you can design a table structure that defines the relationships between tables, primary keys, foreign keys, etc. For another example, for a NoSQL database system, you can design a document structure that defines the nesting level and index fields of the data. For another example, for a time series database system, you can design a storage method for time series data to define timestamp fields and tags.

[0169] Specifically for the present invention, after obtaining the data storage system based on the DLHT characteristics, the DLHT can be used as the underlying storage structure to design a data model to support efficient data storage and retrieval, ensuring that the data storage system can adapt to the needs of dynamic resizing.

[0170] More specifically, different hash tables may be configured in the database system, wherein one hash table corresponds to one type of monitoring parameter to store monitoring data of different types of monitoring parameters.

[0171] For example, air quality data is stored in the "air_quality" hash table, and water quality data is stored in the "water_quality" hash table.

[0172] In some embodiments, the key-value pair structure of each hash table is as follows:

[0173] Key: unique identifier of the data record (such as timestamp + sensor ID);

[0174] Value: Detailed information of monitoring data (such as JSON format, including monitoring indicators and values).

[0175] S43, according to the storage requirements of the data of each monitoring parameter, combined with the non-blocking characteristics of the non-blocking adjustable hash table with fast deletion and memory awareness functions, the corresponding hash table is divided into multiple small fragments, and they are respectively set on different nodes in the database system, and the nodes are used to store the monitoring data.

[0176] Specifically, when configuring hash tables corresponding to monitoring parameters in the database system, multiple hash tables can be divided into multiple small fragments to achieve partitioned and / or sharded storage of monitoring data of each monitoring parameter. In this way, the monitoring data of each monitoring parameter can be stored on different nodes respectively, and combined with the non-blocking characteristics of DLHT, the scalability and query performance of data storage of monitoring data can be improved.

[0177] To facilitate understanding, the generation process of the data storage system is described in detail through an example.

[0178] DLHT is an advanced hash table structure with fast deletion and memory awareness, as well as non-blocking and dynamic resizing. This hash table is very suitable for data storage and management needs in a high-concurrency environment, and can effectively improve the efficiency of data operations and system stability.

[0179] Based on this, the generation process of the data storage system includes: First, selecting a suitable database storage system, including: Based on the characteristics of DLHT, selecting a database system that supports high concurrency, fast deletion, and dynamic resizing, such as a NoSQL database (such as MongoDB) or an in-memory database (such as Redis).

[0180] Second, design the data storage structure, including: defining the data model, that is, using DLHT as the underlying storage structure, designing the data model to support efficient data storage and retrieval, and ensuring that the data model can adapt to the needs of dynamic resizing; partitioning and sharding strategy: combining the non-blocking characteristics of DLHT, designing a reasonable data partitioning and sharding strategy to improve the scalability and query performance of data storage.

[0181] More specifically, based on the characteristics of DLHT, the Redis database system was selected, among which: Redis supports high-concurrency read and write operations and is suitable for using the DLHT feature; Redis has the characteristics of an in-memory database and can quickly respond to data insertion and deletion requests; Redis provides a variety of data structures (such as hash tables, lists, sets, etc.) and supports dynamic expansion and partitioning.

[0182] Then, the hash table data structure of the Redis database system is used to implement DLHT, making full use of Redis's fast deletion and memory awareness functions, and configuring Redis's cluster mode to achieve partitioned storage and load balancing of data.

[0183] Finally, a DLHT-based data storage structure and partitioning strategy are designed to ensure efficient data storage and management.

[0184] Therefore, the sharding mechanism of the Redis database system is used to divide the hash table with large amounts of data into multiple small fragments, which are stored in multiple nodes to improve data access efficiency.

[0185] In some embodiments of the present disclosure, after converting the standard data in a plurality of data packets according to the storage format set by the data storage system and storing them in the data storage system respectively, the processing method further includes:

[0186] Manages the standard data corresponding to each data packet stored in the data storage system.

[0187] In some embodiments, the method may specifically include at least one of the following:

[0188] According to the weight value and update frequency of the data of each monitoring parameter, formulate the data backup frequency and type, as well as the data recovery strategy.

[0189] Specifically, the following steps may be included: formulating a data backup strategy, including backup frequency and backup type, where backup frequency may refer to formulating a reasonable backup plan (such as daily, weekly or monthly backup) based on the importance and update frequency of the data to ensure data security; backup types may include full backup (full backup of all data, suitable for scenarios with small data volumes or critical data), incremental backup (only backing up data that has changed since the last backup, saving storage space and backup time), and differential backup (backing up data that has changed since the last full backup, between full backup and incremental backup).

[0190] Perform data backup and verification regularly, including: regular backup (i.e., regularly perform data backup according to the backup strategy to ensure data security) and backup verification (i.e., regularly verify the integrity and availability of backup files to ensure that data can be successfully restored when needed) to ensure that data can be restored in the event of an accident.

[0191] Develop a data recovery plan, including: recovery process and recovery drills. The recovery process includes developing a detailed data recovery process, such as recovery steps and required tools; recovery drills include regular data recovery drills to ensure rapid recovery when actual data is lost or damaged.

[0192] As an optional example, you can use the fast deletion function of DLHT to design efficient incremental backup and differential backup strategies to reduce backup time and storage space. For example, configure the RDB (RedisDatabase) snapshot and AOF (Append Only File) persistence mechanism of the Redis database system to regularly generate data snapshots and incremental log files.

[0193] Among them, the backup frequency of the RDB snapshot of the Redis database system is configured to be once a day, and the synchronization frequency of the AOF log is once a second (it can be understood that the synchronization frequency of the AOF log for different monitoring parameters may be different).

[0194] As an optional example, the memory awareness function of DLHT is used to periodically verify the integrity and availability of backup data. For example, RDB snapshots and AOF log files are periodically copied to external storage devices (such as cloud storage or network storage) for off-site backup.

[0195] As an optional example, the non-blocking feature of DLHT can be used to develop a detailed data recovery process. For example, when data is damaged or lost, the latest RDB snapshot is restored first, and then the incremental operations in the AOF log file are applied to ensure that the data is restored to the latest state.

[0196] Perform encryption operations on each data packet stored in the data storage system, that is, encrypt the stored data, for example, using a strong encryption algorithm (such as AES-256) to prevent data leakage.

[0197] As an optional example, when performing storage encryption, the memory awareness function of DLHT may be combined.

[0198] When transmitting each data packet stored in the data storage system, encryption operation is performed, that is, encryption protocols such as SSL / TLS are used during the data transmission process to ensure the security of the data during the transmission process.

[0199] Set access permissions, generate audit logs, and refuse to perform corresponding operations and return corresponding warning information when it is detected that the visitor's request exceeds the corresponding data management permissions.

[0200] Specifically, it includes: assigning different access rights to different users to ensure that only authorized users can access and operate data (i.e., permission management) and recording all data access and operation behaviors, regularly reviewing logs, and discovering and handling potential security issues (i.e., audit logs). In some embodiments, for permission management operations, different access rights can be assigned to different users by configuring Redis's ACL (access control list) to ensure that only authorized users can access and operate specific data.

[0201] Also, use Redis's command auditing function to record all users' operation logs and review the logs regularly to identify and address potential security issues.

[0202] Furthermore, by utilizing DLHT’s quick deletion function, unnecessary or expired data can be deleted in a timely manner to free up memory space and prevent data leakage.

[0203] Evaluate the standard data corresponding to each data packet according to the pre-defined data quality indicators and within the set time limit.

[0204] Specifically, it includes: defining data quality indicators (such as completeness, accuracy, consistency, etc.), regularly monitoring and evaluating data quality, and promptly processing and repairing data quality issues when they are discovered to ensure high data quality.

[0205] In some embodiments, the memory awareness function of DLHT can be used to monitor and evaluate data quality.

[0206] Record the data update information of the standard data corresponding to each data packet.

[0207] Specifically, it includes version management and change review. Version management refers to recording detailed information of each data update for easy traceability and recovery; change review refers to conducting an audit before data is changed to ensure the rationality and correctness of the change.

[0208] Through the above steps, a sound data storage and management system is established to ensure the safe, reliable and efficient storage and management of environmental monitoring data, providing a solid guarantee for subsequent data analysis and application. In some embodiments of the present disclosure, after storing multiple data packets in the data storage system, in order to improve the efficiency of data query and retrieval, a data index can also be established in the data storage system.

[0209] See also Figure 5 A flowchart of another method for generating a data index in an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the following steps can be performed:

[0210] S51, classifying the standard data in each data packet according to the monitoring parameters and uses corresponding to the standard data in each data packet.

[0211] Specifically, the monitoring parameters and uses corresponding to the standard data in each data packet are different, and therefore the storage structures used when storing the data packets in the data storage system are different. Therefore, the standard data in each data packet needs to be classified in order to confirm the corresponding storage structure.

[0212] In some embodiments, according to the type, source and purpose of environmental monitoring data, it can be classified into: air quality data, water quality data, noise data, soil data and atmosphere data, etc.

[0213] Air quality data is stored by geographical area (such as city, region) and time period (such as hour, day, month). Each classification node stores data on specific indicators such as PM2.5, PM10, SO2, NOx, and CO.

[0214] Water quality data is stored by water type (such as rivers, lakes) and monitoring point. Each classification node stores data such as pH value, dissolved oxygen, turbidity, temperature, and heavy metal content.

[0215] Noise data is stored by monitoring area (such as industrial area, residential area) and time period. Noise level, frequency and other data are stored under each classification node.

[0216] Soil data is stored by monitoring area and depth level. Soil pH value, humidity, organic matter content and other data are stored under each classification node.

[0217] Atmospheric data: stored by monitoring altitude and time period. Temperature, humidity, wind speed, air pressure and other data are stored under each classification node.

[0218] S52, according to the type of standard data in each data packet, using an adaptive hotspot-aware adaptive index as the data index, and setting an index field for representing the standard data.

[0219] In some embodiments, an adaptive hotspot-aware adaptive index (such as a B+ tree, a hash index, or a time series-based index) can be selected as the index type for the standard data in each data packet, and based on the type of the standard data in each data packet, an appropriate index field can be selected to characterize the standard data.

[0220] For example, index fields for data such as air quality might include geographic region, timestamp, and pollutant index.

[0221] Among them, Adaptive Hotspot-Aware Index for Oscillating Write and Read-Heavy Workloads is an advanced data indexing technology that is particularly suitable for data management systems that handle frequent write and read operations. This technology can dynamically sense the hot spots of data access and adaptively adjust the index structure according to the access pattern to improve the efficiency of data retrieval and storage.

[0222] S53, monitors the access frequency of each index field, and according to the hotspot perception mechanism of the adaptive index of the adaptive hotspot perception, caches the data packets corresponding to the index fields with higher access frequencies into the memory, or increases the priority of the data packets corresponding to the index fields with higher access times.

[0223] Specifically, the hotspot perception mechanism of adaptive hotspot perception can monitor the access pattern of data in real time and identify frequently accessed data areas (i.e. hotspot data). At this time, the index fields corresponding to these areas can be identified as fields with higher access frequency, so that the hotspot data can be cached in memory.

[0224] Alternatively, the priority of the index structure can be dynamically adjusted based on the access pattern monitored in real time. For example, when the frequency of data access in a certain area increases, the data index in that area can be promoted to a higher priority, or redundant indexes can be added to improve retrieval efficiency.

[0225] In addition, you can also regularly analyze data access logs and optimize indexing strategies, such as merging infrequently accessed index nodes to reduce indexing overhead.

[0226] Therefore, by dynamically sensing the hot spots of data access and adaptively adjusting the index structure according to the access pattern, the efficiency of data retrieval and storage can be improved.

[0227] In some embodiments of the present disclosure, after the monitoring data related to environmental monitoring is converted and stored according to the storage format set by the storage system, a pre-trained prediction model can be used to generate corresponding environmental prediction results based on the standard data in each data packet stored in the data storage system.

[0228] Among them, the pre-trained prediction model can be obtained in the following way:

[0229] Read the standard data in each data packet stored in the data storage system, process the read standard data, and then divide it into training set, verification set and test set (for example, 70% of the standard data is used as the training set, 15% of the standard data is used as the verification set, and 15% of the standard data is used as the test set).

[0230] Select the initial model.

[0231] Use the training set data to train the initial model and adjust the initial model parameters to improve prediction accuracy. During the training process, it is necessary to set appropriate training rounds (epochs) and batch size (batch size).

[0232] Use the validation set data to evaluate the model performance and adjust the parameters of the initial model to optimize the prediction effect until the prediction effect of the initial model meets the requirements, and then generate a pre-trained prediction model.

[0233] To facilitate understanding, a brief explanation is given with examples.

[0234] Example 1: Building a pollution trend prediction model

[0235] The data related to air quality indicators (such as PM2.5, PM10, SO2, NO2, CO, O3, etc.) are read from the data storage system, and the meteorological data (such as temperature, humidity, wind speed, precipitation) and socio-economic data (such as population density, industrial activity data) in the corresponding time period are obtained.

[0236] We use methods such as Min-Max standardization to handle missing values ​​and outliers in the data to ensure data integrity, and divide the data into training, validation, and test sets in a ratio of 70%, 15%, and 15%.

[0237] Select the long short-term memory network (LSTM) as the initial model and build the LSTM model, including the input layer, hidden layer and output layer. The hidden layer can contain multiple LSTM units, and set appropriate parameters such as the number of hidden units, dropout rate, etc.

[0238] Use the training set data to train LSTM, set appropriate training rounds (epochs) and batch size (batch size), and adjust LSTM parameters to improve prediction accuracy.

[0239] Afterwards, the performance of LSTM is evaluated using the validation set data, and the structure and parameters of LSTM are adjusted to optimize the prediction effect.

[0240] Finally, the trained LSTM model is used to predict the test set data to generate air quality prediction results for future time periods.

[0241] Example 2: Building a water quality anomaly detection model

[0242] Read water quality-related data (such as water temperature, pH value, dissolved oxygen, turbidity, conductivity, permanganate index, ammonia nitrogen, total phosphorus, total nitrogen, heavy metal PM2.5, etc.) stored in the data storage system, and clean the data to ensure data quality.

[0243] Select the Isolation Forest model as the initial model and set the parameters of the Isolation Forest model, such as the number of trees and the proportion of outliers. Among them, the Isolation Forest algorithm in the Isolation Forest model is particularly suitable for anomaly detection of high-dimensional data and can effectively identify outliers.

[0244] The isolation forest model is trained with water quality related data so that the model can learn the distribution characteristics of normal data.

[0245] Afterwards, the trained isolation forest model is used to detect the test set data to identify potential abnormal data points (i.e., possible water quality anomalies).

[0246] Through the above steps, the introduction of machine learning and artificial intelligence algorithms has effectively improved the analysis capabilities of environmental monitoring data, realized pollution trend prediction and automated anomaly detection, and provided strong technical support for environmental protection work.

[0247] In some embodiments of the present disclosure, after the corresponding environment prediction result is generated, the environment prediction result can be compared with the actual result, and the comparison result can be visualized.

[0248] For example, for Example 1, the difference between the predicted effect and the actual result can be displayed in a graph to evaluate the accuracy and reliability of the model; for another example, for Example 2, the detection results can be visualized through a graph, and the identified abnormal data points can be marked to facilitate intuitive understanding and further analysis.

[0249] In some embodiments of the present disclosure, the visualization step may include:

[0250] A1) Design data visualization tools, including:

[0251] A11) Determine visualization requirements

[0252] Indicator selection: According to the needs of environmental monitoring, select the key indicators to be displayed, such as PM2.5, PM10, SO2, NOx, CO, temperature, humidity, etc.

[0253] User demand analysis: Understand the specific needs of users (such as environmental protection departments, corporate managers, and the public) and determine the focus and form of data display.

[0254] Purpose of presentation: Be clear about the purpose of the visualization, whether it is for trend analysis, anomaly detection, policy support, or public information disclosure.

[0255] A12) Select a visualization tool

[0256] Tool evaluation: Evaluate different data visualization tools such as Tableau, Power BI, D3.js, Plotly, etc. and choose the tool that best suits your needs.

[0257] Tool functions: Make sure that the selected tool has functions such as data interaction, dynamic update, and support for multiple chart types (such as line charts, bar charts, heat maps, map visualization).

[0258] A13) Design visual interface

[0259] Interface layout: Design a user-friendly interface layout to ensure that important information is clear at a glance. Including dashboard design, multi-level information display, etc.

[0260] Color selection: Choose appropriate color combinations to ensure the readability and aesthetics of the chart. Use warning colors (such as red and orange) to mark pollution data.

[0261] Interaction design: Add data interaction functions, such as hovering the mouse to display detailed information, clicking to expand detailed data, scrolling the timeline to view historical data, etc.

[0262] A14) Data visualization

[0263] Data connection: Connect visualization tools to data sources to ensure real-time update and accuracy of data.

[0264] Chart production: Based on the designed interface layout, produce various charts to display the changing trends and spatial distribution of environmental monitoring data such as air quality, water quality, and noise.

[0265] Testing and optimization: Test the visual interface to ensure normal functionality, optimize user experience, and correct possible problems.

[0266] In some optional embodiments, the processing method may further include: generating an environmental monitoring report based on standard data in each data packet stored in the data storage system and based on a preset statistical data generation template.

[0267] In this way, environmental monitoring reports with different forms of expression can be provided according to user needs, which is convenient for users to comprehensively evaluate and monitor the environmental monitoring reports from different dimensions, thereby improving the accuracy, objectivity and comprehensiveness of the evaluation and monitoring.

[0268] In some embodiments of the present disclosure, the step of generating an environmental monitoring report includes:

[0269] B1) Determine the content of the report

[0270] Report framework: Develop the basic framework and structure of the report, including summary, monitoring data overview, trend analysis, description of abnormal situations, conclusions and recommendations, etc.

[0271] Data aggregation: Collect and organize environmental monitoring data to ensure data integrity and accuracy.

[0272] Analysis results: Based on the data analysis results, write each section, especially the data trend analysis and abnormal situation description.

[0273] B2) Report writing

[0274] Data visualization charts: Insert visualization charts in the report to intuitively display monitoring data and analysis results, including time series charts, spatial distribution charts, comparative analysis charts, etc.

[0275] Text description: Use concise and clear text to explain the information in the chart, provide detailed analysis results and professional insights.

[0276] Conclusions and Suggestions: Based on the data analysis results, put forward specific conclusions and feasible suggestions, such as improvement measures, policy recommendations, etc.

[0277] and optionally, B3) Report review and release

[0278] Internal review: Submit the prepared report to relevant experts and departments for review to ensure the accuracy and professionalism of the content.

[0279] Revision and improvement: Revise and improve the report based on review comments.

[0280] Report release: Release the report through official websites, emails, press conferences, etc. to ensure that the report can be communicated to relevant stakeholders and the public in a timely manner.

[0281] Through the above steps, data visualization and report generation can effectively display the analysis results of environmental monitoring data, provide intuitive information support, provide a strong basis for environmental protection decision-making, and enhance the public's understanding and attention to environmental conditions.

[0282] In summary, the disclosed embodiments solve the problem of large data differences and diverse formats by standardizing environmental data from multiple sources (such as noise, air, water, soil, atmosphere, etc.), and provide consistent data services for upper-layer applications through packaging and organization. At the same time, machine learning and artificial intelligence algorithms are introduced to optimize the data analysis process, realize pollution trend prediction and automatic anomaly detection, and improve the analysis and management capabilities of environmental data.

[0283] The disclosed embodiment also provides a processing system corresponding to the method for performing multi-source heterogeneous processing on environmental monitoring data, such as Figure 6 The structure diagram of a system for performing multi-source heterogeneous processing of environmental monitoring data in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the system 100 for performing multi-source heterogeneous processing on environmental monitoring data may include:

[0284] The acquisition unit 110 is configured to acquire monitoring data related to environmental monitoring, wherein the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter;

[0285] The processing unit 120 is configured to perform cleaning processing on the data of each monitoring parameter respectively; and perform standardization processing on the cleaned data of each monitoring parameter respectively to obtain standard data of each monitoring parameter; wherein the standardization processing includes: standardizing the value and data format of the data of the monitoring parameter;

[0286] The encapsulation unit 130 is configured to respectively encapsulate the standard data of each monitoring parameter to form a plurality of data packets; wherein one data packet corresponds to one monitoring parameter;

[0287] A generating unit 140 is configured to generate a data storage system adapted to the characteristic information of the monitoring data;

[0288] The storage unit 150 is configured to convert the standard data in the plurality of data packets according to the storage format set by the data storage system and store the converted data in the data storage system;

[0289] The construction unit 160 is configured to establish a plurality of data indexes in the data storage system, wherein one data index corresponds to one data packet.

[0290] Among them, the specific working process and principle of the acquisition unit 110, the processing unit 120, the encapsulation unit 130, the generation unit 140, the storage unit 150 and the construction unit 160 can refer to the relevant description of the above examples.

[0291] The present disclosure also provides a computer system suitable for implementing a multi-source heterogeneous processing method for environmental monitoring data, wherein the computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage part to a random access memory (RAM), such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM. The CPU, ROM, and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0292] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. The drive 310 is also connected to the I / O interface as needed. Removable media, such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that the computer program read therefrom can be installed into the storage part as needed.

[0293] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the system of the present application are executed.

[0294] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0295] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0296] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0297] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0298] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0299] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0300] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0301] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0302] Although the embodiments of the present disclosure are disclosed as above, the present invention is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for multi-source heterogeneous processing of environmental monitoring data, characterized in that: include: Acquire monitoring data related to environmental monitoring, wherein the monitoring data includes monitoring parameters, data of each monitoring parameter, and data collection points of each monitoring parameter; Clean the data of each monitoring parameter separately; The data of each monitoring parameter after cleaning are respectively subjected to standardization processing to obtain standard data of each monitoring parameter; wherein the standardization processing includes: standardizing the numerical value and data format of the data of the monitoring parameter; The standard data of each monitoring parameter is respectively encapsulated to form a plurality of data packets, wherein one data packet corresponds to one monitoring parameter; Generating a data storage system adapted to characteristic information of the monitoring data, wherein the characteristic information of the monitoring data includes: the amount of the monitoring data, the types of monitoring parameters included in the monitoring data, and the storage requirements of the data of each monitoring parameter; Converting the standard data in the plurality of data packets according to the storage format set by the data storage system and storing the converted data in the data storage system; Establishing a plurality of data indexes in the data storage system, wherein one data index corresponds to one data packet; The standard data of each monitoring parameter is respectively encapsulated to form multiple data packets, including: For any monitoring parameter, all standard data of the monitoring parameter are split according to a preset time period to form multiple time entries; and all standard data of the monitoring parameter are split according to the geographical location of each data collection point to form multiple geographical location data; For any monitoring parameter, a multi-entry general search tree is used to encapsulate multiple time entries into entries and insert them into the encapsulation module; and a multi-entry spatial partition generalized search tree is used to encapsulate multiple geographic location data into multiple geographic sub-objects and insert them into the encapsulation module to generate a data packet; Wherein, for any data packet, the encapsulation module is suitable for encapsulating the data of each data collection point as a primary object, the data in each time period as a secondary object, and the data of each pollutant index as a sub-object; Wherein, the monitoring parameter is air quality, a multi-entry general search tree is used to encapsulate multiple time entries into entries, and insert them into the encapsulation module; and a multi-entry spatial partition generalized search tree is used to encapsulate multiple geographic location data into multiple geographic sub-objects, and insert them into the encapsulation module to generate a data packet, including: Define the package structure, including: Main object: data of each data collection point; Secondary objects: data for each time period; Sub-object: pollutant index data; Design the encapsulation function of the encapsulation module: Use the MGeST module to split the daily data by hour and encapsulate it into multiple entries; and use the MSP-GiST module to split the data of each data collection point by geographic location and encapsulate it into multiple sub-objects; Encapsulation data processing, including: receiving daily raw data, splitting it into items by hour using an encapsulation module, and encapsulating the data of each data collection point into sub-objects according to geographical location; The step of generating a data storage system adapted to the characteristic information of the monitoring data includes: According to the amount of the monitoring data, based on the characteristics of a non-blocking adjustable hash table with fast deletion and memory awareness functions, a database system with high concurrency, fast deletion and dynamically adjustable data structure is selected from multiple different types of database systems; According to the types of monitoring parameters included in the monitoring data, multiple hash tables are configured in the database system, wherein one hash table corresponds to one monitoring parameter; According to the storage requirements of the data of each monitoring parameter, combined with the non-blocking characteristics of the non-blocking adjustable hash table with fast deletion and memory awareness functions, the corresponding hash table is divided into multiple small fragments, and respectively set on different nodes in the database system, and the nodes are used to store the monitoring data; Wherein, after converting the standard data in the plurality of data packets according to the storage format set by the data storage system and storing them in the data storage system respectively, the processing method further includes: Managing standard data corresponding to each data packet stored in the data storage system; including at least one of the following: According to the weight value and update frequency of each monitoring parameter, formulate the frequency and type of data backup, as well as the data recovery strategy; Performing encryption operations on each data packet stored in the data storage system; Performing encryption operations when transmitting each data packet stored in the data storage system; Set access permissions, generate audit logs, and refuse to perform corresponding operations and return corresponding warning information when it is detected that the visitor's request exceeds the corresponding data management permissions; Evaluate the standard data corresponding to each data package according to the pre-defined data quality indicators and within the set time limit; Recording each data update information of the standard data corresponding to each data packet; Wherein, the establishing of multiple data indexes in the data storage system includes: Classify the standard data in each data packet according to the monitoring parameters and uses corresponding to the standard data in each data packet; According to the type of standard data in each data packet, an adaptive index of adaptive hotspot perception is used as the data index, and an index field for characterizing the standard data is set, wherein the hotspot perception mechanism of adaptive hotspot perception is used to monitor the access mode of the data in real time and identify the frequently accessed data area; Monitoring the access frequency of each index field, and caching the data packets corresponding to the index fields with higher access frequency into the memory according to the hotspot perception mechanism of the adaptive index of the adaptive hotspot perception, or raising the priority of the data packets corresponding to the index fields with higher access frequency, including: Identify the index fields corresponding to the hotspot areas as fields with high access frequency, so as to cache the hotspot data in memory; or dynamically adjust the priority of the index structure according to the access pattern monitored in real time; Regularly analyze data access logs and optimize indexing strategies.

2. The processing method according to claim 1, characterized in that: The obtaining of monitoring data related to environmental monitoring includes: According to the monitoring parameters, determining at least one data collection point adapted to the monitoring parameters; A data acquisition device for acquiring data of the monitoring parameters is provided at the data acquisition point; According to the monitoring data of the monitoring parameters, configure the collection frequency and data transmission mode of the data collection device; The initial monitoring data related to environmental monitoring is acquired through the data acquisition device, and the data format of the real-time initial monitoring data is converted to obtain the monitoring data.

3. The processing method according to claim 1, characterized in that: The cleaning process of the data of each monitoring parameter comprises: Identify missing values ​​in the data of each monitoring parameter, and process the missing values ​​based on the missing ratio and weight of the missing values; including at least one of the following: when it is determined that the missing ratio of the missing values ​​is lower than a preset ratio threshold and the weight is less than a preset weight threshold, delete the missing values; when it is determined that the missing ratio of the missing values ​​is higher than the preset ratio threshold and the weight is higher than the preset weight threshold, use interpolation processing to fill the missing values; Identify abnormal values ​​in the data of each monitoring parameter, and process the abnormal values ​​based on the relationship between the abnormal values ​​and the numerical values ​​in the data; including at least one of the following: when it is determined that the absolute value of the difference between the abnormal value and the numerical value of the data is greater than a preset difference threshold, delete the abnormal value; when it is determined that the absolute value of the difference between the abnormal value and the numerical value of the data is less than a preset difference threshold, select multiple numerical values ​​adjacent to the abnormal value and replace the abnormal value with the average value of the multiple numerical values; retain the abnormal value and mark it.

4. The processing method according to claim 1, characterized in that: Also includes: According to the standard data in each data packet stored in the data storage system, a pre-trained prediction model is used to generate a corresponding environmental prediction result; And, comparing the environmental prediction result with the actual result, and visualizing the comparison result.

5. The processing method according to claim 1, characterized in that: Also includes: Based on the standard data in each data packet stored in the data storage system, an environmental monitoring report is generated based on a preset statistical data generation template.

Images