Environmental Data Quality Correction Method

Abstract

The present invention relates to the technical field of environmental data processing, and specifically to an environmental protection data quality correction method, which uses a coding rule as the factor coding or unique number of a data generating device for application; the coding rule includes at least four coding layers, wherein: the first layer of coding includes a first letter number set according to the type of pollution emission; the second layer of coding includes a second letter number set according to the type of treatment process; the third layer of coding includes a first digit number set according to the monitored data generating device; the fourth layer of coding includes a third digit number set for distinguishing multiple data generating devices in the same working condition environment. Through the application of these four layers of coding, the data generating device can be set with factor coding or unique number, so that the data transmission control platform using it has the characteristics of reliable data quality acquisition and easy operation and maintenance.

Description

Technical Field

[0001] The present invention relates to the technical field of environmental data processing, and particularly relates to a method for correcting the quality of environmental protection data. Background Art

[0002] As the environmental protection department gradually strengthens the supervision of enterprises' environmental protection, especially the supervision of typical pollution source enterprises, due to the immature environmental supervision mechanism and the low level of environmental management informatization, there are many problems in the process of enterprises reporting environmental protection data and disclosing environmental information. These problems are the topics of concern to the supervision department and the environmental protection industry. Using big data analysis technology to monitor, correct and analyze the quality of ecological environment monitoring data has become an effective means for the quality control of pollutant monitoring data.

[0003] There is no good feedback and processing mechanism for the abnormal data of environmental on-line monitoring equipment under abnormal conditions, which leads to a large operable space in the process of enterprises reporting data, and there is no suitable system for effective data quality supervision. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for correcting the quality of environmental protection data to overcome the deficiencies of the prior art.

[0005] A method for correcting the quality of environmental protection data, which uses a coding rule as the factor coding or unique number of the data generating device for application; the coding rule includes at least four coding layers, wherein: the first layer of coding includes the first letter number set according to the type of pollution discharge; the second layer of coding includes the second letter number set according to the type of treatment process; the third layer of coding includes the first digital number set according to the monitored data generating device; the fourth layer of coding includes the third digital number set for distinguishing multiple data generating devices in the same working condition environment.

[0006] Furthermore, preliminary data monitoring results are obtained, and according to the application of the factor coding, retrospective analysis is carried out on the data generating device involved in the data monitoring results, and the data monitoring results are corrected and changed accordingly.

[0007] Furthermore, a communication connection for interaction is formed between the gateway receiving end for data reception and the data generating device, and through the gateway receiving end, feedback control processing for data quality correction is performed on the data generating device.

[0008] Furthermore, the feedback control processing includes one or more correction instructions such as data supplementation processing, firmware upgrade processing, device restart processing or time correction processing.

[0009] Further, it is preset that there is a standard protocol specification, and the data is compared with the corresponding standard protocol specification, and the data is processed with label identification according to the situation of the standard protocol specification.

[0010] Further, based on the application of factor coding, the third-layer coding includes a second digital number set according to the monitoring factor after the first digital number; the second digital number includes a 1-bit second digital value set for the status of the data generation device, real-time current, cumulative power consumption, real-time voltage or real-time power situation, and the second digital value includes the numbers 0-9.

[0011] Further, for the application setting of the unique number, it is set based on the data processing method of distributed stream processing, which includes the following steps: S1. There is a data generation device, and the data received from the data generation device is divided; S2. The divided data is cleaned; S3. The cleaned data is merged; S4. The data is directly output or inductively calculated; in step S1, each of the monitoring devices is set with a unique number of the device through the coding rule, so that the data in the data stream can be divided according to the unique number.

[0012] Further, in step S4, it includes the following steps: the data is shunted, and the data obtained by the shunting includes: real-time data packets and phase data packets divided according to the time division standard, and the data volume of the real-time data packets is greater than that of the phase data packets; the obtained real-time data packets are directly output and directly displayed; the obtained phase data packets are inductively calculated and displayed in the form of a data chart after the inductive calculation.

[0013] Further, in step S1, it also includes the following steps: the gateway receiving end for data reception is marked with a unique TCP connection identifier; a KeyBy operator operation is performed on the received data stream, so that the data in the data stream can be divided according to the unique number or unique identifier; in step S2, the data stream after KeyBy is cleaned, and the cleaning includes confirming whether the corresponding data contains the above-mentioned unique device number; when the data fails the verification, the corresponding data packet is marked as dirty data and sent to the dirty packet message queue; in step S3, the merging method includes performing a window operator operation on the cleaned data, and the operator operation time is 30 seconds.

[0014] Further, in the first - layer coding, the first letter code includes the letter "g" set for gas pollution emissions; in the second - layer coding, the second letter code includes the letter "a" set for the pollution - generating link, the letter "b" set for the collection link, and the letter "c" set for the treatment link; in the third - layer coding, the first digit code includes a 1 - digit first digit value set for different monitoring equipment categories involved in the above - mentioned pollution - generating link, collection link, or treatment link, and the first digit value includes digits 0 - 9; in the fourth - layer coding, the third digit code includes a 2 - digit third digit value, and the third digit value includes digit combinations 00 - 99.

[0015] The beneficial effects of the present invention are as follows:

[0016] 1. Through the application of the four - layer coding set for differentiating pollution emission types, treatment process types, monitored data - generating devices, and multiple data - generating devices in the same working condition environment, the data - generating devices can be set with factor coding or unique numbers, enabling the data transmission and control platform using the data to have the characteristics of reliable data quality and easy operation and maintenance.

[0017] 2. Based on the application of factor coding set by the coding rules, users can conveniently judge the authenticity and rationality of the uploaded monitoring data, effectively preventing data fraud and improving data quality.

[0018] 3. Based on the application of the unique number set by the coding rules, the corresponding data partitioning and processing program can decouple the reported monitoring data according to the unique number of the device, improving the throughput of data processing and the cohesion of the program architecture, and effectively reducing the complexity and hardware cost in the data partitioning and processing process.

[0019] 4. Through the clear partitioning application of real - time data packets with a large amount of basic data and those with a small amount of basic data, while ensuring an effective data display effect, it can avoid excessive computational workload in the report calculation process of the database, reduce the database burden, and effectively reduce the hardware setting cost. Detailed implementation manners

[0020] In order to make the technical solutions, objectives, and their advantages of the present invention clearer, the following embodiments are used to further explain and illustrate the present invention.

[0021] A data transmission control platform of the present invention is set based on a traditional web development architecture with front-end and back-end separation. In this data transmission control platform, a gateway receiving end, a data processing end, and a data terminal are provided; based on the communication transmission method of the prior art, the gateway receiving end can effectively collect relevant pollution source data of pollution source generating equipment or pollution source treatment equipment. After the data processing end obtains and cleans the data of the corresponding gateway receiving end, it analyzes and statistically processes the obtained clean data and displays it in the data terminal.

[0022] Embodiment 1:

[0023] In the data processing end: The distributed stream processing application uses the message queue as the data source, and pulls the qualified messages that have been simply processed by the gateway receiving end in real time. According to predefined data quality and permission check rules (such as the time identifier of the message, the range of multiple factors, the access item information of the sensor, the affiliated enterprise information, and the geographical location information, etc.), it filters and processes the original messages of the pollution source upload monitoring data in real time, performs multi-dimensional, multi-factor, and multi-level cleaning, filtering, and correction on the problem data, tags the original problem data respectively and sends it to the abnormal data queue (automatically eliminating invalid data), and sends the cleaned data to the clean data queue.

[0024] The specific processing steps of this data processing end are as follows:

[0025] 1. Set a unique number for the data generating device, and mark the gateway receiving end for data reception with a unique TCP connection identifier, and perform a KeyBy operator operation on the received data stream, so that the data in the data stream can be unpacked and divided according to the unique number or unique identifier. The setting of the above unique number includes setting according to the identification code of the pollution discharge enterprise manufacturer to which the device belongs, or based on the identification code of the affiliated business (such as atmosphere, surface water, sewage, etc.), or the identification code corresponding to the production line (production facilities, collection facilities, treatment facilities, reuse facilities, etc.), and integrating them as the unique number of the device.

[0026] 2. Clean and process the data stream after KeyBy. For example: confirm whether the received data segment contains the unique number or unique identifier, unpacking and response mark, total number of packets, packet number, request code (message sending time), system code, access password, and command code (this command code can be used to distinguish subsequent real-time data, minute data, or hourly data, etc.); the sending time of the data cannot exceed the set range; whether the factors in the data area are included in the set configuration dictionary. If the data fails the verification, mark the corresponding data message as dirty data and send it to the dirty message queue for statistics.

[0027] 3. Merge the data after cleaning. For example, perform a 30 - second window operator operation on the cleaned data stream. The goal is to verify all the packets received within 30 seconds after the window is triggered, and merge the packets that conform to the packet - combining rules in the national standard protocol.

[0028] 4. Shunt the data packets after cleaning. According to the command encoding of the packets, the obtained data packets include real - time data packets collected at a period of 1 to 30 seconds, minute - data packets collected at a period of 1 to 30 minutes, hour - data packets collected at a period of 1 to 24 hours (generally speaking, for real - time data packets, the data obtained at a period of 30 seconds is selected; for minute - data packets, the collected data obtained at a period of 30 minutes is selected; for hour - data packets, the collected data obtained at a period of 24 hours is selected), and device - status data packets (including device - online status data for confirming whether the device is online).

[0029] In this solution, based on the settings in steps 1 to 3, by first dividing and unpacking the received data and then cleaning it, the cleaning process can be made targeted, thereby improving the cleaning speed, reducing the pressure on subsequent ETL, and at the same time simplifying the development difficulty of other applications.

[0030] Based on the setting in step 4, the purpose is to facilitate the decoupling of the program architecture for subsequent services. In terms of the data volume, the number of real - time data packets is much larger than that of minute - data packets, hour - data packets, and device - status data packets. If the real - time data packets are directly added to the database report for statistical calculation, it will be a huge burden on the database and will also bear high hardware costs. Therefore, for subsequent services, this processing process will divide and set the real - time data packets, directly calculate the information digest of each real - time data in the real - time data packets, and provide it to the front - end for direct display in the form of a hash - table cache. While the minute, hour, and even daily data packets collected at a daily cycle will participate in the induction and calculation of specific different service reports and be used for display after further data charts are generated. By clearly dividing and applying the real - time data packets with a large basic data volume and the data packets with a small basic data volume, it can ensure an effective data display effect, avoid excessive computation in the database report calculation process, reduce the database burden, and effectively reduce the hardware setting cost.

[0031] Example 2:

[0032] On the other hand, in practical applications, the data generation device involves the application of multiple monitoring factors. Therefore, the data corresponding to each monitoring factor in the data generation device can be grouped during collection and application. Then, we set a factor code for the data generation device to perform traceable analysis and judgment on the data generated by the data generation device subsequently.

[0033] Taking the one-way application from the upstream to the downstream of a sewage production line as an example, from the upstream to the downstream, there are pollution generation links, collection links, and treatment links due to different processes. Corresponding to each different treatment link, different treatment devices are applied to the sewage production line. Each independent treatment device is used as a corresponding data generation device and is monitored in real time, and the generated corresponding data is sent to the gateway receiving end.

[0034] The factor code of the relevant treatment device records the code number corresponding to the treatment link.

[0035] For example, when the user obtains the relevant displayed data after cleaning processing in the data terminal, if it is confirmed that the displayed message shows that the operating state of the device in the pollution generation link is an abnormal state (preliminary data monitoring result), the user can trace back the treatment link according to the factor code of the device involved in the message, and analyze that the monitoring data in the pollution generation link and the subsequent collection link and treatment link may be considered untrustworthy; then the user can perform invalid marking processing on the data obtained in the corresponding link and subsequent links of the designed device to complete the correction and change of the data monitoring result.

[0036] The factor code of the relevant treatment device records the code number of the monitoring factor corresponding to the data generation device.

[0037] For the monitoring devices of certain specific pollutants, the corresponding program presets the upper and lower limits of the setting of its monitoring factors according to the setting of its factor code. If the relevant factor data in the uploaded data of the corresponding device exceeds its set upper and lower limits, the message will be directly marked as dirty data. This can avoid the situation of data fraud or excessive data deviation affecting data quality.

[0038] Embodiment 3:

[0039] Based on the quality management requirements of data, the further settings of the present invention are as follows:

[0040] According to the standards of the national standard HJ212 protocol, corresponding standard protocol specifications are set. In step 4 above, the data obtained after cleaning (including the directly output data or the data obtained through inductive calculation) is compared with the corresponding standard protocol specifications. According to the standard protocol specifications, when the obtained data does not meet the HJ212 protocol standard of the country, label identification processing is performed on the data (including but not limited to the highlighted display of the label), which is convenient for data quality assurance personnel to effectively investigate, maintain, and manage the monitoring situation of pollution sources.

[0041] The application of label identification in this embodiment can be used as a supplement to the application in Embodiment 2 above; the message from label identification to display is in an abnormal state, which helps users to indicate the monitoring of equipment anomalies.

[0042] Through the application of label marking, for the reported data of pollution source enterprises with high frequency, large density, and many factors, real-time display and alarm of the data are realized according to industry experience and pain points in operation and maintenance. This enables the platform users and data quality assurance personnel to timely understand the pollution discharge status, data reporting status, and equipment operation status of pollution source enterprises, make decisions and take actions, realize the management of pollution source equipment, the informatization and automation of equipment operation, and realize the "cloudification" of pollution source monitoring equipment.

[0043] Embodiment 4:

[0044] Based on the equipment operation and maintenance application requirements of the above embodiments, the further settings of the present invention are as follows:

[0045] The gateway receiving end for data reception forms an interactive communication connection with the data generating device. Through the gateway receiving end, for the directly output data or the inductive calculation result obtained in step S4, the above data quality assurance personnel can perform feedback control processing for data quality correction on the data generating device to which the labeled data belongs. Specifically, the feedback control processing includes one or more correction instructions such as data supplementation processing, firmware upgrade processing, device restart processing, or time correction processing. Through the corresponding correction instructions, the corresponding device makes correction actions, effectively ensuring the quality of the subsequent obtained data.

[0046] Embodiment 5:

[0047] In the application of the above embodiments, if there is an application of the unique number of the corresponding data generating device or factor coding, as a preferred embodiment, a coding rule is preferably set in this embodiment, so that the unique number or factor coding is set based on this coding rule to effectively promote the application of the data transmission control platform.

[0048] The coding rule includes: at least four layers of coding layers are set, where: the first layer of coding includes a first letter number set according to the type of pollution emission; the second layer of coding includes a second letter number set according to the type of treatment process; the third layer of coding includes a first digit number set according to the monitored data generating device, and / or a second digit number set according to the monitoring factors of the corresponding data generating device when the factor coding is applied, and the second digit coding can be cancelled in the application of the unique number of the device; the fourth layer of coding includes a third digit number set for distinguishing multiple devices located at different positions in the same working condition environment.

[0049] Taking the VOCs working condition monitoring process as an example:

[0050] In the first layer of coding, the first letter number includes the letter "g" set for gas pollution emissions.

[0051] In the second layer of coding, the second letter number includes the letter "a" set for the pollution generation link, the letter "b" set for the collection link, and the letter "c" set for the treatment link.

[0052] In the third layer of coding, the first digit number includes a 1-digit first digit value set for different monitored data generating device categories involved in the above pollution generation link, collection link or treatment link, and the first digit value includes the numbers 0-9; the second digit number includes a 1-digit second digit value set for monitoring factors such as the status of the data generating device, real-time current, cumulative power consumption, real-time voltage or real-time power, and the second digit value includes the numbers 0-9.

[0053] In the fourth layer of coding, the third digit number includes a 2-digit third digit value, and the third digit value includes the number combination 00-99. For example, the current of the No. 1 UV purification device and the current of the No. 2 UV purification device are distinguished by 01 and 02 respectively in the fourth layer.

[0054] Based on the setting of this coding rule, corresponding to the basic setting application situation in the above Embodiment 1, in the cleaning process, it can more effectively implement the unpacking and partitioning application for the required data types, making the cleaning process more targeted, thereby more effectively improving the cleaning processing speed.

[0055] Based on the setting of this coding rule, corresponding to the basic setting application situation in the above Embodiment 2, based on the application of the first letter number recording the corresponding processing link in its factor coding, when the device status code is abnormal, the monitoring data collected in the application link and subsequent links of the device are directly marked as invalid, ensuring the quality of the final obtained data.

[0056] In addition, for monitoring devices for certain specific pollutants, the corresponding program presets upper and lower limits for the setting of their monitoring factors according to the factor coding setting (for example, in the application of the second digit number of the corresponding third-layer coding, the real-time current situation is involved, and the program correspondingly sets the upper limit value of the real-time current to 20 A). If the uploaded data of the corresponding device exceeds the set upper and lower limits, the message will be directly marked as dirty data. This can avoid the situation of data fraud or excessive data deviation affecting data quality.

[0057] The above is only the preferred implementation mode of the present invention. For those skilled in the art of this technology, without departing from the implementation principle of the present invention, the described embodiments can still be modified, and the corresponding modification schemes should also be regarded as the protection scope of the present invention.

Claims

1. An environmental protection data quality correction method, characterized in that, It adopts a coding rule as the factor coding or unique number of the data generating device for application; the coding rule includes at least four coding layers, where: The first-layer coding includes a first letter number set according to the pollution emission type; The second-layer coding includes a second letter number set according to the treatment process type; The third-layer coding includes a first digit number set according to the monitored data generating device; and / or a second digit number set according to the monitoring factors of the corresponding data generating device when the factor coding is set for application; The fourth-layer coding includes a third digit number set for differentiating multiple data generating devices in the same working condition environment; For the application setting of the unique number, it is set based on the data processing method of distributed stream processing, which includes the following steps: S1. There is a data generating device, and the data received from the data generating device is divided; S2. The divided data is cleaned; S3. The cleaned data is merged; S4. The data is directly output or inductively calculated; In step S1, each monitoring device is set with a unique number of the device through the coding rule, so that the data in the data stream can be divided according to the unique number; In step S4, it includes the following steps: the data is shunted, and the obtained data includes: real-time data packets and phase data packets divided according to the time division standard, and the data volume of the real-time data packets is greater than that of the phase data packets; the obtained real-time data packets are directly output; the obtained phase data packets are inductively calculated; A preliminary data monitoring result is obtained. According to the application of the factor coding, a retrospective analysis is carried out on the data generating device involved in the data monitoring result, and the data monitoring result is corrected and changed accordingly; a communication connection is formed between the gateway receiving end for data reception and the data generating device, and the direct output or inductive calculation result of the data obtained in step S4 is carried out; through the gateway receiving end, a feedback control process for data quality correction is carried out on the data generating device.

2. The environmental protection data quality correction method according to claim 1, wherein The feedback control process includes one or more correction instructions such as data supplementation process, firmware upgrade process, device restart process or time correction process.

3. The environmental protection data quality correction method according to claim 1, characterized in that, A standard protocol specification is preset, and the data is compared with the corresponding standard protocol specification, and the data is labeled according to the standard protocol specification.

4. The environmental protection data quality correction method according to claim 1, characterized in that Based on the application of the factor coding, the third-layer coding includes a second digit number set according to the monitoring factors after the first digit number; the second digit number includes a 1-bit second digit value set for the status, real-time current, cumulative power consumption, real-time voltage or real-time power condition of the data generating device, and the second digit value includes the numbers 0-9.

5. The environmental protection data quality correction method according to claim 1, characterized in that, In step S1, the following steps are further included: making the gateway receiver for data reception be marked with a unique TCP connection identifier; performing a KeyBy operator operation on the received data stream so that the data in the data stream can be divided according to the unique number or unique identifier; in step S2, performing a cleaning process on the data stream after KeyBy, and the cleaning process includes confirming whether the corresponding data contains the above-mentioned device unique number; when the data fails the verification, the corresponding data packet is marked as dirty data and sent to the dirty packet message queue; in step S3, the merging method includes performing a window operator operation on the data after the cleaning process, and the operator operation time is 30 seconds.

6. The environmental protection data quality correction method according to any one of claims 1 to 5, characterized in that In the first-layer encoding, the first letter number includes the letter "g" set for gas pollution emissions; In the second-layer encoding, the second letter number includes the letter "a" set for the pollution generation link, the letter "b" set for the collection link, and the letter "c" set for the treatment link; In the third-layer encoding, the first digit number includes a 1-digit first digit value set for different monitoring device categories involved in the above-mentioned pollution generation link, collection link, or treatment link, and the first digit value includes the digits 0-9; In the fourth-layer encoding, the third digit number includes a 2-digit third digit value, and the third digit value includes the digit combination 00-99.