Environmental monitoring data validity judgment method and system based on data marking

Abstract

The invention provides an environmental monitoring data validity judgment method and system based on data marking, and relates to the field of data monitoring. The method comprises the steps of extracting multiple pieces of to-be-archived data from environment monitoring data according to an acquisition format; for the multiple pieces of to-be-archived data, determining the to-be-archived data matched with the to-be-collected object of the preset contract table as to-be-standardized data; converting the format of the to-be-standardized data; according to the standard monitoring data, performing assignment processing on a corresponding object needing to be collected in a preset contract table; according to a data validity calculation rule and the value of the subordinate factor in the assigned preset contract table, calculating a validity mark of the archive target factor in the assigned preset contract table; performing data archiving processing on the assigned preset contract table according to the validity mark to obtain archive data; and storing the archive data. According to the method, the environment monitoring data validity judgment capability can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring, and in particular to a method and system for judging the validity of environmental monitoring data based on data labeling. Background Art

[0002] As the basis for judging environmental quality, the effectiveness and traceability of environmental monitoring data have very important guiding significance for the supervision and governance of the entire ecological environment.

[0003] Currently, there are three main ways to determine whether monitoring data is valid (that is, to determine effectiveness):

[0004] 1. Simple data labeling. This method is mainly achieved by marking the data (the label is simply represented by letters) when a fault or alarm occurs in the analytical instrument or a part of the system. For example, in a water quality monitoring system, when the system determines that water sampling has failed (it may be that the sensor is damaged, not a real water sampling failure), the measured data will be marked as "Z". This method has a single meaning and poor scalability. It cannot reflect the complex working environment on site and the complex coupling factors between working conditions. In addition, when such labeling involves data labeling priority, it will cause the problem of cross-coverage of abnormal situations.

[0005] 2. Platform data review. This method mainly uses manual methods to review the validity of data. When the data reviewer sees the data and feels abnormal, he or she will determine whether the data is valid by looking up logs, monitoring, alarms, historical environmental parameters, and telephone inquiries. This method is inefficient and has serious lags. Faced with a huge amount of data, it is impossible to quickly and accurately identify it.

[0006] 3. Quality control of the instrument. This method mainly involves: verifying the monitoring instrument with standard substances. If the verification fails, the data generated between the last verification and the current verification failure is considered invalid. The disadvantage of this method is that it is not timely enough. After multiple sets of data have been generated, the data that has been generated is judged invalid.

[0007] In general, the current method of judging data validity is mainly achieved through the superposition of the above methods, which can achieve data validity judgment to a certain extent. However, there are still problems such as insufficient validity judgment ability, such as single dimension of data validity judgment, lag, and difficulty in using in complex environments with multiple devices integrated. Summary of the invention

[0008] In view of this, the purpose of the present invention is to provide a method and system for judging the validity of environmental monitoring data based on data labeling, which can improve the ability to judge the validity of environmental monitoring data, and has stronger flexibility, timeliness, and time and space specificity. The data processing process is easier to unify on each independent device, and the amount of calculation and resource occupancy are lower.

[0009] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0010] In a first aspect, the present invention provides a method for judging the validity of environmental monitoring data based on data labeling, the method comprising the following steps: obtaining environmental monitoring data collected by a monitoring device, wherein the storage format of the environmental monitoring data is determined by the collection format of the monitoring device; extracting a plurality of data to be archived from the environmental monitoring data according to the collection format; for the plurality of data to be archived, determining the data to be archived that matches the objects to be collected in a preset contract table as data to be standardized; wherein the preset contract table includes a plurality of objects to be collected, and the plurality of objects to be collected include a system code, an archive target factor, and a subordinate factor, the system code is used to represent the identification code of the monitoring device, the archive target factor is used to represent the target monitoring data that needs to be judged for validity in the environmental monitoring data, and the subordinate factor is used to represent the target monitoring data that needs to be judged for validity in the judgment that the target monitoring data is The environmental monitoring data required in the process of determining whether the environmental monitoring data is valid or not; converting the format of the data to be standardized according to the predetermined storage format corresponding to the object to be collected, so as to obtain the standard monitoring data corresponding to the data to be standardized; assigning the corresponding object to be collected in the preset contract table according to the standard monitoring data, so as to obtain the preset contract table after assignment; calculating the validity mark of the archive target factor in the preset contract table after assignment according to the data validity calculation rules and the value of the subordinate factor in the preset contract table after assignment; wherein the validity mark is used to indicate whether the archive target factor is valid; archiving the data of the preset contract table after assignment according to the validity mark, so as to obtain archive data; wherein the archive data includes information used to indicate the validity mark; storing the archive data.

[0011] It should be understood that the method provided in the embodiments of the present application has the following beneficial effects:

[0012] The process of extracting the data to be archived and matching it with the preset contract table to determine the data to be standardized can ensure the pertinence and accuracy of the data processing process. Using the system coding in the preset contract table, different monitoring devices can be clearly identified, which is convenient for unified management in a complex environment with multiple devices integrated. The archive target factor determines the key monitoring data that needs to be judged for effectiveness, and the subordinate factor provides reference information for the effectiveness judgment of the target monitoring data, enriching the dimension of effectiveness judgment, thereby overcoming the problem of single judgment dimension in the existing technology.

[0013] The standardized data is converted into a format to obtain standard monitoring data and assign values ​​to a preset contract table, thus achieving data format unification. This enables environmental monitoring data in different collection formats to be analyzed in a unified format, thus improving the efficiency of data processing. In addition, the validity mark of the archive target factor is calculated according to the data validity calculation rules and the value of the subordinate factor, which can enhance the accuracy of the judgment. Compared with the prior art, this technical solution is more timely and can make validity judgments at the same time as the data is generated, thus avoiding the lag of post-judgment and the inefficient manual judgment process.

[0014] Furthermore, the data is archived according to the validity mark, and the obtained archive data contains validity mark information, which can facilitate the subsequent data query and analysis process. The storage of archive data also makes the environmental monitoring data traceable, which facilitates the regulatory authorities to conduct comprehensive and accurate supervision of the ecological environment. Moreover, due to the unified data format, the amount of calculation and resource usage are lower, which is more suitable for the processing needs of large-scale environmental monitoring data.

[0015] In a second aspect, the present invention provides a device for determining the validity of environmental monitoring data based on data labeling, comprising a unit for implementing the method described in the first aspect.

[0016] In a third aspect, the present invention provides an environmental monitoring data validity judgment system based on data labeling, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the system executes the method described in the first aspect.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A module block diagram of the environmental monitoring data validity judgment system provided by an embodiment of the present invention;

[0020] Figure 2 A flow chart of a method for determining the validity of environmental monitoring data based on data labeling provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a flow chart of another method for recording the validity of environmental monitoring data provided by an embodiment of the present invention;

[0022] Figure 4 A functional module diagram of a device for determining the validity of environmental monitoring data based on data labeling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0027] In view of some or all of the problems mentioned in the background technology, the embodiments of the present invention provide a method and system for judging the validity of environmental monitoring data based on data labeling, which relates to the field of data monitoring, can improve the ability to judge the validity of environmental monitoring data, and has stronger flexibility, timeliness, and time and space specificity. The data processing process is easier to unify on each independent device, and the amount of calculation and resource occupancy are lower.

[0028] The technical solution provided by the present invention will be described below in conjunction with the accompanying drawings.

[0029] First, the environmental monitoring data validity judgment system in the embodiment of the present invention may include at least one computing device, and each computing device includes a processor and a memory.

[0030] The processor of at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the system executes the method for determining the validity of environmental monitoring data based on data marking as recorded in the technical solution.

[0031] Specifically, the environmental monitoring data validity judgment system based on data marking provided by the embodiment of the present invention is described in conjunction with the accompanying drawings. Figure 1 , Figure 1 A module block diagram of the environmental monitoring data validity judgment system provided by an embodiment of the present invention. The environmental monitoring data validity judgment system 100 may include: a monitoring system 1, a data acquisition device 2, a data archive device 3 and a platform device 4.

[0032] The monitoring system 1 is used to measure the concentration of pollutants in the current environment. Those skilled in the art should know that the monitoring equipment does not only include an analyzer for analyzing pollutants, but also includes other hardware equipment to ensure its operation, including but not limited to power supply equipment, environmental monitoring equipment, etc. The monitoring content can be soil monitoring data, water quality monitoring data, air monitoring data, noise monitoring data, etc. In other words, the monitoring system 1 can be used to monitor the environment in real time and generate environmental monitoring data, and can be a device type including various sensors, detectors, etc. For example, it can include but not limited to water quality sensors, temperature sensors, pressure sensors, etc.

[0033] The data acquisition device 2 is used to obtain the monitoring information sent by the monitoring system 1, and send the monitoring information to the data archive device 3 and the platform device 4. In other words, it can be used to collect the environmental monitoring data generated by the monitoring system 1, and perform preliminary processing and transmission on the data. It can be a data acquisition terminal with data receiving, conversion and transmission functions.

[0034] The data archive device 3 can be used to obtain monitoring information from the monitoring system 1 and / or the data acquisition device 2. In other words, it can be used to store and manage archive data that has been processed for validity judgment, and provide support for data query, analysis and tracing.

[0035] The monitoring information refers to: all parameter information, status information, data information, etc. of concern that may affect the validity of the data during the operation and data generation process of the monitoring system 1 and / or the data acquisition device 2. Monitoring information and environmental monitoring data can be interchangeable. Those skilled in the art should be aware that there is no single way to obtain the monitoring information. In this example, it can be the parameters, status, and data information directly obtained from the actual equipment in the monitoring system 1, or it can be the result obtained indirectly and calculated through an algorithm.

[0036] Optionally, the data archive device 3 can be a computing device such as a computer, notebook, microcomputer, server, etc. with large-capacity storage and efficient data management functions, or the method embodiment can be implemented on the data acquisition device 2 through computer program instructions.

[0037] The data archive device 3 can obtain and store the monitoring information of multiple or single monitoring systems 1 and / or data acquisition devices 2 using unified preset factor coding and definition through a preset contract. For specific instructions, please refer to the method embodiment below.

[0038] In the data archive device 3, data preprocessing modules can be built-in and defined, and these modules can be directly called and opened. When called, the preset factor codes in the preset contract are associated. The algorithm module performs calculations based on the factor codes and their values, and generates calculation results. For specific instructions, please refer to the method embodiments below.

[0039] The platform device 4 can be used to centrally display, analyze and manage environmental monitoring data, and provide a basis for decision-making for regulatory authorities and relevant personnel. It can be a computer, server, platform system or other type of device with computing power and a visual interface.

[0040] In an optional embodiment, the method for determining the validity of environmental monitoring data based on data labeling in the technical solution can be executed and implemented by the above-mentioned environmental monitoring data validity determination system 100. For example, the data acquisition device 2 is used to execute the environmental monitoring data acquisition, analysis, and processing processes in the embodiment of the method to obtain archival data, and then store the archival data in the data archive device 3 and / or the platform device 4.

[0041] Of course, the method embodiment in this solution can also be executed by other devices in the system 100, without limitation, for example, by the monitoring system 1, the data archive device 3 and / or the platform device 4.

[0042] Combined with the above Figure 1 The following is a method for implementing a method for determining the validity of environmental monitoring data based on data labeling. Figure 2 , Figure 2 The flowchart of a method for determining the validity of environmental monitoring data based on data labeling provided by an embodiment of the present invention is shown. The method can be applied to the above Figure 1 System shown.

[0043] Specifically, the method may include the following steps S210-S280, which are described in sequence below.

[0044] S210, obtaining environmental monitoring data collected by monitoring equipment.

[0045] The storage format of environmental monitoring data is determined by the acquisition format of the monitoring device. For example, when the monitoring device is a water quality monitoring sensor, the corresponding acquisition format can be a specific time series data format, including specific values ​​of parameters such as timestamp, water temperature, pH, dissolved oxygen content, etc.

[0046] Environmental monitoring data and collection formats vary depending on the type of monitoring equipment. For example, air monitoring equipment may collect data on various pollutant concentrations at different time points. The format may be a table format that includes time, pollutant name, and concentration value. There is no limitation on this.

[0047] In this embodiment, the monitoring device and Figure 1 The monitoring systems 1 shown have the same definition and can be replaced with each other.

[0048] S220: extracting a plurality of data to be archived from the environmental monitoring data according to the collection format.

[0049] The data to be archived may include multiple key data used to represent the key data recorded by the monitoring equipment during the environmental monitoring process. For example, for water quality monitoring equipment, the data to be archived may include water temperature, pH, dissolved oxygen content, conductivity and other data at different time points; for air monitoring equipment, the data to be archived may include PM2.5 concentration, PM10 concentration, sulfur dioxide concentration, nitrogen dioxide concentration and other data at different time points.

[0050] S230: For a plurality of data to be archived, the data to be archived that matches the objects to be collected in the preset contract table are determined as data to be standardized.

[0051] The preset contract table can unify and standardize environmental monitoring data in the form of codes and / or factors. When transmitting the same environmental monitoring data (that is, monitoring information) in different systems / devices, the codes used may be different. In view of this situation, the preset codes and / or factors are used for unification in this embodiment, that is, the monitoring content of different systems can be standardized through the preset contract table.

[0052] The preset contract table may include multiple objects to be collected, which may include: system code, archive target factor and subordinate factor.

[0053] The system code is used to represent the identification code of the monitoring equipment. For example, each water quality monitoring sensor can have a unique digital code (SN code) to distinguish different monitoring devices.

[0054] The archival target factor is used to represent the target monitoring data in the environmental monitoring data that needs to be judged for effectiveness. For example, for water quality monitoring data, the archival target factor can be the dissolved oxygen content, which is one of the important indicators for judging the quality of water.

[0055] The subordinate factors are used to represent the environmental monitoring data needed in the process of judging whether the target monitoring data is valid. For example, to judge whether the dissolved oxygen content is valid, water temperature, pH value and other data can be used as subordinate factors because these factors may affect the measurement results of dissolved oxygen.

[0056] In other words, "matching the objects to be collected in the preset contract table" means that the attributes of some data in the data to be archived correspond to the system code, archive target factor or subordinate factor in the preset contract table. Accordingly, the data corresponding to the system code, archive target factor or subordinate factor in the multiple data to be archived can be determined as the data to be standardized. For example, for water quality monitoring data, if the subordinate factor includes water temperature, then the water temperature data in the multiple data to be archived can be determined as the data to be standardized.

[0057] For more specific explanations on system coding, archive target factors and subordinate factors, please refer to the example section below and will not be repeated here.

[0058] S240, converting the format of the data to be standardized according to a predetermined storage format corresponding to the object to be collected, so as to obtain standard monitoring data corresponding to the data to be standardized.

[0059] An example is given below. Assuming that the preset contract table specifies that the storage format of the dependent factor "water temperature" is a numerical format with one decimal place, and the water temperature in the data to be standardized is "7.13", it needs to be converted into the standard monitoring data format of "7.1".

[0060] S250, assigning values ​​to the corresponding objects to be collected in the preset contract table according to the standard monitoring data to obtain the preset contract table after the assignment.

[0061] S260, calculating the validity mark of the archive target factor in the preset contract table after the value is assigned according to the data validity calculation rule and the value of the subordinate factor in the preset contract table after the value is assigned.

[0062] Among them, the validity mark is used to indicate whether the archive target factor is valid.

[0063] Specifically, the data validity calculation rules may include a data preprocessing algorithm, a tag operation algorithm, and a tag generation algorithm. S260 may include the following steps in the implementation process:

[0064] S261, using a data preprocessing algorithm, according to the value of the subordinate factor in the preset contract table after the assignment, calculate the auxiliary monitoring data required in the process of judging the validity of the target monitoring data.

[0065] For example, the implementation of the data preprocessing algorithm can be achieved through a predefined data preprocessing module (for example, the data preprocessing module in Table 2 below), and the data preprocessing module may include multiple predefined preprocessing calculation modules (for example, the one-variable linear equation algorithm module and the state continuity algorithm module in Table 2 below), so as to realize the calculation of auxiliary monitoring data.

[0066] It can be understood that the auxiliary monitoring data here refers to: in the process of judging the effectiveness of the target monitoring data, additional data is obtained by calculating the values ​​of the subordinate factors through a specific data preprocessing algorithm. These additional data can provide more comprehensive information support for further effectiveness judgments. For example, in water quality monitoring, subordinate factors may include water temperature, pH, etc. Through the data preprocessing algorithm, auxiliary monitoring data such as the compensated dissolved oxygen reference value and the theoretical concentration range of specific pollutants adjusted according to the water temperature can be calculated based on these subordinate factors. These auxiliary monitoring data are not collected directly, but are generated by processing existing data, and are used to more accurately judge the effectiveness of archive target factors (such as key pollutant concentrations).

[0067] S262, using a marking operation algorithm, based on the auxiliary monitoring data and the values ​​of the subordinate factors in the preset contract table after assignment, calculate the conditional operation input value in the conditional operation process of judging the validity of the target monitoring data.

[0068] For example, the implementation of the marking operation algorithm can be implemented through a predefined marking operation module (for example, the marking operation module in Table 2 below), and the marking operation module can include multiple predefined marking processing modules (for example, the theoretical ammonia nitrogen concentration value marking operation module, the water tank high liquid level state continuous value marking operation module, the ambient temperature value marking operation module, etc. in Table 2 below), so as to realize the calculation of the conditional operation input value.

[0069] It can be understood that the conditional operation input value here refers to: in the conditional operation process of judging the validity of the target monitoring data, the value used for conditional judgment is calculated by combining the auxiliary monitoring data and the value of the subordinate factor through the marking operation algorithm. For example, in environmental monitoring, the auxiliary monitoring data (such as the adjusted theoretical value) and the actual collected subordinate factor value (such as the relevant parameters of the current measurement) may be compared and operated through the marking operation algorithm to obtain an intermediate value for judging whether the target monitoring data is valid. This intermediate value is the conditional operation input value. It provides key input parameters for the subsequent generation of validity tags according to the preset validity judgment conditional operation rules.

[0070] S263, using a tag generation algorithm, according to the conditional operation input value and the preset validity judgment conditional operation rules, calculate the validity tag of the archive target factor in the preset contract table after the assignment.

[0071] For example, the implementation of the tag generation algorithm can be achieved through a predefined tag generation module (for example, the tag generation module in Table 2 below), and the tag generation module can include multiple predefined generation processing modules (for example, tag generation module 1 and tag generation module 2 in Table 2 below), so as to realize the calculation of the validity tag.

[0072] It can be understood that the preset validity judgment conditional operation rules here refer to: a series of pre-set logical judgment criteria for determining the validity of the archive target factor based on the conditional operation input value. For example, in air quality monitoring, it may be stipulated that if the conditional operation input value (such as the degree of deviation between the actual measured value of a certain pollutant and the theoretical calculated value) is within a certain range, the archive target factor (specific pollutant concentration data) is considered valid, otherwise it is invalid. These rules can be formulated based on industry standards, empirical formulas or specific monitoring needs, and their purpose is to ensure that the validity judgment of environmental monitoring data is scientific and accurate. The preset validity judgment conditional operation rules can include a combination of multiple specific conditional judgment statements and logical operators to adapt to different types of environmental monitoring data and different validity judgment scenarios.

[0073] In other words, the data preprocessing algorithm is used to calculate the auxiliary monitoring data required in the process of judging the validity of the target monitoring data according to the values ​​of the subordinate factors in the preset contract table after assignment; the marking operation algorithm is used to calculate the conditional operation input value in the conditional operation process of judging the validity of the target monitoring data according to the auxiliary monitoring data and the values ​​of the subordinate factors in the preset contract table after assignment; the marking generation algorithm is used to calculate the validity mark of the archive target factor in the preset contract table after assignment according to the conditional operation input value and the preset validity judgment conditional operation rules.

[0074] In an optional embodiment, the input values, output values ​​and algorithm rules of the data preprocessing algorithm, the marking operation algorithm and the marking generation algorithm in the operation process are stored in the form of a preset data processing table. The specific implementation process can refer to the implementation form and related instructions of Table 2 below, which will not be repeated here.

[0075] S270, archiving the data of the preset contract table after the assignment according to the validity mark to obtain archive data.

[0076] The archive data may include information for indicating a validity mark.

[0077] In an optional embodiment, the archival data may include archival codes, data tags, and archiving information.

[0078] The archive code may include: information on the identification code of the monitoring device, time information during the calculation process of the validity mark of the archive target factor, and the value of the archive target factor.

[0079] The data tag may include: information for indicating a validity tag.

[0080] The filing information may include: at least one item of information in the preset contract table after assignment, and at least one item of information in the preset data processing table.

[0081] For the specific implementation of S270, please refer to Table 3 and related instructions below, which will not be elaborated here.

[0082] S280, storing the archive data.

[0083] Among them, when the embodiment of the method is executed by the data acquisition device 2, the data acquisition device 2 can send the generated archive data to the data archive device 3 and / or the platform device 4 for storage. When the embodiment of the method is executed by the data archive device 3, the data archive device 3 can send the generated archive data to the platform device 4 for storage, and this method is not limited. In this way, after the platform device 4 obtains the archive data, it does not need to perform data cleaning and invalid data elimination. It only needs to determine the data tag and then use the data information in the archive.

[0084] It should be understood that according to the above Figure 2 The method embodiment shown in the present invention has the following beneficial effects:

[0085] 1. The process of extracting the data to be archived and matching it with the preset contract table to determine the data to be standardized can ensure the pertinence and accuracy of the data processing process. Using the system coding in the preset contract table, different monitoring devices can be clearly identified, which is convenient for unified management in a complex environment with multiple devices integrated. The archive target factor determines the key monitoring data that needs to be judged for effectiveness, and the subordinate factor provides reference information for the effectiveness judgment of the target monitoring data, enriching the dimension of effectiveness judgment, thereby overcoming the problem of single judgment dimension in the prior art.

[0086] 2. By converting the format of the standardized data, standard monitoring data is obtained and assigned to the preset contract table, so as to achieve the unification of data format. This enables environmental monitoring data of different collection formats to be analyzed in a unified format, thus improving the efficiency of data processing. In addition, the validity mark of the archive target factor is calculated according to the data validity calculation rules and the value of the subordinate factor, which can enhance the accuracy of the judgment. Compared with the prior art, this technical solution is more timely and can make validity judgments at the same time as the data is generated, thus avoiding the lag of post-judgment and the inefficient manual judgment process.

[0087] 3. Data archiving is performed according to validity marks. The obtained archive data contains validity mark information, which can facilitate the subsequent data query and analysis process. The storage of archive data also makes environmental monitoring data traceable, which facilitates the regulatory authorities to conduct comprehensive and accurate supervision of the ecological environment. Moreover, due to the unified data format, the amount of calculation and resource usage is lower, which is more suitable for the processing needs of large-scale environmental monitoring data.

[0088] According to the above Figure 1 , Figure 2 The system and method embodiments provided may be understood as follows:

[0089] When conducting data analysis and environmental situation forecasting, a large amount of data cleaning, elimination, screening and other work is required to identify and utilize the increasingly generated environmental monitoring big data. In view of the deficiencies in the data validity judgment method and data marking method in the prior art, the advantage of this technical solution is that all the influencing factors that users consider necessary in the process of data generation are recorded to form the archival information of the data, and a preset algorithm is set to perform calculations and analysis on the archival data to finally generate a data tag that represents whether the data is valid. Through this tag, it can be intuitively judged whether the data is valid.

[0090] This technical solution proposes a new data tagging and recording method for judging the validity of environmental monitoring data. The data tag can be customized and expanded according to the complexity of the system, and is a universal coding tagging method. This method can effectively form a data archive for each piece of data. The so-called data archive records all the information in the process of the generation of the monitoring data. This information does not require manual judgment and is automatically associated with the data, greatly reducing the time cost of manual judgment.

[0091] In addition, the advantage of this technical solution is that the original Internet data cleaning, data processing, and data analysis functions can be integrated and applied on the device side, changing the original centralized data processing to distributed storage analysis and operation. In addition, when judging the validity of data, the judgment method can be set up in a distributed and adapted manner, which is more flexible, more timely, more targeted in time and space, and easier to unify data processing on each independent system, and has lower computing power and resource occupancy than centralized systems. Among them, the so-called distributed and adapted to local conditions are illustrated by examples: if the temperature in the north is very low in winter, the data measured by the monitoring instrument at low temperatures at this time and place is normal data; but if the equipment is installed in a warm area all year round, the data measured by the monitoring instrument at low temperatures at this time and place will be invalid data (low temperature, proving the occurrence of abnormal conditions). In the face of this situation, the current method is generally manual identification. Data verifiers conduct data checks during daily verification, but this method is easy to miss these abnormal situations, and inefficient in the face of large amounts of data; there are also platforms that use set conditions to filter data, setting the temperature within a threshold to be valid, otherwise invalid, but in the face of a variety of complex scenarios, it is easy to "cut across the board", and it is also easy to make mistakes and omissions. There are many examples like this, which will not be repeated in this article. This shows the advantages of distributed data validity judgment methods / equipment / systems.

[0092] In addition, the advantages of this technical solution also lie in its openness, scalability and ease of use. The openness lies in that each system can be independently marked and the method is universal; the scalability lies in that new modules can be continuously connected and new algorithms can be involved in marking according to the iteration and upgrade of the system hardware; the ease of use lies in that when the platform side conducts big data related analysis, it can complete the data cleaning work by parsing the required data tags, and transfer part of the original platform's functions to the edge devices of the environmental protection Internet of Things through the method described in the present invention.

[0093] Next, we will use a possible example to illustrate the above Figure 2 The specific definition and implementation of each step in the above method embodiment can be supported and illustrated by the following examples.

[0094] Please refer to Figure 1 , Figure 3 And the following Tables 1, 2, and 3. The contents shown in Tables 1, 2, and 3 are an explanation of the validity of the data for judging the water pollutant index [ammonia nitrogen concentration value] according to the above method embodiment. Tables 1, 2, and 3 correspond to the data acquisition and unification, data validity calculation, and archiving process in the above method embodiment.

[0095] 1. For data acquisition and unification process and preset contract table:

[0096] Please refer to Table 1, which shows a preset contract table with the archive target factor set to the ammonia nitrogen concentration value. The preset codes and / or factors in the preset contract table shown in Table 1 include three parts: system code, archive target factor, and subordinate factor. Among them, the system code refers to Figure 1 The unique code of the monitoring system 1 shown is used to identify which system the data comes from. The archive target factor refers to the factor information for judging the validity of the data. The subordinate factor refers to all the monitoring information factors that will be used to judge the validity of the data.

[0097] In Table 1, it is necessary to monitor and judge the data validity of the water pollutant index [ammonia nitrogen concentration value], and to judge whether the index data is valid, factors such as [analyzer curve slope] and [analyzer curve intercept] are needed. Then the archive target factors include [ammonia nitrogen concentration value], and the subordinate factors include [analyzer curve slope], [analyzer curve intercept] and other factors.

[0098] Table 1 Preset Contract Table

[0099] In the preset contract table, the monitoring contents of different systems will be standardized. For example: in one monitoring system, the monitoring information is [water temperature], the unit is Fahrenheit, the data is rounded to N3.4, and the storage register is 40001; in another monitoring system, the monitoring information is [water temperature], the unit is Celsius, the data is rounded to N3.3, and the storage register is 40011. According to the corresponding relationship recorded in the preset contract, the [water temperature] data obtained from different registers will be converted into a unified measurement unit and data format, then: in data archive device 3, the monitoring information [water temperature], the unit is Celsius, the data is rounded to N3.4, and the preset factor code is i01010.

[0100] In one device, multiple preset contract tables can be set according to different archive target factors. Therefore, there can be multiple archive target factors, and each archive target factor can be set to a corresponding preset contract table. In other words, in an optional embodiment, the number of preset contract tables can include multiple, and the archive target factors in different preset contract tables among the multiple preset contract tables are different.

[0101] 2. Regarding the data validity calculation process and data validity calculation rules:

[0102] Please refer to Table 2, which shows a data processing table. The data processing table includes: a data preprocessing module, a tag operation module, and a tag generation module. The data preprocessing module is used to implement the data preprocessing algorithm in the above method embodiment; the tag operation module is used to implement the tag operation algorithm in the above method embodiment; the tag generation module is used to implement the tag generation algorithm in the above method embodiment.

[0103] Table 2 Data processing table

[0104] For example, in Table 2, the linear equation algorithm module can calculate the theoretical ammonia nitrogen concentration value CNH4 = original signal value * analyzer curve slope + analyzer curve intercept according to the original signal value (i12077), analyzer curve intercept (i13007), and analyzer curve slope (i13008), thereby obtaining the theoretical ammonia nitrogen concentration value CNH4. The state continuity algorithm module can monitor the high water level state value of the water tank during the process every 5 seconds according to the process signal (i12088) and the high water level state of the water tank (i12104). If the state value continues to be 01, the output result A=1 is output, otherwise the output A=0. By analogy, the functions of other algorithm modules in Table 2 are not repeated.

[0105] For the above-mentioned data preprocessing module, new algorithms can be set therein according to actual needs, and the number and form are not limited.

[0106] In an optional embodiment, a tag calculation module and a tag generation module may be built in and defined in the data archive device 3 .

[0107] The marking operation module refers to: defining the marking operation algorithm, defining the algorithm function according to actual needs, and calculating the subordinate factors in the preset contract table according to the algorithm. The calculated results are uniformly represented by 0 for valid and 1 for invalid. The so-called valid means that the corresponding validity judgment data result meets the validity judgment condition, otherwise it is invalid.

[0108] In the above Table 2, the data involved in the effectiveness judgment are: theoretical ammonia nitrogen concentration value (CNH4), water tank high liquid level state duration value (A), ambient temperature value (e01011), and alarm state (i12031).

[0109] Among them, the logic of the theoretical ammonia nitrogen concentration value marking operation module is: if the theoretical ammonia nitrogen concentration value = ammonia nitrogen concentration value (W21003), it is valid, and the output value YS1=0; if the theoretical ammonia nitrogen concentration value ≠ ammonia nitrogen concentration value (W21003), it is invalid, and the output value YS1=1.

[0110] The logic of the water tank high liquid level state continuous value mark operation module: if the water tank high liquid level state continuous value A=0, it is valid and the output value YS2=0; if the water tank high liquid level state continuous value A=1, it is invalid and the output value YS2=1.

[0111] The logic of the ambient temperature value marking operation module: Different data sources are distinguished according to the system SN code (i13097), and the ambient temperature threshold is set according to the physical space distance where the system is located, and the ambient temperature values ​​(e01011) of different systems in the acquired area are compared with the threshold. If it exceeds the threshold range, the file target factor corresponding to the system is invalid, and the output value YS3=1; if it is within the threshold range, the file target factor is valid, and the output value YS3=0.

[0112] Alarm status mark operation module logic: If the value of the alarm status (i12031) is equal to 00, the target factor is valid and the output value YS4=0; otherwise, the target factor is invalid and the output value YS4=1.

[0113] The tag generation module means that users can customize logical operations, and according to the results of all tag operation modules, all results are aggregated into one or several results through logical operators. These results will be arranged in ascending order of module name after the data tag as a new data tag, separated by ".". The result is expressed as 0 or 1. 0 means valid, and 1 means invalid.

[0114] In the above Table 2, the logical operation in the tag generation module 1 is defined as: SC1=YS1||YS2||YS3, when SC1=0, it is valid, and 1 is invalid. The logical operation in the tag generation module 2 is defined as: SC2=YS4, when SC2=0, it is valid, and 1 is invalid.

[0115] Optionally, the data archive device 3 can convert the above logic operation results into data tags, which are used as supplements to the existing tags. In the environmental protection HJ212 reporting protocol, it is required that when the data is normal, the tag is N.

[0116] Based on this, the mark in this technical solution sets the data mark SJBJ to: NXX. Among them, N is the existing mark, and the subsequent content is the implementation content of this solution. X represents the result of the logical operation, and the number of X can be customized and expanded according to user needs and algorithm classification. In Table 2, there are 2 Xs, namely SC1 and SC2. If SC1=SC2=1, the data mark SJBJ is: N.1.1. That is to say, in an optional embodiment, the format of the validity mark includes: the first data bit determined according to the environmental information reporting protocol and other data bits, wherein the other data bits are used to store the output value of the mark generation algorithm during the operation process.

[0117] 3. For the archiving process and archival data:

[0118] Please refer to Table 3, which shows a data archive example table. The data archive example table may include archive code, data tag and filing information. In an optional embodiment, the data archive device 3 can generate a unique archive code according to the start and end time of the data production process. The archive code is related to the data, the data production process, and the system SN code.

[0119] Specifically, the method for generating the archive code may include: when the process signal (i12088) is monitored to change from idle (00) to in progress (01), the system time T1=20230815120000 is recorded; then, when the process signal (i12088) is monitored to change from in progress (01) to idle (00), the system time T2=20230815140000 is recorded; the system SN code, i13097=000001, is obtained, and the ammonia nitrogen concentration value W21003 is obtained; then, the content is integrated into the archive code DABM=T1.T2.000001.W21003 (that is: DABM=20230815120000.20230815140000.000001.W21003). In this way, the time information during the calculation process of the validity mark of the archive target factor can be determined according to the system process operation factor in the subordinate factor in the preset contract table after the assignment.

[0120] Table 3 Data archive example table

[0121] The archive code generation method in the data archive device 3 is not unique, and it can achieve the corresponding relationship between recording production processes, systems, and archive target factors to meet the idea of ​​this solution.

[0122] After obtaining the archive code, the data archive device 3 can archive the archive code and the filing information for future reference. When querying, a query instruction is sent to the data archive device 3, and the instruction includes the archive code. After receiving the instruction, the data archive device 3 sends the corresponding archive code and filing information to the requesting device according to the instruction to complete the data transmission.

[0123] The filing information may include: the codes and / or factors defined in the preset contract table and their corresponding values, the algorithm calculation result variables in the data processing table and their corresponding values ​​and generation time, as well as data tags.

[0124] In Table 3, if the system with system SN code 000001 runs a data production process from 12:00 on August 15, 23 to 14:00 on August 15, 23, and produces 3 data, namely ammonia nitrogen concentration value (factor code: W21003), total phosphorus concentration value (no factor coding), total nitrogen concentration value (no factor coding), and ammonia nitrogen concentration value is the archive target factor, then the archive code is 20230815120000.20230815140000.000001.W21003.

[0125] The data tags in Table 3 can be generated by the data processing table in Table 2 and the corresponding algorithm process, that is, the above-mentioned data tag SJBJ. The tag generation algorithm in Table 2 is run in the production process, and the data tag N.1.1 is finally generated. At this time, the data tag can be used to determine that the data is invalid. And the data file content corresponding to the ammonia nitrogen concentration value is shown in Table 3. During actual transmission, the transmission content can be cut as needed.

[0126] The data file device 3 can convert the file information into a code that can be interpreted by the platform device 4 according to the communication protocol, and transmit the code through 4G wireless, wired network, etc. In an optional embodiment, when the data file is transmitted to the platform device 4, a hexadecimal ASCII code transmission method can be used. The transmission protocol is already a mature technology and will not be explained in detail here. The idea of ​​this solution is satisfied if the information transmission can be realized.

[0127] After the platform device 4 obtains the content of the data archive, there is no need to perform data cleaning or invalid data elimination. It only needs to determine the data tag and then use the data information in the archive.

[0128] Among them, the work of data cleaning and invalid data elimination is processed by the functions defined in the data preprocessing module and the tag generation function defined in the data tag generation module described above. By this means, the workload of data screening and elimination of the platform device 4 can be reduced, and the data processing can be flexibly adjusted in the actual application scenario without affecting each other, thereby reducing coupling, improving accuracy, and preventing the unreasonable phenomenon of "one size fits all" when processing and eliminating data. Specifically, any one of the data preprocessing algorithm, the tag operation algorithm, and the tag generation algorithm can be modified by a modification instruction. The modification instruction can be a manual instruction of the system administrator or a signal automatically triggered by a preset rule, which is not limited to this.

[0129] In combination with the above embodiments, please refer to Figure 3 , Figure 3 A flow chart of another method for recording the validity of environmental monitoring data provided by an embodiment of the present invention. The main implementation method of this technical solution is: 1. Obtain information provided by each monitoring system 1; 2. Establish a preset contract table, and obtain factor information according to the preset contract table to correspond parameter variables of different manufacturers and different systems to the existing factors / codes in the system one by one, so as to achieve standardization and unification of acquired data. 3. Obtain the monitoring time, monitoring data, parameter value, and state value of one or more systems. 4. Define algorithm submodules, each submodule corresponds to a validity judgment concern point, and the result of the concern point is quantified as 0 and 1, and the specific meanings represented by 0 and 1 vary according to the different concern points. Define a logical operation module, which is used to finally determine whether the data is valid and generate a data tag. The data tag indicates whether the corresponding monitoring data is valid and available. Then, the data in the preset contract table is processed, including data preprocessing, tag operation and tag generation. 5. Generate a data archive code according to the system information, establish a data archive, including archive code, data tag and file entry information, so as to integrate the file entry information and data tag with the archive code into a data archive and store it. 6. Transmit all information in the data file through the communication protocol for the recipient to use.

[0130] In order to execute the corresponding steps in the above embodiments and various possible methods, the following is an implementation method of an environmental monitoring data validity judgment device based on data marking, please refer to Figure 4 , Figure 4 The functional module diagram of a device for determining the validity of environmental monitoring data based on data labeling provided by an embodiment of the present invention is shown. The device 300 can be used to implement the above Figure 2 The method embodiment shown.

[0131] It should be noted that the basic principle and technical effect of the device 300 provided in this embodiment are the same as those of the above method embodiment. For the sake of brief description, for the parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiment. The device 300 may include: an acquisition module 310 and a processing module 320.

[0132] Optionally, the above modules can be stored in the memory in the form of software or firmware or fixed in the memory provided by the present invention. Figure 1 The device shown (for example, the data acquisition device 2 or the data archive device 3) is in the operating system (Operating System, OS), and can be Figure 1 The processor in the device shown in the figure executes. Meanwhile, the data, program codes, etc. required to execute the above modules can be stored in the memory.

[0133] It is understandable that the acquisition module 310 and the processing module 320 can be used to support Figure 1 The device shown in the figure performs the relevant steps in the above method embodiment, and / or other processes used for the technology described in this article, such as the above Figure 2 The method embodiments shown and the various method embodiments described above are not limited to this.

[0134] Based on the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method embodiment are executed.

[0135] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, it can execute the method in the above embodiment, thereby solving the problems pointed out in the background technology, and can improve the ability to judge the validity of environmental monitoring data. It is more flexible, timely, and time-space specific. The data processing process is easier to unify on each independent device, and the amount of calculation and resource occupancy are lower.

[0136] Those skilled in the art will appreciate that, for clarity of description, the division of the above-mentioned devices and functions is illustrated by way of examples, and in practical applications, the above-mentioned devices can be integrated into one device through software or hardware as needed; they can exist separately in physical form through each device, or they can be implemented in the form of software functional units.

[0137] Those skilled in the art will appreciate that the method steps described in conjunction with the embodiments disclosed herein can be implemented in hardware or a combination of software and hardware. The implementation of these functions depends on the application scenario and technical constraints of the technical solution. Professional and technical personnel can use different methods in different application scenarios to implement the described functions, but such implementation should not be considered to be beyond the scope of the present invention. The names of the various devices or functions are only for the convenience of distinguishing each other and are not intended to limit the scope of protection of this application.

[0138] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for judging the validity of environmental monitoring data based on data labeling, characterized in that: The method comprises the following steps: Acquire environmental monitoring data collected by a monitoring device, wherein the storage format of the environmental monitoring data is determined by the collection format of the monitoring device; Extracting a plurality of data to be archived from the environmental monitoring data according to the acquisition format; For the plurality of data to be archived, the data to be archived that matches the objects to be collected in the preset contract table are determined as the data to be standardized; wherein the preset contract table includes a plurality of objects to be collected, and the plurality of objects to be collected include a system code, an archive target factor, and a subordinate factor, wherein the system code is used to represent an identification code of the monitoring device, the archive target factor is used to represent the target monitoring data that needs to be judged for validity in the environmental monitoring data, and the subordinate factor is used to represent the environmental monitoring data that needs to be used in the process of judging whether the target monitoring data is valid; According to the predetermined storage format corresponding to the object to be collected, the format of the data to be standardized is converted to obtain standard monitoring data corresponding to the data to be standardized; According to the standard monitoring data, the corresponding objects to be collected in the preset contract table are assigned values ​​to obtain the preset contract table after the assignment; Calculate the validity mark of the archive target factor in the preset contract table after the assignment according to the data validity calculation rule and the value of the subordinate factor in the preset contract table after the assignment; wherein the validity mark is used to indicate whether the archive target factor is valid; Performing data archiving processing on the assigned preset contract table according to the validity mark to obtain archive data; wherein the archive data includes information for indicating the validity mark; The archive data is stored.

2. The method for determining the validity of environmental monitoring data based on data labeling according to claim 1, characterized in that: The data validity calculation rules include a data preprocessing algorithm, a marking operation algorithm and a marking generation algorithm; The data preprocessing algorithm is used to calculate the auxiliary monitoring data required in the process of judging the validity of the target monitoring data according to the value of the subordinate factor in the preset contract table after the assignment; The marking operation algorithm is used to calculate the conditional operation input value in the conditional operation process of judging the validity of the target monitoring data according to the auxiliary monitoring data and the value of the subordinate factor in the preset contract table after the assignment; The tag generation algorithm is used to calculate the validity tag of the archive target factor in the preset contract table after the assignment according to the conditional operation input value and the preset validity judgment conditional operation rule.

3. The method for determining the validity of environmental monitoring data based on data labeling according to claim 2, characterized in that: The input values, output values ​​and algorithm rules of the data preprocessing algorithm, the mark calculation algorithm and the mark generation algorithm during the calculation process are stored in the form of a preset data processing table.

4. The method for determining the validity of environmental monitoring data based on data labeling according to claim 3 is characterized in that: The file data includes file code, data mark and filing information; The file code includes: information of the identification code of the monitoring device, time information of the operation process of the validity mark of the file target factor, and the value of the file target factor; The data mark includes: information for indicating the validity mark; The file entry information includes: at least one item of information in the preset contract table after assignment, and at least one item of information in the preset data processing table.

5. The method for determining the validity of environmental monitoring data based on data labeling according to claim 4 is characterized in that: The time information during the calculation process of the validity mark of the archive target factor is determined according to the system process operation factor in the subordinate factor in the preset contract table after the assignment.

6. The method for determining the validity of environmental monitoring data based on data labeling according to claim 2, characterized in that: The format of the validity mark includes: a first data bit and other data bits determined according to an environmental information reporting protocol, wherein the other data bits are used to store output values ​​of the mark generation algorithm during the operation process.

7. The method for determining the validity of environmental monitoring data based on data labeling according to claim 2, characterized in that: Any one of the data preprocessing algorithm, the labeling operation algorithm and the labeling generation algorithm may be modified by a modification indication.

8. The method for determining the validity of environmental monitoring data based on data labeling according to claim 1, characterized in that: The number of the preset contract tables includes a plurality, and the archive target factors in different preset contract tables among the plurality of preset contract tables are different.

9. A device for judging the validity of environmental monitoring data based on data labeling, characterized in that: The method comprises means for implementing the method as claimed in any one of claims 1 to 8.

10. A system for judging the validity of environmental monitoring data based on data labeling, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the system performs the method according to any one of claims 1 to 8.

Images