Time series data quality classification system based on big data real-time engine

Abstract

The invention relates to a time series data quality classification system based on a big data real-time engine, and the system is used for obtaining time series data of a measurement point, and inputting the time series data into the real-time engine; obtaining time sequence data of the measuring points, and inputting the time sequence data into a real-time engine; and respectively routing the time sequence data obtained by each data filter to different time sequence databases through a data quality coprocessor. The method has the beneficial effects that the time sequence data is subjected to time sequence data quality processing through the rule freely customized by the business party, the data is divided into abnormal data, doubt data and reliable data, and the time sequence database is automatically subjected to database division and layering operation through an agent mode, so that the credibility of the data is improved.

Description

Technical Field

[0001] The present invention relates to the field of time series data classification, and more specifically, to a time series data quality classification system based on a big data real-time engine. Background Art

[0002] Existing time series databases, including Apache IOTDB, InfluxDB, and OSIsoft, all have good storage and query performance in data storage and have been widely used in many application scenarios. Traditional data is written by calling the time series database module. When abnormalities occur in the written data, including extreme abnormal values, timeout writes, bad values, etc., no distinction is made. Problems can only be discovered through post-statistical analysis and statistics of abnormal values.

[0003] However, the current mainstream time series databases have the following defects: 1. Functional deficiencies. Current time series database products have obvious deficiencies in data quality processing functions and are unable to discover and report abnormal data in real time. As a result, when business parties use data for business processing again, they can only discover and correct data problems in extreme cases.

[0004] 2. Insufficient performance: Commercial products such as OSIsoft's PI system, as a leader in the field, need to manually set thresholds to perform statistical queries on the data in the time series database when processing problematic data. When the amount of data is large, it takes a long time to calculate.

[0005] 3. Insufficient processing capabilities: When data quality problems occur, the business system is usually required to delete and reorganize the abnormal data in the database, and then the business system recalculates and processes it, which has poor timeliness. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art and to propose a time series data quality classification system based on a big data real-time engine.

[0007] In a first aspect, a time series data quality classification method based on a big data real-time engine is provided, comprising: Step 1, obtaining time series data of the measuring point, and inputting the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter and a reliable data filter; Step 2: setting corresponding data quality rules in each data filter in the real-time engine, and receiving corresponding time series data according to the data quality rules; Step 3: The time series data obtained by each data filter is routed to different time series databases through the data quality coprocessor.

[0008] Preferably, step 2 comprises: Step 2.1: According to the data quality rules, the real-time engine loads the custom data quality function and compiles it; Step 2.2, passing the measurement point name, time, and value of the time series data as parameters into the custom data quality function for calculation to obtain a verification result; the verification result is the quality type of the time series data, and the quality type includes abnormal data, questionable data, and reliable data; Step 2.3: According to the verification result, the time series data is input into the corresponding data filter.

[0009] Preferably, in step 2.1, the custom data quality function includes an over-limit function and a timeout determination function; The over-limit function defines the upper limit and lower limit of the time series data. If the value of the time series data exceeds the upper limit, the verification result of the time series data is abnormal data. If the value of the time series data is between the upper limit and the lower limit, the verification result of the time series data is questionable data. The timeout determination function defines an arrival duration threshold of the time series data. If the arrival duration of the time series data exceeds the duration threshold, the verification result of the time series data is abnormal data.

[0010] Preferably, in step 2.1, the custom data quality function also includes a statistical judgment function; the statistical judgment function defines reference values ​​of the maximum value, minimum value, discrete rate and data density of the time series data. If the deviation of any actual value of the maximum value, minimum value, discrete rate and data density of the time series data from the reference value of the maximum value, minimum value, discrete rate and data density exceeds a preset threshold, the verification result of the time series data is questionable data.

[0011] Preferably, step 3 further includes: querying the time series data in the time series database through a data quality coprocessor.

[0012] In a second aspect, a time series data quality classification system based on a big data real-time engine is provided, which is used to execute any method described in the first aspect, including: An acquisition module, used for acquiring time series data of a measuring point and inputting the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter and a reliable data filter; A receiving module, used to set corresponding data quality rules in each data filter in the real-time engine, and receive corresponding time series data according to the data quality rules; The routing module is used to route the time series data obtained by each data filter to different time series databases through the data quality coprocessor.

[0013] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0014] In a fourth aspect, an electronic device is provided, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement any method as described in the first aspect.

[0015] The beneficial effects of the present invention are: 1. The present invention processes the time series data quality through rules freely customized by the business party, divides the data into abnormal data, questionable data and reliable data, and automatically divides the time series database into different databases and layers through agents to improve the credibility of the data. For abnormal data and questionable data, since the data set is small, statistical query can be convenient, and it is convenient for later data correction and data quality improvement.

[0016] 2. The present invention uses a customized time series data quality judgment function to combine time series data and rules as parameters with indicator expressions to verify quality, thereby solving the problem of flexible and changeable quality rule definition for business.

[0017] 3. The present invention solves the problem of quality verification of massive time series data by utilizing the batch integration, scalability and precise consistency capabilities of the big data distributed processing engine, and by integrating the time series data quality rule flow with the time series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of auxiliary processing of time series database data quality provided by an embodiment of the present invention; Figure 2 A functional calculation flow chart of time series data quality rules provided by an embodiment of the present invention; Figure 3 A flow chart of data quality processing of flink internal time series data provided by an embodiment of the present invention; Figure 4 A flowchart of a data quality coprocessor for time series data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0020] Embodiment 1: To solve the problems of the prior art, such as Figure 1 As shown, Embodiment 1 of the present application provides a time series data quality classification method based on a big data real-time engine, including: Step 1: Obtain time series data of the measuring points, and input the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter, and a reliable data filter.

[0021] Specifically, the real-time engine uses the stream processing engine flink. Flink is used to perform stateful computing on unbounded and bounded data streams. It has the characteristics of batch-stream integration, precise state management, event time support, and exactly-once state consistency guarantee. At the same time, it can still maintain high throughput and low latency under the premise of massive data. The time series data processing program developed based on its framework has powerful, accurate, and scalable data processing capabilities. Time series data can be regarded as an unbounded data stream. Flink is used to merge the rule stream and the data stream, process data quality rules, and implement abnormal data filters, suspicious data filters, and reliable data filters.

[0022] Step 2: Set corresponding data quality rules in each data filter in the real-time engine, and receive corresponding time series data according to the data quality rules.

[0023] like Figure 2 As shown, step 2 includes: Step 2.1: According to the data quality rules, the real-time engine loads the custom data quality function and compiles it.

[0024] For example, the data quality rules may refer to the measurement point rule table shown in Table 1 and the measurement point statistics table shown in Table 2.

[0025] Table 1

[0026] Table 2

[0027] The measurement point rule table is used by the business side to classify data by configuring measurement point business rules, including configurable rules such as greater than, less than, and range for differentiation. The timeout data quality type mainly verifies the real-time arrival of data, and distinguishes the reliability of data by comparing the difference between the current time and the data arrival time with the data in the rule table.

[0028] The measurement point statistics table mainly screens data from the perspective of historical sample data (reliable time series data). Historical sample data is data that has been judged by the business and is highly reliable. The time series data entering the processing engine is compared with the measurement point statistics rules, including but not limited to the maximum value, minimum value, discrete rate, data density and other statistical rules. The rule types in the measurement point statistics table can be added or deleted according to business needs, which can better adapt to business needs.

[0029] According to the above data quality rules, this application functionalizes the rules and adopts the Java expression engine MVE1 as the data quality function to determine the quality of time series data. Through the custom data quality function, the structured time series data including the measurement point, time and value are passed into the custom data quality function as parameters for calculation and output.

[0030] Specifically, the custom data quality functions include the over-limit function TSOutOfRangeJudge (measurement point name, upper limit value, lower limit value, quality type) and the timeout judgment function TSOutOfTimeJudge (measurement point name, duration, quality type).

[0031] The over-limit function defines the upper and lower limits of the time series data. If the value of the time series data exceeds the upper limit, the verification result of the time series data is abnormal data. If the value of the time series data is between the upper and lower limits, the verification result of the time series data is questionable data. The timeout judgment function defines the arrival duration threshold of the time series data. If the arrival duration of the time series data exceeds the arrival duration threshold, the verification result of the time series data is abnormal data. It should be noted that if the arrival duration of the time series data does not exceed the arrival duration threshold, the time series data continues to participate in the screening of other data quality rules.

[0032] In addition, the custom data quality function also includes a statistical judgment function MAXValJudge (measurement point name, rule data, quality type); the statistical judgment function defines the reference values ​​of the maximum value, minimum value, discrete rate and data density of the time series data. If the deviation of any actual value of the maximum value, minimum value, discrete rate and data density of the time series data from the reference value of the maximum value, minimum value, discrete rate and data density exceeds a preset threshold, the verification result of the time series data is questionable data.

[0033] Step 2.2: pass the measurement point name, time, and value of the time series data as parameters into the custom data quality function for calculation to obtain the verification result; the verification result is the quality type of the time series data, and the quality type includes abnormal data, questionable data, and reliable data.

[0034] Specifically, this application uses the Mvel expression parser as a data quality filtering rule operator, and functionalizes the data quality rules through a custom function method. The value or time of the time series data is placed in the expression as a parameter to perform calculations to obtain the data quality result.

[0035] Step 2.3: According to the verification result, the time series data is input into the corresponding data filter.

[0036] Step 3: The time series data obtained by each data filter is routed to different time series databases through the data quality coprocessor.

[0037] Embodiment 2: Based on Example 1, Example 2 of the present application provides a more specific time series data quality classification method based on a big data real-time engine, including: Step 1: Obtain time series data of the measuring points, and input the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter, and a reliable data filter.

[0038] Step 2: Set corresponding data quality rules in each data filter in the real-time engine, and receive corresponding time series data according to the data quality rules.

[0039] In step 2, the processing operation of the real-time engine such as Flink processing engine is as follows Figure 3 As shown in the figure, the time series data stream and the rule-defined data stream are intersected in Flink through a custom multi-stream fusion processing function to associate the rules with the time series data.

[0040] When the Flink program is started, the quality rule function will be loaded. When the data quality rules intersect with the time series data, the functional processing and calculation process of the time series data quality rule is completed here. The time series data includes the measurement point name, measurement point value, time and upper and lower limits defined by the quality rule, timeout and corresponding statistical dimension information, and rule level.

[0041] Step 3: The time series data obtained by each data filter is routed to different time series databases through the data quality coprocessor.

[0042] In step 3, the present application proposes a hierarchical library table approach to solve the data quality classification problem. The time series data of different quality types are placed in different library tables through the data quality coprocessor, and the data confirmed by the business party to be correct can be archived through the data quality merge archiving strategy.

[0043] Specifically, Figure 4As shown in the figure, the data quality coprocessor writes the time series data labeled by Flink streaming computing into the abnormal data, questionable data and reliable databases.

[0044] In addition, as the proxy layer of the time series database, the data quality coprocessor also undertakes the query function of the time series database. Users determine the query scope according to their needs and call the data by passing parameters. For {"abnormal", "doubtful", "reliable"}, the data quality coprocessor will query in different data quality tables. The benefit of doing so is that on the one hand, the reliability of data business data is guaranteed, especially in applications with high data requirements at the algorithm level. On the other hand, by archiving and classifying abnormal data and doubtful data, the abnormal data situation can be better statistically analyzed and the data link can be improved.

[0045] The data quality coprocessor also undertakes the function of data merging. When the questionable data is confirmed by the business party, it will be transferred to the abnormal data or reliable database and provided as sample data to the quality rule setter for reference.

[0046] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0047] Embodiment 3: Based on Examples 1 and 2, Example 3 of the present application provides a time series data quality classification system based on a big data real-time engine, including: An acquisition module, used for acquiring time series data of a measuring point and inputting the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter and a reliable data filter; A receiving module, used to set corresponding data quality rules in each data filter in the real-time engine, and receive corresponding time series data according to the data quality rules; The routing module is used to route the time series data obtained by each data filter to different time series databases through the data quality coprocessor.

[0048] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A time series data quality classification method based on a big data real-time engine, characterized in that: include: Step 1: Obtain time series data of the measuring points and input the time series data into the real-time engine; The real-time engine includes an abnormal data filter, a doubtful data filter and a reliable data filter; Step 2: setting corresponding data quality rules in each data filter in the real-time engine, and receiving corresponding time series data according to the data quality rules; Step 3: The time series data obtained by each data filter is routed to different time series databases through the data quality coprocessor.

2. The time series data quality classification method based on big data real-time engine according to claim 1 is characterized in that: Step 2 includes: Step 2.1: According to the data quality rules, the real-time engine loads the custom data quality function and compiles it; Step 2.2, passing the measurement point name, time, and value of the time series data as parameters into the custom data quality function for calculation to obtain a verification result; the verification result is the quality type of the time series data, and the quality type includes abnormal data, questionable data, and reliable data; Step 2.3: According to the verification result, the time series data is input into the corresponding data filter.

3. The time series data quality classification method based on big data real-time engine according to claim 2 is characterized in that: In step 2.1, the custom data quality function includes an over-limit function and a timeout determination function; The over-limit function defines the upper limit and lower limit of the time series data. If the value of the time series data exceeds the upper limit, the verification result of the time series data is abnormal data. If the value of the time series data is between the upper limit and the lower limit, the verification result of the time series data is questionable data. The timeout determination function defines an arrival time threshold of the time series data. If the arrival time of the time series data exceeds the arrival time threshold, the verification result of the time series data is abnormal data.

4. The time series data quality classification method based on big data real-time engine according to claim 3 is characterized in that: In step 2.1, the custom data quality function also includes a statistical judgment function; the statistical judgment function defines the reference values ​​of the maximum value, minimum value, discrete rate and data density of the time series data. If the deviation of any actual value of the maximum value, minimum value, discrete rate and data density of the time series data from the reference value of the maximum value, minimum value, discrete rate and data density exceeds a preset threshold, the verification result of the time series data is questionable data.

5. The time series data quality classification method based on big data real-time engine according to claim 4 is characterized in that: Step 3 also includes: querying the time series data in the time series database through a data quality coprocessor.

6. A time series data quality classification system based on a big data real-time engine, characterized in that: The method for executing any one of claims 1 to 5 comprises: An acquisition module, used for acquiring time series data of a measuring point and inputting the time series data into a real-time engine; the real-time engine includes an abnormal data filter, a doubtful data filter and a reliable data filter; A receiving module, used to set corresponding data quality rules in each data filter in the real-time engine, and receive corresponding time series data according to the data quality rules; The routing module is used to route the time series data obtained by each data filter to different time series databases through the data quality coprocessor.

7. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 5.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 5.

Images