An energy equipment streaming data quality detection method based on Flinkx

Abstract

The application is suitable for the field of Flinkx data synchronization, and provides an energy equipment stream data quality detection method based on Flinkx. The method initializes a Flinkxjob task first, constructs a source reading end to read to-be-inspected data, then creates a rule engine processing end to process the to-be-inspected data, and then writes the check result stream data and the statistical result stream data through a sink writing end. The application is based on a Flinkx stream batch integrated computing framework, efficiently processes real-time stream data of energy management and control, guarantees high throughput, low delay and consistency of data, provides a checkpoint mechanism based on Flinkx, realizes power-off data transmission, greatly improves data processing efficiency, avoids data repeated processing, and adopts a Rete algorithm forward reasoning to quickly check the to-be-inspected data in the rule engine processing end, guarantees the stability of the check under high throughput and low delay.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of Flinkx data synchronization, and particularly relates to an energy equipment stream data quality detection method based on Flinkx. BACKGROUND

[0002] With the acceleration of urbanization and the development of information management, energy consumption is rapidly increasing, and various energy consumption data are also showing a rapid growth trend. How to real-time quality inspect the accuracy and real-time performance of massive energy consumption big data, analyze massive energy consumption data for energy consumption units to better grasp the energy consumption law, and provide important data protection.

[0003] In the prior art, two schemes are generally used for data quality. One scheme is to obtain the to-be-inspected data for quality inspection in a Jdbc SQL mode, and this scheme can only process million-level data and is suitable for small data quality inspection. The other scheme is to obtain the to-be-inspected data for quality inspection in a Spark SQL method, which is a relatively mature scheme in the field of big data, but has high delay and is limited by storage media, and the to-be-inspected data must be stored in a hive library, so it is only suitable for offline data quality inspection. SUMMARY

[0004] In view of the above problems, the purpose of the present application is to provide an energy equipment stream data quality detection method based on Flinkx, which aims to solve the technical problems of small data volume and low efficiency of the existing data quality inspection scheme.

[0005] The present application adopts the following technical scheme:

[0006] The energy equipment stream data quality detection method based on Flinkx comprises the following steps:

[0007] Step S1, creating a Flinkx job task and initializing task configuration;

[0008] Step S2, creating a source reading end, reading the to-be-inspected data of the energy equipment from the kafka message queue through the source reading end;

[0009] Step S3, creating a ruleSolve rule engine processing end, performing quality verification on the to-be-inspected data according to the pre-set quality inspection rules and outputting a verification result stream and a statistical result stream;

[0010] Step S4, creating two sink writing ends, one sink writing end receives the verification result stream and writes it into a storage medium, and the other sink writing end receives the statistical result stream and writes it into a storage medium.

[0011] Further, the step S1 has the following specific process: a Flinkx job task is created, a reader plug-in detailed configuration, a writer plug-in detailed configuration and setting information are obtained by parsing a Flinkx Job script, and then task configuration is initialized, wherein the initialization of the task configuration includes starting a checkpoint mechanism, configuring a split key, configuring a readerChannel concurrency degree and configuring a writerChannel concurrency degree.

[0012] When the checkpoint mechanism is started, the Flinkx job task realizes breakpoint continuation through the checkpoint mechanism, and specifically, the source reading end is notified of the offset of the to-be-inspected data in the kafka message queue last time to realize breakpoint continuation next time.

[0013] When the readerChannel concurrency degree is configured, the Flinkx job task triggers multiple source reading ends to read the to-be-inspected data in parallel according to the readerChannel concurrency degree value.

[0014] When the writerChannel concurrency degree is configured, the Flinkx job task triggers multiple sink writing ends to write the verification result stream and the statistical result stream in parallel according to the writerChannel concurrency degree value.

[0015] Further, the step S2 has the following specific process:

[0016] According to the reader plug-in detailed configuration information, source library connection information is obtained, and a source reading end is created based on a SourceFactory abstract class provided by Flinkx, and the to-be-inspected data of the energy equipment is read from the kafka message queue according to the set initialization task configuration and the kafka consumer end start mode.

[0017] If the checkpoint mechanism is started, the last reading data offset of the source reading end in the kafka message queue is recorded, and the source reading end reads the to-be-inspected data from the last reading data offset position next time.

[0018] The kafka consumer end start mode includes a group-offsets mode, an earliest-offset mode, a latest-offset mode, a timestamp mode and a specific-offsets mode.

[0019] Further, in the step S3, the quality inspection rule of the ruleSolve rule engine processing end includes a virtual field rule and a quality inspection rule. The virtual field rule is a rule that has not yet bound the field of the data to be inspected. The quality inspection rule is a rule that has bound the field of the data to be inspected. The ruleSolve rule engine processing end constructs a Rete network according to the quality inspection rule, quickly checks the data to be inspected through a Rete algorithm forward reasoning, and outputs a checking result stream and a statistical result stream to the sink writing end.

[0020] The checking result stream includes correct and incorrect data, and the statistical result stream includes the total number of data to be checked, the total number of rules to be checked, and the total number of incorrect data of each rule.

[0021] Further, the step S4 specifically includes the following processes.

[0022] S41, the writer plug-in detailed configuration in the Flinkx Job script establishes a HIVE partition table, obtains a storage path in a HDFS distributed file system, and creates two sink writing ends.

[0023] S42, when the checking result stream and the statistical result stream output by the rule engine processing end are received, the two sink writing ends write the received result stream data into the storage path corresponding to the Hdfs distributed file system and create a Hive file index.

[0024] S43, after the writing is completed, the sink writing end outputs a termination task instruction. When the flinkx job task receives the termination task instruction, the flinkx job task is terminated, and the task execution is completed.

[0025] The beneficial effects of the present application are: the present application provides an energy equipment streaming data quality detection method based on Flinkx, first, initialize the Flinkx job task, and build the source reading end to read the data to be detected, then create a rule engine processing end to process the data to be detected, and then write the check result stream data and statistical result stream data through the sink writing end Stream data quality link; the present application is based on the Flinkx stream batch integrated computing framework, and the energy management and control real-time stream data is efficiently processed, the data high throughput, low delay and consistency are guaranteed; and based on the Flinkx checkpoint mechanism, the data power failure transmission is realized, the data processing efficiency is greatly improved, and the data repeated processing is avoided; in addition, the rule engine processing end of the present application adopts the Rete algorithm forward reasoning to quickly check the data to be detected, and guarantees the stability of the data under high throughput and low delay. In short, the method of the present application greatly improves the real-time data quality detection efficiency, can also support the quality detection of heterogeneous data sources, and effectively reduces the manpower and time cost required for real-time data quality detection. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flow chart of the energy equipment streaming data quality detection method based on Flinkx provided by the embodiment of the present application;

[0027] Figure 2 is a virtual field rule setting interface schematic diagram provided by the embodiment of the present application;

[0028] Figure 3 is a quality detection rule setting interface schematic diagram provided by the embodiment of the present application;

[0029] Figure 4 is a rule engine execution flow chart provided by the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0031] In order to explain the technical scheme described in the present application, the following specific embodiments are used to explain.

[0032] Figure 1 The flow of the energy equipment streaming data quality detection method based on Flinkx provided by the embodiment of the present application is shown, and only the part related to the embodiment of the present application is shown for convenience of description.

[0033] As Figure 1As shown, the energy equipment streaming data quality detection method based on Flinkx provided by the embodiment includes the following steps:

[0034] Step S1, initialization step: create a Flinkx job task and perform initialization task configuration.

[0035] When creating the Flinkx job task, first parse the Flinkx Job script to obtain reader plug-in detailed configuration, writer plug-in detailed configuration, and setting information, and then initialize the task configuration, that is, initialize the task environment. The reader plug-in detailed configuration and the writer plug-in detailed configuration both include source database basic information, table information, field information, etc. The setting information includes task type and breakpoint continuation configuration, split key, etc.

[0036] In this step, the initialization task configuration includes starting the checkpoint mechanism, configuring the split key, configuring the readerChannel concurrency, and configuring the writerChannel concurrency.

[0037] If the checkpoint mechanism is configured, the Flinkx job task realizes breakpoint continuation through the checkpoint mechanism, notifies the next task source reading end of the offset of the to-be-inspected data in the kafka message queue on the source reading end last time to realize power failure continuation. That is, the Flinkx Job listens to the file in the checkpoint directory on Hdfs to obtain the last kafka to-be-inspected data offset on the source reading end, and the source reading end obtains the to-be-inspected data from the last offset position when the next task.

[0038] If the readerChannel concurrency is configured, the Flinkx Job task triggers multiple source reading ends to read the to-be-inspected data from the kafka message queue in parallel according to the size of the readerChannel concurrency value.

[0039] If the writerChannel concurrency is configured, the Flinkx Job task triggers multiple sink writing ends to write the verification result stream and the statistical result stream in parallel according to the size of the writerChannel concurrency value. Specifically, multiple sink writing ends can be instantiated to write the verification result stream and the statistical result stream into the Hive database.

[0040] Step S2, source reading end reading step: create a source reading end, and read the to-be-inspected data of the energy equipment from the kafka message queue through the source reading end.

[0041] This step uses the source reading end to read the data to be inspected from the kafka message queue; the data to be inspected is synchronized to the Kafka message queue by the energy equipment (such as one of Mysql, Oracle, Hive, etc.).

[0042] The specific operation is as follows: according to the reader plug-in detailed configuration information, the source library connection information is obtained, such as kafka connection configuration, offset, kafka consumer start mode, etc.; and based on the SourceFactory abstract class provided by Flinkx, a source reading end is created, and according to the set initialization task configuration and kafka consumer start mode, the energy equipment data to be inspected is read from the kafka message queue.

[0043] For the initialization task configuration, when the checkpoint mechanism is started, the flinkx job starts the breakpoint resume mechanism, records the offset of the last reading of the kafka data on the source reading end, and when the source reading end reads the data to be inspected next time, the data to be inspected is read from the position of the last reading data offset.

[0044] And the kafka consumer start mode has group-offsets mode, earliest-offset mode, latest-offset mode, timestamp mode, specific-offsets mode. 1) When it is group-offsets mode, the Source reading end starts consuming from the offset submitted by the specified consumer group in ZK / Kafka brokers; 2) When it is earliest-offset mode, the Source reading end starts from the earliest offset (if possible); 3) When it is latest-offset mode, the Source reading end starts from the latest offset (if possible); 4) When it is timestamp mode, the Source reading end starts from the specified timestamp of each partition; 5) When it is specific-offsets mode, the Source reading end starts from the specified specific offset of each partition.

[0045] Step S3, create a ruleSolve rule engine processing end, according to the pre-set quality inspection rule, the quality of the data to be inspected is checked and the checking result stream and the statistical result stream are output. The checking result stream contains correct and incorrect data, and the statistical result stream contains the total number of data to be checked, the total number of rules to be checked, and the total number of incorrect data for each rule.

[0046] The quality inspection rule of the ruleSolve rule engine processing end includes a virtual field rule and a quality inspection rule. The virtual field rule is a rule that has not yet bound a field of to-be-inspected data, such as an Id greater than 1 field rule, where Id is a virtual field and id>1 is a quality inspection rule. The rule setting interface is as shown in Figure 2 The quality inspection rule is a rule that has bound a field of to-be-inspected data, such as an Id greater than 1 quality inspection rule, where Id is bound to a database field and the quality inspection rule id>1 is a quality inspection rule. The rule setting interface is as shown in Figure 3 .

[0047] The ruleSolve rule engine processing end implements a custom component based on an interface provided by Flinkx, which is used to process a variety of rule sets, such as null value checking, value range checking, fluctuation checking, and the like.

[0048] As shown in Figure 4 , when the ruleSolve rule engine is created, the rule engine is first initialized, and then a Rete network is constructed according to the quality inspection rule. A WorkingMemory is then created to receive to-be-inspected data output by the source reading end. The to-be-inspected data is quickly checked through forward reasoning of the Rete algorithm, and a check result stream and a statistical result stream are output to the sink writing end. If there is no to-be-inspected data, the source reading end and the sink writing end are closed. If to-be-inspected data is received, a quality inspection rule fact is constructed, and the fact is then inserted into the WorkingMemory. The rule is matched in the Rete network, and a result stream is returned. Finally, the sink writing end writes the result stream to a storage medium.

[0049] The input of the ruleSolve rule engine is two parameters: one is a DataStream, and the other is a ruleMeta. Two streams are output: one is a check result stream, and the other is a statistical result stream. The rule engine checks whether to-be-inspected data is abnormal according to a quality inspection rule. If the data is abnormal, an error result stream and a statistical result stream are output.

[0050] Step S4, two sink writing ends are created. One sink writing end writes a check result stream to a storage medium, and the other sink writing end writes a statistical result stream to a storage medium.

[0051] The specific process of this step is as follows:

[0052] S41, the writer plug-in detailed configuration in the Flinkx Job script establishes a HIVE partition table, obtains a storage path in a HDFS distributed file system, and creates two sink writing ends.

[0053] According to the reading target end data connection information in the writer plug-in detailed configuration, such as: Hive connection address, username, password, etc., based on the SinkFactory abstract class provided by Flinkx, two sink write ends are created, when the target table of Hive does not exist, the HIVE partition table is established first, and the storage path in the HDFS distributed file system is obtained.

[0054] S42, when receiving the verification result stream and the statistical result stream output by the rule engine processing end, the two sink write ends write the received result stream data into the storage path corresponding to the Hdfs distributed file system and create a Hive file index.

[0055] Two sink write ends, one is used for receiving and saving the verification result stream, and the other is used for receiving and saving the statistical result stream. The verification result stream and the statistical result stream output by the rule engine processing end are received by the corresponding sink write end, and the received result stream data is written into the storage path corresponding to the Hdfs distributed file system and a Hive file index is created.

[0056] S43, after writing is completed, the sink write end outputs a termination task instruction, when the flinkx job task receives the termination task instruction, the flinkx job task is terminated, and the task execution is completed.

[0057] In summary, the present application provides an energy equipment stream data quality detection method based on Flinkx, first, a Flinkx Job task is created and started, and the Flinkx start environment is initialized; then a source read end is created, the source read end pulls the data to be detected from the kafka message queue; then a ruleSolve rule engine processing end is created, the ruleSolve rule engine processing end receives the data output by the source read end, processes the data according to the pre-set detection rules, and outputs a verification result stream and a statistical result stream; finally, two write ends are created, which are a verification result write end and a statistical result write end, the error result write end receives the verification result stream to the storage medium; the statistical result write end receives the statistical result stream to the storage medium. This quality detection method can not only process offline data, but also process real-time data, and the data to be checked is not limited to storage media, the data to be checked can be stored in relational databases such as MySQl, Oracle, Dream, Postgress, etc., and can also be stored in Kafka, Hive, Hbase, etc.

[0058] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for quality inspection of streaming data from energy devices based on Flinkx, characterized in that, The method includes the following steps: Step S1: Create a Flinkx job and initialize the task configuration; Step S2: Create a source reader to read the energy equipment's quality inspection data from the Kafka message queue. Step S3: Create a ruleSolve rule engine processing terminal, perform quality verification on the data to be inspected according to the pre-set quality inspection rules, and output the verification result stream and statistical result stream; Step S4: Create two sink write ends. One sink write end receives the verification result stream and writes it to the storage medium, while the other sink write end receives the statistical result stream and writes it to the storage medium. The specific process of step S1 is as follows: Create a Flinkx job task, parse the Flinkx Job script to obtain detailed configuration of the reader plugin, detailed configuration of the writer plugin, and setting information, and then initialize the task configuration, which includes enabling the checkpoint mechanism, configuring the split key, configuring the readerChannel concurrency and configuring the writerChannel concurrency. When the checkpoint mechanism is enabled, the Flinkx job task can resume interrupted transmission through the checkpoint mechanism. Specifically, the offset of the data to be inspected in the Kafka message queue consumed by the source reading end in the last time is notified to the source reading end of the next task to resume transmission after power failure. Once the readerChannel concurrency level is configured, the Flinkx Job task will trigger multiple source readers to concurrently read the data to be inspected based on the readerChannel concurrency level value. After configuring the writerChannel concurrency level, the Flinkx Job task triggers multiple sinks to concurrently write the verification result stream and the statistical result stream according to the writerChannel concurrency level value; In step S3, the quality inspection rules of the ruleSolve rule engine processing end include virtual field rules and quality detection rules. The virtual field rules are rules for fields that have not yet been bound to the data to be inspected, and the quality detection rules are rules for fields that have been bound to the data to be inspected. The ruleSolve rule engine processing end constructs a Rete network based on the quality inspection rules, quickly verifies the data to be inspected through forward inference using the Rete algorithm, and outputs the verification result stream and statistical result stream to the sink writing end. The verification result stream contains both correct and incorrect data, and the statistical result stream contains the total number of data to be checked, the total number of rules to be checked, and the total number of quality inspection errors for each rule.

2. The method for quality inspection of streaming data from energy devices based on Flinkx as described in claim 1, characterized in that, The specific process of step S2 is as follows: Based on the detailed configuration information of the reader plugin, obtain the source library connection information, and create a source reading end based on the SourceFactory abstract class provided by Flinkx. According to the initialization task configuration and Kafka consumer startup mode, read the energy equipment's quality inspection data from the Kafka message queue accordingly. If the checkpoint mechanism is enabled, the offset of the last data read from the Kafka message queue by the source read end is recorded. When the next task is performed, the source read end starts reading the data to be inspected from the position of the last data read offset. Kafka consumer startup modes include group-offsets mode, early-offset mode, latest-offset mode, timestamp mode, and specific-offsets mode.

3. The method for quality inspection of streaming data from energy devices based on Flinkx as described in claim 2, characterized in that, The specific process of step S4 is as follows: The detailed configuration of the writer plugin in the S41 Flinkx Job script establishes the HIVE partition table, obtains the storage path in the HDFS distributed file system, and creates two sink write ends. S42. After receiving the verification result stream and statistical result stream output by the rule engine processing end, the two sink write ends write the received result stream data to the storage path corresponding to the HDFS distributed file system and create a Hive file index. S43. After writing is complete, the sink write end outputs a task termination command. When the FlinkX job receives the task termination command, the FlinkX job terminates and outputs "Task execution completed".

Images