Real-time data quality detection method and system

Abstract

The invention relates to a real-time data quality detection method and system.The method comprises the steps that rule configuration of a user is received, a data quality plug-in is generated according to the rule configuration, the data quality plug-in is integrated in an Flink program, and when the Flink program receives a data stream in real time, the data stream is stored in the Flink program; and obtaining corresponding rule configuration from the data quality plug-in to carry out real-time quality detection on the data streams, obtaining a quality detection result of each data stream, storing the quality detection result in the ClickHouse, obtaining an alarm rule from the rule configuration, and carrying out early warning on the data streams corresponding to the quality detection results according to the alarm rule and the materialized view capability of the ClickHouse. Therefore, the data quality plug-in is integrated into the Flink program in the form of the plug-in, redundant storage does not need to be carried out on the original data stream, the real-time performance of real-time data quality detection is ensured, the real-time data quality detection does not need to be intruded into codes of specific services, the complexity of real-time data quality detection is simplified, meanwhile, the flexibility is improved, and the data quality detection efficiency is improved. And early warning is carried out according to the materialized view capability of the ClickHouse, so that the user experience is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality detection, and in particular to a method and system for real-time data quality detection. Background Art

[0002] Currently, data quality detection mainly includes offline detection and online detection.

[0003] Offline detection is to verify data through batch processing, that is, the data is first stored, and then quality inspection and cleaning are performed regularly. Therefore, offline detection has obvious time delay and cannot meet the data quality detection scenarios with high real-time requirements. In addition, the original data of real-time data may not be available during offline detection. The data generated after real-time processing may be different from the original input, or even unrecognizable. The processed data may be sent directly to the application party, or enter the message queue, and may not be stored persistently, resulting in the inability to establish the basic conditions for subsequent offline detection and the inability to implement data quality detection.

[0004] Online detection is mainly based on business-side inspection and is only suitable for specific business data or specific business scenarios. It needs to invade the coding of specific businesses, is not universal, and requires complex configuration and a large amount of computing resources. It is highly complex and has poor flexibility. Summary of the invention

[0005] The technical problem to be solved by the present invention is: the present invention provides a method and system for real-time data quality detection, which ensures the real-time performance of real-time data quality detection while improving the flexibility of real-time data quality detection and reducing the complexity of real-time data quality detection.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a method and system for real-time data quality detection, comprising:

[0008] Receive a user's rule configuration, and generate a data quality plug-in according to the rule configuration;

[0009] Obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain a corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain a quality detection result for each data stream;

[0010] The quality inspection result is stored in ClickHouse, and an early warning rule is obtained from the rule configuration. According to the early warning rule and the materialized view capability of ClickHouse, an early warning is issued for the data stream corresponding to the quality inspection result.

[0011] The beneficial effects of the present invention are as follows: a data quality plug-in is generated according to rule configuration, and the data quality plug-in is integrated into the Flink program in the form of a plug-in, without redundant storage of the original data stream, thereby ensuring the real-time performance of real-time data quality detection, and without intruding into the coding of specific businesses, thereby simplifying the complexity of real-time data quality detection while improving flexibility, and storing the quality detection results in ClickHouse, utilizing the OLAP capability of ClickHouse to further improve the real-time performance of real-time data quality detection, and combining the early warning rules with the materialized view capability of ClickHouse to implement early warning of the data stream corresponding to the quality detection results, thereby optimizing the user experience.

[0012] Optionally, storing the quality detection result in ClickHouse includes:

[0013] Determine whether the quality inspection result is passed. If so, release the data stream corresponding to the quality inspection result to the mainstream. If not, encapsulate the data stream corresponding to the quality inspection result to obtain the encapsulated data stream, intercept the encapsulated data stream to the dirty data stream, and store the data stream corresponding to the quality inspection result to ClickHouse.

[0014] According to the above description, what is stored in ClickHouse is not all data, but only the data stream that fails the quality inspection, which reduces the consumption of storage resources and improves the efficiency of subsequent processing.

[0015] Optionally, the dirty data stream is open to users, and the user is allowed to optimize the dirty data stream, and the dirty data stream is optimized according to the optimized configuration.

[0016] According to the above description, dirty data streams are not only open to users but also can be accepted and optimized according to the user's optimization configuration, thereby optimizing the user experience and data processing at the same time.

[0017] Optionally, storing the quality inspection result in ClickHouse, obtaining an early warning rule from the rule configuration, and issuing an early warning for the data stream corresponding to the quality inspection result according to the early warning rule and the materialized view capability of ClickHouse includes:

[0018] Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list;

[0019] ClickHouse's materialized view capability is used to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table.

[0020] Acquire warning rules from the rule configuration, and issue warnings to the data stream corresponding to the quality detection result according to the warning rules and the summary table.

[0021] According to the above description, ClickHouse's materialized view capability allows us to complete a series of stateless streaming processing and improve the flexibility and scalability of processing by focusing on the design of the detail table and the SQL processing logic.

[0022] Optionally, acquiring a warning rule from the rule configuration, and issuing a warning for the data stream corresponding to the quality detection result according to the warning rule and the summary table includes:

[0023] The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

[0024] According to the above description, when judging whether the summary value reaches the threshold, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the early warning rule and compared with the summary value. Compared with the traditional method of rewriting the Flink program to first read the summary value from the summary table, then read the threshold, and then compare the two, it reduces complexity and processing costs. Compared with the method of associating ClickHouse with the MySQL table engine to do a Cartesian product to determine whether the threshold is reached, it reduces resource consumption and improves processing efficiency.

[0025] In a second aspect, the present invention provides a system for real-time data quality detection, comprising:

[0026] A plug-in generation module, used to receive a user's rule configuration and generate a data quality plug-in according to the rule configuration;

[0027] A quality detection module is used to obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain a corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain a quality detection result for each data stream;

[0028] An early warning module is used to store the quality inspection results in ClickHouse, obtain early warning rules from the rule configuration, and issue early warnings to the data stream corresponding to the quality inspection results according to the early warning rules and the materialized view capability of ClickHouse.

[0029] The beneficial effects of the present invention are as follows: a data quality plug-in is generated according to rule configuration, and the data quality plug-in is integrated into the Flink program in the form of a plug-in, without redundant storage of the original data stream, thereby ensuring the real-time performance of real-time data quality detection, and without intruding into the coding of specific businesses, thereby simplifying the complexity of real-time data quality detection while improving flexibility, and storing the quality detection results in ClickHouse, utilizing the OLAP capability of ClickHouse to further improve the real-time performance of real-time data quality detection, and combining the early warning rules with the materialized view capability of ClickHouse to implement early warning of the data stream corresponding to the quality detection results, thereby optimizing the user experience.

[0030] Optionally, the early warning module includes:

[0031] The judgment module is used to judge whether the quality inspection result is passed. If so, the data stream corresponding to the quality inspection result is released to the mainstream. If not, the data stream corresponding to the quality inspection result is encapsulated to obtain the encapsulated data stream, the encapsulated data stream is intercepted to the dirty data stream, and the data stream corresponding to the quality inspection result is stored in ClickHouse.

[0032] According to the above description, what is stored in ClickHouse is not all data, but only the data stream that fails the quality inspection, which reduces the consumption of storage resources and improves the efficiency of subsequent processing.

[0033] Optionally, the dirty data stream is open to users, and the user is allowed to optimize the dirty data stream, and the dirty data stream is optimized according to the optimized configuration.

[0034] According to the above description, dirty data streams are not only open to users but also can be accepted and optimized according to the user's optimization configuration, thereby optimizing the user experience and data processing at the same time.

[0035] Optionally, the early warning module is specifically:

[0036] Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list;

[0037] ClickHouse's materialized view capability is used to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table.

[0038] Acquire warning rules from the rule configuration, and issue warnings to the data stream corresponding to the quality detection result according to the warning rules and the summary table.

[0039] According to the above description, ClickHouse's materialized view capability allows us to complete a series of stateless streaming processing and improve the flexibility and scalability of processing by focusing on the design of the detail table and the SQL processing logic.

[0040] Optionally, the early warning module is specifically:

[0041] The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

[0042] According to the above description, when judging whether the summary value reaches the threshold, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the early warning rule and compared with the summary value. Compared with the traditional method of rewriting the Flink program to first read the summary value from the summary table, then read the threshold, and then compare the two, it reduces complexity and processing costs. Compared with the method of associating ClickHouse with the MySQL table engine to do a Cartesian product to determine whether the threshold is reached, it reduces resource consumption and improves processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a method for real-time data quality detection provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the overall process of a method for real-time data quality detection provided by an embodiment of the present invention;

[0045] Figure 3A schematic diagram of the structure of a real-time data quality detection system provided by an embodiment of the present invention.

[0046] [Description of Reference Numerals]

[0047] 1. A system for real-time data quality detection;

[0048] 2. Plug-in generation module;

[0049] 3. Quality inspection module;

[0050] 4. Early warning module; 41. Judgment module. DETAILED DESCRIPTION

[0051] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0052] Embodiment 1

[0053] Please refer to Figure 1 to Figure 2 The present invention provides a method for real-time data quality detection, comprising the steps of:

[0054] S1. Receive a user's rule configuration and generate a data quality plug-in according to the rule configuration;

[0055] In this embodiment, if Figure 2 As shown, the user's rule configuration is received. If the user does not configure the rules, the data quality plug-in will be generated according to the built-in rules. Whether it is the user's rule configuration or the built-in rules, their scope is at the field level. It can be a single rule or a combination of multiple rules.

[0056] The built-in rules are as follows:

[0057] Built-in rule 1: Field mismatch: The collected data does not match the number of schemas input by the collection source.

[0058] Built-in rule 2: Type conversion error: The collected data does not match the source input schema type and cannot be forcibly converted to the data type defined in the schema.

[0059] The default thresholds of built-in rule 1 and built-in rule 2 are 1. Users can adjust the thresholds, and the built-in rules can also be adjusted according to actual conditions.

[0060] The user's rule configuration needs to follow the configuration specifications, which are as follows:

[0061] Configuration specification 1: Not empty: The user configures the rule name, rule action field, and alarm threshold. The rule is hit when the data in the rule action field is empty.

[0062] Configuration rule 2: Length check: The user configures the rule name, rule action field, minimum value, maximum value and alarm threshold. The maximum value can be positive infinity and the minimum value can be negative infinity. The rule is hit when the character length of the data in the rule action field is not within the range.

[0063] Configuration specification 3: Numerical range: The user configures the rule name, rule action field, minimum value, maximum value and alarm threshold. The maximum value can be positive infinity and the minimum value can be negative infinity. The rule is hit when the value of the rule action field is not within the range.

[0064] Configuration specification 4: Enumeration: The user configures the rule name, rule action field, enumeration value, hit mode selection, alarm threshold, when the value of the rule action field matches the enumeration value, when the value of the rule action field does not match the enumeration value, and the hit selection mode is: match enumeration hit, do not match enumeration hit, and hit the rule according to the hit mode.

[0065] Configuration rule 5: Regular: The user configures the rule name, rule action field, regular expression, hit mode selection, alarm threshold, when the value of the rule action field matches the regular expression, when the value of the rule action field does not match the regular expression, the hit selection mode is: match enumeration hit, do not match enumeration hit, hit the rule according to the hit mode.

[0066] Configuration rule 6: Time range: The user configures the rule name, rule action field, time type, action field offset before and after the current time, and alarm threshold. The rule is hit when the rule action field time is not within the range of the current program processing time.

[0067] Configuration rule 7: Data delay: User-configured rule name, Flink program time model: eventtime, processtime to find the value greater than the set hit rule.

[0068] The above configuration specifications can be set according to actual conditions.

[0069] S2. Obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain a corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain a quality detection result for each data stream;

[0070] In this embodiment, if Figure 2 As shown, the data quality plug-in generated in step S1 is integrated into the Flink program, and the data quality plug-in is integrated into the data stream that needs to be monitored in the Flink program. When the Flink program receives the data stream that needs to be monitored in real time, it will enter the data quality detection, that is, obtain the corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream to obtain the quality detection result of each data stream.

[0071] S3. Store the quality inspection result in ClickHouse, obtain the warning rule from the rule configuration, and issue a warning for the data stream corresponding to the quality inspection result according to the warning rule and the materialized view capability of ClickHouse.

[0072] In this embodiment, the ClickHouse environment is pre-configured so that the quality inspection results obtained in step S2 are stored in ClickHouse, and an early warning is issued for the data stream corresponding to the quality inspection results based on the early warning rules obtained from the rule configuration and the materialized view capability of ClickHouse.

[0073] At this time, storing the quality detection results in ClickHouse in step S3 includes:

[0074] S31. Determine whether the quality inspection result is passed. If so, release the data stream corresponding to the quality inspection result to the mainstream. If not, encapsulate the data stream corresponding to the quality inspection result to obtain the encapsulated data stream, intercept the encapsulated data stream to the dirty data stream, and store the data stream corresponding to the quality inspection result to ClickHouse.

[0075] In this embodiment, if Figure 2 As shown in the figure, data streams that pass the quality inspection will be released to the mainstream, while data streams that fail the quality inspection will be uniformly encapsulated and intercepted into dirty data streams. At the same time, the corresponding data streams are stored in ClickHouse. Data streams that fail the quality inspection are intercepted into dirty data streams by default, but users are provided with options. If the user chooses to keep it in the mainstream, it will be processed according to the user's choice.

[0076] At this time, the dirty data stream in step S31 is open to the user, and the user is accepted to optimize the configuration of the dirty data stream, and the dirty data stream is optimized according to the optimization configuration.

[0077] In this embodiment, the dirty data stream is open to the user, and the user can access it through: ".getSideOutput(newOutputTag<row>("dirtyTag"))" to obtain the dirty data stream, and the user can optimize the dirty data stream, modify the dirty data stream, or back up the dirty data stream, according to the actual situation.

[0078] At this time, the quality inspection result is stored in ClickHouse in step S3, and the warning rule is obtained from the rule configuration. The data stream corresponding to the quality inspection result is warned according to the warning rule and the materialized view capability of ClickHouse, including:

[0079] S32. Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list;

[0080] In this embodiment, if Figure 2 As shown in the figure, the quality inspection results and the quality details corresponding to each quality inspection result will be written into the detail table. The preset detail table includes: Flink job name, rule name, rule ID, rule type, effect field, field value, original information, etc.

[0081] S33. Use the materialized view capability of ClickHouse to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table;

[0082] In this embodiment, if Figure 2 As shown in the figure, the materialized view capability of ClickHouse is used to automatically aggregate the written detail table in real time according to the aggregation rules using the AggregatingMergeTree table engine to obtain a summary table. The aggregation rules, that is, the aggregation dimensions, are based on the Flink job name, rule name, rule type, and action field. The summary table summarizes the number of data streams, the earliest trigger time, and the latest trigger time of the hit rules.

[0083] S34. Obtain an early warning rule from the rule configuration, and issue an early warning to the data stream corresponding to the quality detection result according to the early warning rule and the summary table.

[0084] At this time, step S34 includes:

[0085] The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

[0086] In this embodiment, if Figure 2 As shown, the materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table. When changes are detected in the summary table, the corresponding threshold is backfilled into the preset data dictionary in the memory according to the rule ID in the warning rule. The data dictionary is pre-set and is a code table. The engine of the table is Dictionary. Its main function is to store the threshold corresponding to the rule ID, which is used for the fact join operation of the alarm table. When the summary value reaches the threshold, the data stream corresponding to the summary table is automatically updated to the alarm table, and the data stream in the alarm table is warned. The warning methods include but are not limited to: email warning and webhook warning.

[0087] Embodiment 2

[0088] Please refer to Figure 3 The present invention provides a real-time data quality detection system 1, comprising: a plug-in generation module 2, a quality detection module 3, an early warning module 4 and a judgment module 41.

[0089] Wherein, the plug-in generation module 2 is used to receive the user's rule configuration and generate a data quality plug-in according to the rule configuration;

[0090] The quality detection module 3 is used to obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain the corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain the quality detection result of each data stream;

[0091] The early warning module 4 is used to store the quality inspection results in ClickHouse, obtain the early warning rules from the rule configuration, and issue an early warning for the data stream corresponding to the quality inspection results according to the early warning rules and the materialized view capability of ClickHouse.

[0092] Specifically, the early warning module 4 includes:

[0093] The judgment module 41 is used to judge whether the quality inspection result is passed. If so, the data stream corresponding to the quality inspection result is released to the mainstream. If not, the data stream corresponding to the quality inspection result is encapsulated to obtain the encapsulated data stream, the encapsulated data stream is intercepted to the dirty data stream, and the data stream corresponding to the quality inspection result is stored in ClickHouse.

[0094] The dirty data stream is open to users, and accepts optimization configuration of the dirty data stream by users, and optimizes the dirty data stream according to the optimization configuration.

[0095] Specifically, the early warning module 4 is specifically:

[0096] Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list;

[0097] ClickHouse's materialized view capability is used to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table.

[0098] Acquire warning rules from the rule configuration, and issue warnings to the data stream corresponding to the quality detection result according to the warning rules and the summary table.

[0099] The early warning module 4 is specifically:

[0100] The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

[0101] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0102] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0104] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0105] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples", etc., refers to the specific features, structures, materials or characteristics described in conjunction with the embodiment or example included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0106] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0107] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.< / row>

Claims

1. A method for real-time data quality detection, characterized in that: include: Receive a user's rule configuration, and generate a data quality plug-in according to the rule configuration; Obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain a corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain a quality detection result for each data stream; The quality inspection result is stored in ClickHouse, and an early warning rule is obtained from the rule configuration. According to the early warning rule and the materialized view capability of ClickHouse, an early warning is issued for the data stream corresponding to the quality inspection result.

2. A method for real-time data quality detection as claimed in claim 1, characterized in that: Storing the quality inspection results in ClickHouse includes: Determine whether the quality inspection result is passed. If so, release the data stream corresponding to the quality inspection result to the mainstream. If not, encapsulate the data stream corresponding to the quality inspection result to obtain the encapsulated data stream, intercept the encapsulated data stream to the dirty data stream, and store the data stream corresponding to the quality inspection result to ClickHouse.

3. A method for real-time data quality detection as claimed in claim 2, characterized in that: The dirty data stream is open to users, and accepts optimization configuration of the dirty data stream by users, and optimizes the dirty data stream according to the optimization configuration.

4. A method for real-time data quality detection as claimed in claim 1, characterized in that: The storing of the quality inspection result to ClickHouse, obtaining the warning rule from the rule configuration, and issuing a warning for the data stream corresponding to the quality inspection result according to the warning rule and the materialized view capability of ClickHouse include: Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list; ClickHouse's materialized view capability is used to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table. Acquire warning rules from the rule configuration, and issue warnings to the data stream corresponding to the quality detection result according to the warning rules and the summary table.

5. A method for real-time data quality detection as claimed in claim 4, characterized in that: The obtaining of the warning rule from the rule configuration and issuing a warning to the data stream corresponding to the quality detection result according to the warning rule and the summary table comprises: The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

6. A system for real-time data quality detection, characterized in that: include: A plug-in generation module, used to receive a user's rule configuration and generate a data quality plug-in according to the rule configuration; A quality detection module is used to obtain a Flink program, integrate the data quality plug-in into the Flink program, and when the Flink program receives a data stream in real time, obtain a corresponding rule configuration from the data quality plug-in to perform real-time quality detection on the data stream, and obtain a quality detection result for each data stream; An early warning module is used to store the quality inspection results in ClickHouse, obtain early warning rules from the rule configuration, and issue early warnings to the data stream corresponding to the quality inspection results according to the early warning rules and the materialized view capability of ClickHouse.

7. A real-time data quality detection system as claimed in claim 6, characterized in that: The early warning module comprises: The judgment module is used to judge whether the quality inspection result is passed. If so, the data stream corresponding to the quality inspection result is released to the mainstream. If not, the data stream corresponding to the quality inspection result is encapsulated to obtain the encapsulated data stream, the encapsulated data stream is intercepted to the dirty data stream, and the data stream corresponding to the quality inspection result is stored in ClickHouse.

8. A real-time data quality detection system as claimed in claim 7, characterized in that: The dirty data stream is open to users, and accepts optimization configuration of the dirty data stream by users, and optimizes the dirty data stream according to the optimization configuration.

9. A real-time data quality detection system as claimed in claim 6, characterized in that: The early warning module is specifically: Obtain a preset detailed list from ClickHouse, write all the quality inspection results and the quality details corresponding to each quality inspection result into the detailed list, and obtain the written detailed list; ClickHouse's materialized view capability is used to automatically aggregate the written detail table in real time according to the aggregation rules to obtain a summary table. Acquire warning rules from the rule configuration, and issue warnings to the data stream corresponding to the quality detection result according to the warning rules and the summary table.

10. A real-time data quality detection system as claimed in claim 9, characterized in that: The early warning module is specifically: The materialized view capability of ClickHouse is used to generate a summary materialized view of the summary table, and the summary materialized view is used as an alarm table. When changes in the summary table are monitored, the corresponding threshold is backfilled from the preset data dictionary in the memory according to the rule ID in the alarm rule, and the threshold is compared with the summary value of the rule ID in the summary table. If the summary value reaches the threshold, the data flow corresponding to the summary value is automatically updated to the alarm table, and an alarm is issued for the data flow in the alarm table.

Images