Distributed real-time data quality detection method and system based on complex matrix

Abstract

The invention discloses a distributed real-time data quality detection method and system based on a complex matrix, and belongs to the technical field of real-time data quality detection.The method comprises the steps that a real-time data stream is obtained from a data source, and after data cleaning and conversion are conducted, an initial real-time data stream is generated; and performing quality evaluation according to the data sample and the quality index of the initial real-time data stream, and outputting a result through Flink. The real-time data quality detection method based on the Flink streaming obtains significant progress in the field of real-time data quality monitoring, successfully solves the key problem of real-time data quality detection by introducing a complex matrix algorithm and combining Flink streaming processing, and has wide potential and superiority in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time data quality detection, and in particular to a distributed real-time data quality detection method and system based on complex matrix. Background Art

[0002] In the field of real-time computing, data quality detection technology is immature, which leads to the need for algorithms and operations to make a lot of backup strategies when making real-time strategies. This detection technology plays an important role in vehicle-cargo matching algorithms and related operational scenarios. Vehicle-cargo matching algorithms rely on accurate and real-time data, including cargo information, vehicle location, transportation timeliness, etc. This patent provides a method for real-time data quality detection to ensure that the data stream used for vehicle-cargo matching is high-quality and reliable, thereby improving the accuracy and credibility of the matching algorithm.

[0003] Prior art mentioned: Prior art 1: CN113901035A Real-time data quality detection method and system based on Flink, specifically as follows: configure quality detection rules; configure Flink window configuration; select the data source, data table and field to be detected, and configure the corresponding detection rules for the field to be detected; after the data source to be detected, the detection rules and the Flink window configuration are completed, a task is formed, and the task is sent to the task scheduling center for unified scheduling; the scheduling center manages and schedules quality detection tasks, and after the task is triggered, the Flink data processing module is called to perform actual data quality detection. The system includes a rule customization module, a task configuration module, a task scheduling module, a Flink data processing module and a scheduling center. Prior art 2: CN112115121B A data governance real-time data quality detection system, including HBASE database, configuration module, Spark detection module, statistics module and storage module, the HBASE database is used for initialization of data incremental backup table, the configuration module configures detection task, inputs task name, table to be detected, association rules, priority of task execution, the Spark detection module receives detection task configured from the configuration module, and performs detection, the statistics module uses automatic accumulator to perform statistics on detection results of data that meets the detection rules. The present invention can process PB-level data, detect huge amount of data, adopt cluster detection mode, respond quickly, flexible rules support single field pair rules and single rule multiple detection, built-in multiple detection rules, and can also support user-defined detection, can continuously and automatically detect incremental data of configured tasks, and can issue a complete data quality detection report. The existing technologies 1-2 have the following problems: simply relying on flink or Spark plus rules, the configuration of quality detection rules and Flink or Spark window configuration may become complicated, especially when dealing with multiple data sources, data tables and fields, which may require more configuration and management, and requires a lot of experience accumulation, and cannot meet business needs quickly; it does not consider the impact of multiple quality indicators on the business, and multiple quality indicators need to be considered comprehensively in data quality detection to improve the accuracy of the evaluation, which is not considered in the above similar inventions; it does not dynamically adjust the weight of each detection indicator through historical data, and cannot more comprehensively solve the data quality problems in practical applications. Therefore, it is urgent to design a method to achieve real-time data quality and high-quality detection. Summary of the invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a distributed real-time data quality detection technology based on complex matrices. By integrating complex matrix algorithms, historical data analysis, Flink streaming processing and other technologies, it aims to detect and evaluate the quality of real-time data and ensure the accuracy, completeness and credibility of the data during the processing.

[0005] In order to achieve the above technical objectives, the present application provides a distributed real-time data quality detection method based on a complex matrix, comprising the following steps: Obtain real-time data streams from data sources, perform data cleaning and conversion, and generate initial real-time data streams; Based on the data samples and quality indicators of the initial real-time data stream, quality assessment is performed and the results are output through Flink.

[0006] Preferably, in the process of acquiring the data source, the data source includes a message queue of Kafka or MQTT or an incremental log of a database.

[0007] Preferably, in the process of acquiring the initial real-time data stream, missing values ​​are filled, abnormal data is processed, and data format is converted to generate the initial real-time data stream.

[0008] Preferably, in the process of quality assessment, the quality of the initial real-time data stream is assessed by a complex matrix algorithm, wherein the complex matrix algorithm is expressed as: D[i,j]=(α*f(data[i])+β*g(quality[j])) / (α+β) Where D[i,j] represents the element in the i-th row and j-th column of the data quality matrix; α and β are weight factors used to balance the importance of data samples and quality indicators; f(data[i]) represents the first calculation function applied to the data sample; g(quality[j]) represents the second calculation function applied to the quality indicator. The values ​​of α and β are determined by analyzing historical data to reflect the importance of the data sample and the quality indicator respectively.

[0009] Preferably, in the process of obtaining the first calculation function and the second calculation function, a combination of the first calculation function and the second calculation function is constructed according to data and business requirements, wherein the combination includes: one of a linear combination, a weighted average and a nonlinear function.

[0010] Preferably, after the process of obtaining the quality assessment results, specific processing is performed on the problematic data, including marking, filtering or backfilling.

[0011] Preferably, in the process of outputting results through Flink, the quality assessment results are output to an external system, including a database or a message queue, through Flink's sink, and data quality monitoring and alarm are performed.

[0012] The present invention also discloses a distributed real-time data quality detection system based on complex matrix, comprising: The data source module is used to obtain the incremental logs of Kafka, MQTT message queues and databases; The Flink task module is used to process streaming data and generate quality assessment results through complex matrix algorithms according to the above-mentioned distributed real-time data quality detection method based on complex matrices; The algorithm engine module is used to process the problem data according to the quality assessment results; Data quality monitoring, used to monitor quality indicators in real time and support alarms.

[0013] Preferably, the complex matrix algorithm is expressed as: D[i,j]=(α*f(data[i])+β*g(quality[j])) / (α+β) Where D[i,j] represents the element in the i-th row and j-th column of the data quality matrix; α and β are weight factors used to balance the importance of data samples and quality indicators; f(data[i]) represents the first calculation function applied to the data sample; g(quality[j]) represents the second calculation function applied to the quality indicator. The values ​​of α and β are determined by analyzing historical data to reflect the importance of the data sample and the quality indicator respectively. In the process of obtaining the first calculation function and the second calculation function, a combination of the first calculation function and the second calculation function is constructed according to data and business requirements, wherein the combination includes: one of a linear combination, a weighted average, and a nonlinear function.

[0014] Preferably, the system uses Spring Boot for module management.

[0015] The present invention discloses the following technical effects: The present invention has made significant progress in the field of real-time data quality monitoring. By introducing complex matrix algorithms and combining them with Flink streaming processing, it has successfully solved the key issues of real-time data quality detection and has broad potential and superiority in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the real-time data quality detection method described in the present invention. DETAILED DESCRIPTION

[0018] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0019] like Figure 1 As shown, the present invention provides a distributed real-time data quality detection method based on a complex matrix, which specifically includes the following contents: Data acquisition: First, you need to obtain real-time data streams from the data source. The data source can be a message queue such as Kafka and MQTT, or an incremental log of a database. Flink's source connectors can easily connect to these external systems.

[0020] Data cleaning and conversion: Next, the acquired data needs to be cleaned and converted, missing values ​​need to be filled, abnormal data needs to be processed, data formats need to be converted, etc. This step is very important and directly affects the quality of downstream analysis.

[0021] Data quality assessment: After data cleaning and conversion, data quality needs to be assessed in real time. Here, the present invention uses a complex matrix algorithm to calculate multiple quality indicators to determine whether the data is usable. The implementation process of the complex matrix algorithm: The formula for the complex matrix algorithm is: D[i,j]=(α*f(data[i])+β*g(quality[j])) / (α+β) Where D[i,j] represents the i-th row and j-th column element in the data quality matrix; α and β are weight factors used to balance the importance of data samples and quality indicators; f(data[i]) represents the calculation function applied to the data sample; g(quality[j]) represents the calculation function applied to the quality indicator. The values ​​of α and β can be determined by analyzing historical data, reflecting the importance of the data sample and the quality indicator respectively. Set α+β=1 so that the quality score is between 0 and 1.

[0022] Design of f function and g function: The implementation of the f function and the g function will vary depending on the specific data and business requirements. Here are three common construction methods: Method 1: Linear combination: f(data[i])=data[i] g(quality[j])=quality[j] Method 2: Weighted average: f(data[i])=α*data[i] g(quality[j])=β*quality[j] Method 3: Non-linear function: f(data[i])=α*g(data[i]) g(quality[j])=β*h(quality[j]) Among them, g and h are nonlinear functions, and appropriate functions can be selected according to actual needs, such as logarithmic function, exponential function, etc.

[0023] Problem data processing: Based on the results of quality assessment, the present invention can perform specific processing on problematic data, such as marking, filtering, backfilling, etc., to improve the quality of downstream analysis.

[0024] Result output: Finally, the quality detection results can be output to external systems such as databases, message queues, etc. through Flink's sink, and data quality monitoring and alarms can be performed.

[0025] Implementation process of distributed real-time data quality detection method based on complex matrix: The present invention proposes a real-time data quality detection system based on Flink, which mainly includes: Data source: supports multiple sources such as Kafka and HDFS; Flink task: performs streaming data processing and integrates complex matrix algorithms; Algorithm engine: processes problem data based on quality scoring results; Data quality monitoring: real-time monitoring of quality indicators and support for alarms.

[0026] The system uses SpringBoot for module management and implements functional modules such as data acquisition, data processing, quality inspection, and problem data processing. The key technical point is to integrate streaming data processing and quality inspection algorithms based on Flink.

[0027] Experimental process of distributed real-time data quality detection method based on complex matrix: To verify the effectiveness of the proposed method, the present invention uses simulated data based on the Manbang highway logistics scenario to conduct experiments. The experimental data contains 50 samples, each containing 4 quality indicators. The data quality score is calculated using a complex matrix algorithm and compared with the "usable" label determined manually. The experimental data example is shown in Table 1: Table 1 Data serial number Shipping time Temperature compliance Humidity compliance Is it damaged? Data quality scoring Availability 1 62 hours yes yes no 0.82 yes 2 58 hours yes no yes 0.71 no 3 72 hours no yes no 0.76 yes 4 55 hours yes yes no 0.85 yes ...... ...... ...... ...... ...... ...... ...... The results show that this method can accurately distinguish good and poor quality Manbang highway logistics data and detect quality problems in real time. As the amount of data increases, the algorithm's computing performance can be linearly expanded to meet real-time processing requirements.

[0028] The streaming data quality detection algorithm proposed in the present invention: uses Flink to process streaming data, with low latency and exactlyonce state consistency guarantee; the quality assessment results perform real-time specific processing on problematic data, such as marking, filtering, and backfilling; The real-time complex matrix algorithm proposed in this invention is advanced in that it uses historical data, calculates weight factors α and β, and implements f function and g function. It solves the problem that traditional real-time data quality detection relies too much on rules. The variability of the real-time complex matrix algorithm proposed in the present invention: provides a variety of construction methods, such as linear combination, weighted average, nonlinear function, etc., to meet different data and business requirements.

[0029] The present invention has the following characteristics: 1. Real-time data quality monitoring: This paper provides an algorithm based on Flink and distributed data quality detection technology to achieve real-time data quality monitoring. Through the Flink stream processing engine, the quality of data can be detected and evaluated in real time in the data stream to ensure the timeliness and accuracy of the data.

[0030] 2. Application of complex matrix algorithm: Innovatively introduces complex matrix algorithm as the core method of data quality assessment, making the comprehensive assessment of multiple quality indicators more accurate. Provides variability construction methods, such as linear combination, weighted average, nonlinear function, etc., to adapt to different business needs and improve the flexibility of the algorithm.

[0031] 3. Flexibility and scalability: By combining Flink streaming processing and complex matrix algorithms, flexible processing of different data sources and different quality indicators is achieved. The system design takes scalability into consideration and can meet real-time processing requirements. As the amount of data increases, the algorithm's computing performance can be linearly expanded.

[0032] 4. Real-time processing of problematic data: The problematic data processing step proposed by the present invention performs specific processing on problematic data based on the real-time quality assessment results. This real-time processing can timely identify and process data of poor quality, thereby improving the accuracy of overall data analysis.

[0033] 5. System integrity and usability: The present invention designs a real-time data quality detection system based on Flink, including the design of multiple modules, such as data source, Flink task, rule engine, data quality monitoring, etc. Using SpringBoot for module management improves the integrity and usability of the system, making the technical solution easier to be applied in practice.

[0034] 6. Effectiveness of the validation experiment: The present invention verifies the effectiveness of the proposed method by conducting experiments based on simulated data of the Manbang highway logistics scenario. The experimental results show that the method can accurately distinguish between data of good and poor quality and detect quality problems in real time, proving the feasibility and superiority of the technical solution in practical applications.

[0035] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0036] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0037] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A distributed real-time data quality detection method based on complex matrix, characterized in that: The following steps are involved: Obtain real-time data streams from data sources, perform data cleaning and conversion, and generate initial real-time data streams; After quality assessment is performed based on the data samples and quality indicators of the initial real-time data stream, the results are output through Flink.

2. A distributed real-time data quality detection method based on complex matrix according to claim 1, characterized in that: In the process of acquiring the data source, the data source includes a message queue of Kafka or MQTT or an incremental log of a database.

3. A distributed real-time data quality detection method based on complex matrix according to claim 2, characterized in that: In the process of acquiring the initial real-time data stream, missing values ​​are filled, abnormal data is processed, and data format is converted for the real-time data stream to generate the initial real-time data stream.

4. A distributed real-time data quality detection method based on complex matrix according to claim 3, characterized in that: In the process of quality assessment, the quality of the initial real-time data stream is assessed by a complex matrix algorithm, wherein the complex matrix algorithm is expressed as: D[i,j]=(α*f(data[i])+β*g(quality[j])) / (α+β) Where D[i,j] represents the element in the i-th row and j-th column of the data quality matrix; α and β are weight factors used to balance the importance of data samples and quality indicators; f(data[i]) represents the first calculation function applied to the data sample; g(quality[j]) represents the second calculation function applied to the quality indicator. The values ​​of α and β are determined by analyzing historical data to reflect the importance of the data sample and the quality indicator respectively.

5. A distributed real-time data quality detection method based on complex matrix according to claim 4, characterized in that: In the process of obtaining the first calculation function and the second calculation function, a combination of the first calculation function and the second calculation function is constructed according to data and business requirements, wherein the combination includes: one of a linear combination, a weighted average, and a nonlinear function.

6. A distributed real-time data quality detection method based on complex matrix according to claim 5, characterized in that: After obtaining the quality assessment results, specific processing is performed on the problematic data, including marking, filtering or backfilling.

7. A distributed real-time data quality detection method based on complex matrix according to claim 6, characterized in that: In the process of outputting results through Flink, the quality assessment results are output to external systems, including databases or message queues, through Flink's sink, and data quality monitoring and alarms are performed.

8. A distributed real-time data quality detection system based on complex matrix, characterized in that: include: The data source module is used to obtain the incremental logs of Kafka, MQTT message queues and databases; A Flink task module, configured to perform streaming data processing and generate quality assessment results by using a complex matrix algorithm according to a distributed real-time data quality detection method based on a complex matrix as described in any one of claims 1 to 7; The algorithm engine module is used to process the problem data according to the quality assessment results; Data quality monitoring, used to monitor quality indicators in real time and support alarms.

9. A distributed real-time data quality detection system based on complex matrix according to claim 8, characterized in that: The complex matrix algorithm is expressed as: D[i,j]=(α*f(data[i])+β*g(quality[j])) / (α+β) Where D[i,j] represents the element in the i-th row and j-th column of the data quality matrix; α and β are weight factors used to balance the importance of data samples and quality indicators; f(data[i]) represents the first calculation function applied to the data sample; g(quality[j]) represents the second calculation function applied to the quality indicator. The values ​​of α and β are determined by analyzing historical data to reflect the importance of the data sample and the quality indicator respectively. In the process of obtaining the first calculation function and the second calculation function, a combination of the first calculation function and the second calculation function is constructed according to data and business requirements, wherein the combination includes: one of a linear combination, a weighted average, and a nonlinear function.

10. A distributed real-time data quality detection system based on complex matrix according to claim 9, characterized in that: The system uses Spring Boot for module management.

Images