A flink-based streaming data quality monitoring method and system thereof

Abstract

The application provides a kind of Flink-based streaming data quality monitoring method, comprising: the field rule data generated according to the format of JSON is stored in database;Flink starts task, and Flink program reads field rule data in database, parses JSON data and caches to memory;When Flink finishes parsing field rule data, Flink program starts the data of consumption message queue;While Flink program consumes message queue data, trigger Flink program to start the monitoring of message queue processing data volume, to carry out the statistics of current time processing data volume;Flink analyzes the current data by FlinkSQL in combination with custom operator;Flink program carries out statistics to the current data analyzed, and writes the result of statistics into database.By the combination component and design operator of Flink, the data monitoring of streaming data can be carried out, the quality of streaming data is guaranteed and monitored to a certain extent, and can be flexibly changed according to the current business situation by custom operator and rule.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically relating to a streaming data quality monitoring method and system based on Flink. Background Technology

[0002] Business or usage scenarios based on data platforms require these platforms to process large amounts of data to support various business operations. Ensuring data quality becomes a key issue for data platforms. The main reasons for data quality issues can be categorized into several aspects. One aspect is that a data platform may use different technologies and tools for data processing, and the quality of the processed data cannot be guaranteed across these various tools.

[0003] Another aspect is the processing of streaming data in real-time scenarios. When Flink is chosen to process data, although Flink has certain mechanisms to ensure data integrity, it cannot guarantee the accuracy, completeness, consistency, uniqueness, and standardization of the data processed by Flink. For streaming data, it is generally difficult to monitor the integrity and standardization of the data, which can lead to the inability to detect data problems in a timely manner, thus affecting the quality of the data.

[0004] Currently, most data monitoring is based on data warehouses, or uses data warehouses combined with technologies such as Hudi and SQL scripts for data quality monitoring. The main drawback is that there is no quality monitoring for streaming data.

[0005] Therefore, it is of great significance to propose a streaming data quality monitoring method and system based on Flink. Summary of the Invention

[0006] To address issues related to data integrity and standardization in Flink-based streaming data quality monitoring, this invention provides a Flink-based streaming data quality monitoring method and system to resolve the aforementioned technical deficiencies.

[0007] Firstly, this invention proposes a Flink-based method for monitoring the quality of streaming data, which includes the following steps:

[0008] S1. Store the pre-generated field rule data in the database in JSON format;

[0009] S2. Flink starts the task. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory.

[0010] S3. After Flink finishes parsing the field rule data, the Flink program starts consuming the data from the message queue.

[0011] S4. While the Flink program consumes the message queue data, it triggers the Flink program to start monitoring the amount of data processed by the message queue in order to count the amount of data processed at the current time.

[0012] S5 and Flink analyze the current data using FlinkSQL combined with custom operators;

[0013] S6. The Flink program performs statistics on the current data analysis and writes the statistical results into the database.

[0014] Preferably, S2 specifically includes:

[0015] S21. To start a Flink task to read data from the corresponding table in the database, you need to first configure the database user for Flink to use.

[0016] S22. Use Flink CDC to read the field rule data in the database, and use Flink code configuration to read data from the NoSQL database;

[0017] S23. The Flink program parses the JSON data, reads the corresponding fields and regular expressions, caches them in memory, and provides them to Flink's custom operators and FlinkSQL for invocation.

[0018] Further preferred, S4 specifically includes:

[0019] S41. Use Flink's Metrics to monitor metrics and customize the CustomerJsonDeserialization class by overriding the initMertic method.

[0020] S42. Based on Flink's default function of reporting acquired information, further configuration is performed to enable the Flink program to obtain the corresponding monitoring information.

[0021] Furthermore, in S43, the Flink configuration file is configured as follows:

[0022] Configure the Flink metrics reporter in flink-conf.yml:

[0023] # metrics configuration

[0024] metrics.reporter.grph.class: org.apache.flink.metrics.graphite.GraphiteReporter

[0025] metrics.reporter.grph.host: ${host}

[0026] metrics.reporter.grph.port: ${port}

[0027] metrics.reporter.grph.protocol: TCP

[0028] # Runtime specification

[0029] metrics.reporter.grph.prefix="flink.${JOB_NAME}

[0030] metrics.latency.interval: 30000.

[0031] Preferably, S5 specifically includes:

[0032] S51. The FlinkSQL uses the fields cached by Flink and the corresponding regular expression rules to execute different FlinkSQL statements through different regular expression matching to analyze the current data situation;

[0033] S52. The Flink custom operator checks whether the common cases and their associated fields are consistent.

[0034] Preferably, the statistical results are written to a NoSQL database in S6.

[0035] Preferably, the database in S1 is an unstructured database.

[0036] Secondly, this invention also discloses a Flink-based streaming data quality monitoring system, comprising:

[0037] Design module: Used to design the field rule data to be generated according to preset rules;

[0038] Storage module: Used to store the pre-generated field rule data into the database in JSON format;

[0039] Task module: Used to start Flink tasks. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory;

[0040] The consumption module is used by the Flink program to start consuming data from the message queue after Flink has finished parsing the field rule data.

[0041] Monitoring module: Used to trigger the Flink program to start monitoring of the amount of data processed by the message queue, so as to count the amount of data processed at the current time;

[0042] Analysis module: Used to start Flink to analyze the current data using FlinkSQL and custom operators;

[0043] Statistics module: Used by the Flink program to perform statistics on the current data being analyzed and write the results into the database.

[0044] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0046] Compared with the prior art, the beneficial results of the present invention are as follows:

[0047] By combining components and designing operators in Flink, streaming data can be monitored, ensuring and monitoring its quality to a certain extent. Furthermore, it can be flexibly modified according to current business needs through custom operators and rules. Attached Figure Description

[0048] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0049] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied;

[0050] Figure 2 This is a flowchart illustrating the Flink-based streaming data quality monitoring method according to an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram illustrating the specific implementation of the Flink-based streaming data quality monitoring method according to an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of the database rule table design in the Flink-based streaming data quality monitoring method according to an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram illustrating the design of the database detection result table in the Flink-based streaming data quality monitoring method according to an embodiment of the present invention.

[0054] Figure 6 This is a schematic diagram of the structure of a Flink-based streaming data quality monitoring system according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0056] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0057] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0058] Figure 1 An exemplary system architecture 100 for processing information, or for processing information, to which embodiments of the present invention can be applied, is shown.

[0059] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0061] Terminal devices 101, 102, and 103 can be various electronic devices with communication functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0062] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain processing results (such as verification success information used to indicate that the verification request is a valid request).

[0063] It should be noted that the information processing method provided in the embodiments of the present invention is generally executed by server 105, and correspondingly, the device for processing information is generally disposed in server 105. Furthermore, the information sending method provided in the embodiments of the present invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the device for sending information is generally disposed in terminal devices 101, 102, and 103.

[0064] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0065] Currently, most data monitoring is based on data warehouses, or uses data warehouses combined with technologies such as Hudi and SQL scripts for data quality monitoring. The main drawback is that there is no quality monitoring for streaming data.

[0066] This invention primarily addresses the issue of data quality monitoring for Flink-based streaming data, mainly focusing on issues such as data integrity and standardization. The invention provides a solution for monitoring the quality of streaming data by designing a monitoring message queue for data consumption, generating field rules, designing custom operators, and using Flink programs and FlinkSQL syntax to analyze whether the data meets the currently defined data specifications. Finally, the calculation results are written to a database.

[0067] Figure 2 An embodiment of the present invention discloses a streaming data quality monitoring method based on Flink, such as... Figure 2 and Figure 3 As shown, the method includes the following steps:

[0068] S1. Store the pre-generated field rule data in the database in JSON format;

[0069] Among them, the main design point of field specification is to design specifications for commonly used fields. Considering that the stored data is unstructured data and the performance of the database, it is recommended to choose an unstructured database for storing the generation rules.

[0070] Specifically, field specifications utilize regular expressions to check data validity, primarily including:

[0071] 1. The standard regularity of Chinese resident identity card ([1-9]\d{5}(18|19|20)\d{2}((0[1-9])|10|11|12)(([0-2][1-9])|10|20|30|31)\d{3}[0-9Xx];

[0072] 2. Email regular [A-Za-z0-9\u4e00-\u9fa5]+@[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+;

[0073] 3. Regular expression for mobile phone number: ^1(3\d|4[5-9]|5[0-35-9]|6

[2567] |7[0-8]|8\d|9[0-35-9])\d{8}$, this mainly refers to whether it is a Chinese mobile phone number;

[0074] 4. IP primarily verifies the validity of IPv4 addresses. Its regular expression is:

[0075] (1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|[1-9])\.(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|\d) \.(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|\d)\.(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|\d)$;

[0076] 5. Regular expression for zip code: ^[0-9]{6}$;

[0077] 6. Regular expression for age: / ([0-9][0-9]) / ;

[0078] 7. The regular expression for the standard date format YY-MM-DD is: ^\d{4}-\d{1,2}-\d{1,2};

[0079] 8. Regular expression rules for virtual identities, such as QQ's regular expression ^[1-9][0-9]{4,10}$, and WeChat's regular expression ^[a-zA-Z]([-_a-zA-Z0-9]{5,19})+$, etc.;

[0080] The SQL statement is used to write data into the corresponding fields according to the JSON pattern [{field:field name, rule:regular expression}]. The subsequent program checks whether the fields conform to the specification using regular expressions. It can also check whether related fields conform to the specification, such as the format of the certificate, to determine whether the corresponding certificate type or the date of birth is correct.

[0081] S2. Flink starts the task. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory.

[0082] Specifically, when a Flink task starts consuming data from the message queue, it first triggers the Flink program to read data from the rule table in the database, then parses the JSON data and caches the data in memory. The specific implementation methods include:

[0083] S21. To start a Flink task to read data from the corresponding table in the database, you need to first configure the database user for Flink to use.

[0084] S22. Use Flink CDC to read the rule data of the fields in the database, and use Flink code configuration to read the data of the NoSQL database;

[0085] S23. The Flink program parses the JSON data, reads the corresponding fields and regular expressions, caches them in memory, and provides them to Flink's custom operators and FlinkSQL for invocation.

[0086] S3. After Flink finishes parsing the field rule data, the Flink program starts consuming the data from the message queue.

[0087] S4. While the Flink program consumes the message queue data, it triggers the Flink program to start monitoring the amount of data processed by the message queue in order to count the amount of data processed at the current time.

[0088] Specifically, to ensure the consistency of streaming data, the amount of data consumed from the Flink message queue needs to be compared with the amount of data processed to determine if any data loss has occurred. This includes:

[0089] S41. Use Flink's Metrics to monitor metrics and customize the CustomerJsonDeserialization class by overriding the initMertic method.

[0090] The `initMertic` method retrieves information such as the amount of data consumed by Flink and the message queue's metadata, especially the calculation of the consumed data volume, to obtain the number of messages read from the Flink Source. This allows you to clearly understand the current amount of data processed by Flink.

[0091] S42. Although the obtained Kafka metadata and offset information are reported by default, Flink needs to be configured to enable the Flink program to obtain the corresponding monitoring information.

[0092] To enable Flink to automatically report metric information, you need to configure it in the Flink configuration file, as follows:

[0093] Configure the Flink metrics reporter in flink-conf.yml:

[0094] # metrics configuration

[0095] metrics.reporter.grph.class:

[0096] org.apache.flink.metrics.graphite.GraphiteReporter

[0097] metrics.reporter.grph.host: ${host}

[0098] metrics.reporter.grph.port: ${port}

[0099] metrics.reporter.grph.protocol: TCP

[0100] # Runtime specification

[0101] metrics.reporter.grph.prefix="flink.${JOB_NAME}

[0102] metrics.latency.interval: 30000.

[0103] S5 and Flink analyze the current data using FlinkSQL combined with custom operators;

[0104] Specifically, let's further elaborate on the content of S5:

[0105] The main task of S51 and FlinkSQL is to analyze the data for this dimension by using the previously cached fields of Flink and the corresponding regular expression rules, and executing different FlinkSQL statements through different regular expression matching.

[0106] This approach is primarily implemented using a custom SQL submission client called Flink-SQL-submit.

[0107] S52, a Flink custom operator, is mainly used for common business scenarios and to check whether the associated fields of the fields match.

[0108] Common uses include checking whether some common fields are empty, and whether there is dirty data in the phone number, such as Chinese characters.

[0109] Business status checks include whether the data entry time conforms to time specifications, whether it is earlier than the business time, and whether business fields meet certain specified specifications, such as whether certain business fields start with a certain prefix.

[0110] Standardize the field association and specify field detection, such as whether the document type associated with resident ID card information is correct, whether the birth region or birth date is consistent with the information extracted by the rules, or analyze the mobile phone number regular expression to determine whether the corresponding region, operator or area code is consistent, etc.

[0111] S6. The Flink program performs statistics on the current data analysis and writes the statistical results into the database.

[0112] Specifically, the database needs automatically generated column names. The table structure of a relational database is generated when it is created and cannot be changed. Therefore, a NoSQL database is chosen to store structured data.

[0113] The general implementation principle of this invention is as follows: When Flink starts a task to process data, Flink reads the JSON data rules set in the rule fields of the database table, parses them, and caches them in memory. Then, it reads the message queue data and triggers Flink to monitor the data in the message queue. After Flink processes the data, it verifies it according to the set detection rules. When the time interval of Flink's time window ends, it submits the calculation results within the time window to the database.

[0114] Furthermore, refer to Figure 2 and Figure 3 The specific implementation steps are as follows:

[0115] S1. A database is used to store the rule data for the fields. This invention uses MongoDB as an example. The field rule table is referenced. Figure 4 The database rules table is designed as follows: the rules field is mainly used to store the rule fields and their corresponding regular expressions as JSON data; the relation field is mainly used to store the relationships between fields; and the com field is mainly used to store fields that are checked for non-empty data.

[0116] S2. When the Flink task starts, the Flink program begins to read the JSON data rules set in the database table, parses them, and caches them in memory.

[0117] S21. Flink reads data from the corresponding table in the database. This invention uses MongoDB as an example, configuring the database user for Flink use. The authorization script configuration is as follows:

[0118] db.createUser({

[0119] user: "flinkuser",

[0120] pwd: "password",

[0121] roles: [

[0122] { role: "read", db: "admin"},

[0123] { role: "readAnyDatabase", db: "admin"} ]

[0125] });

[0126] S22. Flink connects to MongoDB to read data and parses the data, caching it in memory.

[0127] S3. Once Flink has finished parsing the rule fields, it starts consuming data from the message queue and simultaneously triggers the Flink program's monitoring mode for the message queue. This invention takes Kafka as an example to obtain metadata and offset information of the message queue.

[0128] S4. FlinkSQL, combined with Flink custom operators, analyzes and tests specified fields based on field detection rules and custom operator business requirements;

[0129] The S5.Flink program summarizes the calculations from each dimension and writes the detection results into a specified table in the database. See the appendix for details. Figure 5 In the database detection result table design, this invention uses MongoDB as an example. The source field in the table records the message queue name information, the field1 and other fields are automatically generated based on the statistical dimension fields, the count field is the amount of data processed by the program, and catm records the business time or the time of data entry.

[0130] Secondly, embodiments of the present invention also disclose a streaming data quality monitoring system based on Flink, such as... Figure 6 As shown, it includes:

[0131] Design module 61: Used to design the field rule data to be generated according to preset rules;

[0132] Storage module 62: Used to store the pre-generated field rule data into the database in JSON format;

[0133] Task module 63: Used to start a Flink task. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory.

[0134] Consumer module 64: Used to enable the Flink program to start consuming data from the message queue after Flink has finished parsing the field rule data;

[0135] Monitoring module 65: used to trigger the Flink program to start monitoring of the amount of data processed by the message queue, so as to count the amount of data processed at the current time;

[0136] Analysis Module 66: Used to start Flink to analyze the current data using FlinkSQL and custom operators;

[0137] Statistics module 67: Used by the Flink program to perform statistics on the current data being analyzed and write the statistical results into the database.

[0138] This invention enables data monitoring of streaming data through the combination of Flink components and design operators, ensuring and monitoring the quality of streaming data to a certain extent. It can be flexibly changed according to the current business situation through custom operators and custom rules.

[0139] The key technical point of this invention is the implementation design scheme for monitoring the data quality of Flink streaming data. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the protection scope of this invention.

[0140] The following is for reference. Figure 7 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 A schematic diagram of the structure of a computer device 600 (shown as a server or terminal device). Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0141] like Figure 7 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 603 or programs loaded from storage section 609 into random access memory (RAM) 606. Various programs and data required for the operation of device 600 are also stored in RAM 604. The CPU 601, GPU 602, ROM 603, and RAM 604 are interconnected via bus 605. Input / output (I / O) interface 606 is also connected to bus 605.

[0142] The following components are connected to I / O interface 606: an input section 607 including a keyboard, mouse, etc.; an output section 608 including an LCD, speakers, etc.; a storage section 609 including a hard disk, etc.; and a communication section 610 including a network interface card, such as a LAN card or modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to I / O interface 606 as needed. A removable medium 612, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 611 as needed so that computer programs read from it can be installed into storage section 609 as needed.

[0143] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 610, and / or installed from removable medium 612. When the computer program is executed by central processing unit (CPU) 601 and graphics processing unit (GPU) 602, the functions defined above in the methods of the present invention are performed.

[0144] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0145] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0147] The modules described in the embodiments of the present invention can be implemented in software or in hardware. The described modules can also be located in a processor.

[0148] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the method steps as described in the first aspect of the present invention.

[0149] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for monitoring the quality of streaming data based on Flink, characterized in that, The method includes the following steps: S1. Store the pre-generated field rule data in the database in JSON format; S2. Flink starts the task. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory. S3. After Flink finishes parsing the field rule data, the Flink program starts consuming the data from the message queue. S4. While the Flink program consumes the message queue data, it triggers the Flink program to start monitoring the amount of data processed by the message queue in order to count the amount of data processed at the current time. S5 and Flink analyze the current data using FlinkSQL combined with custom operators; S6. The Flink program performs statistics on the current data analysis and writes the statistical results into the database; Specifically, S4 includes: S41, overriding the initMertic method in the CustomerJsonDeserialization class by customizing the monitoring metrics of Flink's Metrics; S42, further configuring the Flink program to obtain the corresponding monitoring information based on the default information reporting function relationship of Flink. S5 specifically includes: S51, the FlinkSQL executes different FlinkSQL statements by matching different regular expressions through the fields cached by Flink and the corresponding regular expression rules, and analyzes the current data situation; S52, the Flink custom operator detects whether common cases and their associated fields are consistent.

2. The streaming data quality monitoring method based on Flink according to claim 1, characterized in that, S2 specifically includes: S21. To start a Flink task to read data from the corresponding table in the database, you need to first configure the database user for Flink to use. S22. Use Flink CDC to read the field rule data in the database, and use Flink code configuration to read data from the NoSQL database; S23. The Flink program parses the JSON data, reads the corresponding fields and regular expressions, caches them in memory, and provides them to Flink's custom operators and FlinkSQL for invocation.

3. The streaming data quality monitoring method based on Flink according to claim 1, characterized in that, The Flink configuration file is configured as follows: Configure the Flink metrics reporter in flink-conf.yml: # metrics configuration metrics.reporter.grph.class: org.apache.flink.metrics.graphite.GraphiteReporter metrics.reporter.grph.host: ${host} metrics.reporter.grph.port: ${port} metrics.reporter.grph.protocol: TCP # Runtime specification metrics.reporter.grph.prefix="flink.${JOB_NAME} metrics.latency.interval: 30000.

4. The streaming data quality monitoring method based on Flink according to claim 1, characterized in that, In S6, the statistical results are written to a NoSQL database.

5. The streaming data quality monitoring method based on Flink according to claim 1, characterized in that, The database mentioned in S1 is an unstructured database.

6. A streaming data quality monitoring system based on Flink, characterized in that, Including the method as described in any one of claims 1-5, further comprising: Design module: Used to design the field rule data to be generated according to preset rules; Storage module: Used to store the pre-generated field rule data into the database in JSON format; Task module: Used to start Flink tasks. The Flink program reads the field rule data from the database, further parses the JSON data, and caches it in memory; The consumption module is used by the Flink program to start consuming data from the message queue after Flink has finished parsing the field rule data. Monitoring module: Used to trigger the Flink program to start monitoring of the amount of data processed by the message queue, so as to count the amount of data processed at the current time; Analysis module: Used to start Flink to analyze the current data using FlinkSQL and custom operators; Statistics module: Used by the Flink program to perform statistics on the current data being analyzed and write the results into the database.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images