A streaming data quality monitoring method based on Flink SQL

Abstract

The present invention relates to the field of data processing technology, and in particular to a streaming data quality monitoring method based on Flink SQL, including: constraining data in a data table with preset field rule data; reading and parsing the field rule data in the database through a Flink application; analyzing the real-time data situation through Flink SQL in combination with a custom operator; the Flink application statistics and stores the data situation analysis results; and initially determining the streaming data quality when Flink SQL in combination with a custom operator analyzes the real-time data situation, and periodically analyzing the analysis results in the database to determine the streaming data quality again. The present invention can monitor streaming data through Flink's combined components and designed operators, ensure and monitor the streaming data quality to a certain extent, and can flexibly change through custom operators and custom rules according to current business conditions.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a streaming data quality monitoring method based on Flink SQL. Background Art

[0002] Based on the business or usage scenarios of the data platform, the data platform needs to process a large amount of data to provide support for various businesses. How to ensure data quality has become a problem that the data platform needs to solve. The reasons for data quality can be mainly divided into the following aspects. On the one hand, a data platform may use different technologies and tools to process data, and the data quality after processing cannot be ensured in the process of various tools; on the other hand, when Flink is selected to process data, although Flink has a certain mechanism to ensure data problems, it cannot ensure the accuracy, completeness, consistency, uniqueness and standardization of the data in Flink processing. For streaming data, it is generally difficult to monitor the integrity and standardization of data, which will lead to the inability to discover data problems in time, resulting in the impact of data quality. At present, most data monitoring is based on data warehouses, or data warehouses are used in combination with hudi, SQL scripts and other technologies for data quality monitoring. When the data platform uses different technologies and tools for data processing, data quality consistency problems may occur. Different tools and technologies may use different algorithms, logic or rules, resulting in differences in the quality of processed data. For example, in the data cleaning stage, different cleaning tools may produce different results, which may affect the consistency and accuracy of the final data. Currently, most data monitoring methods are mainly based on data warehouses, and relatively few are used to monitor streaming data. Streaming data is characterized by fast data flow, large data volume, and diverse data sources, so ensuring the quality of streaming data is more challenging.

[0003] Chinese Patent Publication No.: CN112667614A discloses a data processing method, device and computer equipment. The method includes: obtaining data to be processed, adding the data to be processed to a first data message queue; processing each of the data to be processed in the first data message queue in a streaming data processing manner based on the Flink real-time processing framework to obtain queue processing data; adding the queue processing data to a second data message queue; and performing real-time data processing on each of the queue processing data in the second data message queue based on the Flink real-time processing framework.

[0004] It can be seen that, during the current data processing, there is a lack of data processing technology for monitoring the quality of streaming data, and the quality of the processed streaming data cannot be determined in a timely manner. Summary of the invention

[0005] To this end, the present invention provides a streaming data quality monitoring method based on Flink SQL, including a method for overcoming the problem that in the prior art, there is a lack of a data processing process for monitoring the quality of streaming data during current data processing, and the quality of the processed streaming data cannot be determined in a timely manner.

[0006] To achieve the above object, the present invention provides a streaming data quality monitoring method based on Flink SQL, comprising:

[0007] S1, constrain the data in the data table with the preset field rule data and store it in the database in JSON format;

[0008] S2, start the Flink task, read the field rule data in the database through the Flink application, parse the data stored in JSON format, and cache it in memory;

[0009] S3: After the Flink application completes data parsing, it starts consuming message queue data.

[0010] S4, while consuming the message queue data, the Flink application monitors the amount of data processed by the message queue and counts the amount of data processed in real time;

[0011] S5: Analyze real-time data using Flink SQL combined with custom operators.

[0012] S6, the Flink application performs statistics on the data situation analysis results and stores the statistical results in a database;

[0013] When Flink SQL combines custom operators to analyze real-time data, it initially determines the quality of streaming data, and periodically analyzes the analysis results in the database to determine the quality of streaming data again.

[0014] Furthermore, the constraints imposed by the field rule data on the data in the data table include constraints on the type, length, and format of each field in the data table.

[0015] Furthermore, the initial determination of the streaming data quality includes determining the consistency of the fields of the constrained data and determining the field consistency fluctuation.

[0016] Furthermore, when determining the consistency of a field, parameters for determining the difference of each constraint item are set, and streaming data quality monitoring is performed by integrating the single difference of each constraint item with the overall difference.

[0017] Furthermore, when determining the consistency of a field, if the data of multiple threads is monitored, the data with pending results is analyzed based on the total number of threads and the number of threads with problems to determine the data quality.

[0018] Furthermore, when determining the consistency fluctuation, the quality of the streaming data is monitored by cumulatively analyzing the integration results of the data that have undergone consistency determination.

[0019] Furthermore, when determining the consistency fluctuation, each segment of data that has passed the consistency determination is integrated. During the integration, a calculation compensation parameter for determining the consistency fluctuation is set for any segment of data that has passed the consistency determination, and the set calculation compensation parameter is related to the output content of the consistency determination.

[0020] Furthermore, the analysis results in the database are periodically analyzed, including periodically performing cumulative analysis on the streaming data based on the data results of the initial determination of the streaming data quality, wherein the data results of the initial determination of the streaming data quality are presented through expression parameters.

[0021] Furthermore, when performing the periodic analysis, a calculation adjustment value is set for any of the expression parameters, and the calculation adjustment values ​​of different expression parameters are different.

[0022] Furthermore, the database is an unstructured database.

[0023] S2 includes,

[0024] S21. Flink starts a task to read the data in the corresponding table of the database. You need to configure the database user for Flink to use.

[0025] S22. Use FlinkCDC to read the field rule data in the database, and use Flink code configuration to read the data of the NOSQL database;

[0026] S23. Parse the JSON data, read the corresponding fields, perform regular cache in the memory, and provide them to Flink's custom operators and Flink SQL for calling.

[0027] S4 specifically includes:

[0028] S41. Use Flink's Metrics monitoring indicators to customize the CustomerJsonDeserialization class and rewrite the initMertic method.

[0029] S42. According to the functional relationship of Flink to report the acquired information by default, further configuration is performed to enable the Flink program to acquire the corresponding monitoring information.

[0030] Compared with the prior art, the beneficial effect of the present invention is that the streaming data can be monitored through the combination of Flink components and the design of operators, and the streaming data quality can be guaranteed and monitored to a certain extent, and can be flexibly changed according to the current business situation through customized operators and customized rules. By setting up multi-level analysis and judgment, the data flow quality is determined to ensure the accuracy of the data flow quality determination.

[0031] Furthermore, by standardizing the data in the data table, standard indicators can be obtained when analyzing the data, and unified analysis and determination of multiple categories of data can be performed, further ensuring the accuracy of data flow quality determination.

[0032] Furthermore, for different categories of data, output association items with corresponding standard constraints are set to further ensure the accuracy of data flow quality determination.

[0033] Furthermore, for any data segment to be analyzed, if its field type is determined, data constraints are imposed on it under the built-in algorithm of the present application, and the data field length and data format of the same type of data are standardized. For any data segment to be analyzed, if its format is different from its corresponding standard format, it is judged as unqualified in quality. If the format is the same, it is judged by the field length. At the same time, for individual data, if it appears at a node of a large data segment, its length may be affected. Moreover, for some data, slight data loss has no effect on the overall data flow processing result. Therefore, certain restrictions are imposed on the field length. For field lengths that are inconsistent with the standard length, a one-size-fits-all approach is not adopted, but the quality of the data segment is determined through comprehensive analysis, so that the quality judgment is more accurate.

[0034] Furthermore, if the field length ratio falls within the field length ratio range, it means that the deviation value is relatively small. At this time, the corresponding data is judged to be data of pending quality. When it does not fall within the field length ratio range, it means that the deviation value is relatively large. When the overall stream data is processed and analyzed, the deviation value has a greater impact on the overall analysis results. Therefore, for the field length ratio that does not fall within the field length ratio range, the corresponding data is judged to be data of unqualified quality, so that the quality judgment is more accurate.

[0035] Furthermore, for multi-threaded data processed simultaneously, the data quality of the processed stream data is determined by analyzing the data processing results of multiple threads as a whole, so that the quality judgment can be more accurate. When threads are processed simultaneously, the value of the evaluation parameter is adjusted according to the number of unqualified data threads. When there are quality problems in the data of multiple threads, it means that the proportion of data with quality problems is relatively large. At this time, the value of the evaluation parameter of the proportion of problem threads is reduced, so that the data quality judgment is more reasonable and the quality judgment is more accurate.

[0036] Furthermore, when there are problems with the data, the problems often have a certain continuity. By collecting small-scale data analysis results on the data processed by the same thread and making a comprehensive judgment, the judgment on the single thread data can be more reasonable, and the data with pending quality can be judged more accurately, making the quality judgment more accurate.

[0037] Furthermore, for data of a single thread, by associating the result of the field length ratio with its corresponding calculation difference determination parameter, the influence of a single field length ratio on the comprehensive deviation value is increased, so that the calculation amount can be determined more accurately.

[0038] Furthermore, when the comparison results of the comprehensive deviation value of a single thread and the comprehensive deviation value evaluation value are consistent, it means that the data calculated and analyzed by the corresponding thread is relatively stable. At this time, the data stored in the database is not analyzed, and the speed of real-time data processing is increased. When there is inconsistency in the comparison results, it means that the analyzed data has a certain volatility. Then, the stored data is analyzed again to determine the data quality. This application can not only perform quality analysis on real-time data, but also conduct retrospective analysis on the data that has been passed according to the analysis results, so as to further make the quality determination more accurate. By retrospectively analyzing the data comparison results of multiple time periods and introducing variable calculation adjustment values ​​during the retrospective analysis, the calculation results are made more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a streaming data quality monitoring method based on Flink SQL in an embodiment;

[0040] Figure 2 A schematic diagram of a flow chart of consistency determination of fields of constrained data in an embodiment;

[0041] Figure 3 It is a schematic diagram of the process of analyzing multi-threaded data in an embodiment. DETAILED DESCRIPTION

[0042] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0044] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0045] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] See also Figure 1-Figure 3 As shown, Figure 1 It is a flowchart of a streaming data quality monitoring method based on Flink SQL in an embodiment; Figure 2 A schematic diagram of a flow chart of consistency determination of fields of constrained data in an embodiment; Figure 3 It is a schematic diagram of the process of analyzing multi-threaded data in an embodiment.

[0047] The present invention provides a streaming data quality monitoring method based on Flink SQL, comprising:

[0048] S1, constrain the data in the data table with the preset field rule data and store it in the database in JSON format;

[0049] S2, start the Flink task, read the field rule data in the database through the Flink application, parse the data stored in JSON format, and cache it in memory;

[0050] S3: After the Flink application completes data parsing, it starts consuming message queue data;

[0051] S4, while consuming the message queue data, the Flink application monitors the amount of data processed by the message queue and counts the amount of data processed in real time;

[0052] S5, analyze the real-time data situation through Flink SQL combined with custom operators;

[0053] S6, the Flink application performs statistics on the data situation analysis results and stores the statistical results in a database;

[0054] When Flink SQL combines custom operators to analyze real-time data, it initially determines the quality of streaming data, and periodically analyzes the analysis results in the database to determine the quality of streaming data again.

[0055] Through Flink's combined components and designed operators, streaming data can be monitored, and the quality of streaming data can be guaranteed and monitored to a certain extent. It can also be flexibly changed through custom operators and custom rules according to the current business situation. By setting up multi-level analysis and judgment, the quality of data flow can be determined to ensure the accuracy of data flow quality determination.

[0056] S2 includes,

[0057] S21. Flink starts a task to read the data in the corresponding table of the database. You need to configure the database user for Flink to use.

[0058] S22. Use FlinkCDC to read the field rule data in the database, and use Flink code configuration to read the data of the NOSQL database;

[0059] S23. Parse the JSON data, read the corresponding fields, perform regular cache in the memory, and provide them to Flink's custom operators and Flink SQL for calling.

[0060] S4 specifically includes:

[0061] S41. Use Flink's Metrics monitoring indicators to customize the CustomerJsonDeserialization class and rewrite the initMertic method.

[0062] S42. According to the functional relationship of Flink to report the acquired information by default, further configuration is performed to enable the Flink program to acquire the corresponding monitoring information.

[0063] The constraints imposed by field rule data on the data in the data table include the type, length, and format of each field in the data table.

[0064] By standardizing the data in the data table, standard indicators can be obtained when analyzing the data, and unified analysis and determination of multiple categories of data can be performed to further ensure the accuracy of data flow quality determination.

[0065] When performing streaming data quality monitoring, quality determination is performed through a data quality assessment module, in which corresponding standard field lengths and standard field formats are set according to different field types.

[0066] The field type includes a first type field A1, a second type field A2, and a third type field A3, wherein the first type field A1 corresponds to a first standard field length B1 and a first standard field format C1, the second type field A2 corresponds to a second standard field length B2 and a second standard field format C2, and the third type field A3 corresponds to a third standard field length B3 and a third standard field format C3;

[0067] In this embodiment, the field types include character type, numeric type, and time type.

[0068] For different categories of data, set the output association items with corresponding standard constraints to further ensure the accuracy of data flow quality determination.

[0069] The consistency determination of the fields of the constrained data includes, for any constrained data Hk, determining its type field Ai, i=1,2,3 by analysis, obtaining the field length Bk and field format Ck of the constrained data Hk,

[0070] If the field length Bk is consistent with the i-th standard field length Bi and the field format Ck is consistent with the i-th standard field format Ci, the data Hk is judged to be of qualified quality;

[0071] If the field format Ck is inconsistent with the i-th standard field format Ci, the data Hk is judged to be of unqualified quality;

[0072] If the field length Bk is inconsistent with the i-th standard field length Bi, and the field format Ck is consistent with the i-th standard field format Ci, the field length is analyzed to determine the quality of the data Hk;

[0073] The data quality assessment module calculates the field length ratio Zk, Zk=Bk÷Bi. A field length ratio range is set in the data quality assessment module. If Zk does not fall within the field length ratio range, the quality of the data Hk is determined to be unqualified. If Zk falls within the field length ratio range, the quality of the data Hk is determined to be undetermined.

[0074] For any data segment to be analyzed, if its field type is determined, data constraints are imposed on it under the built-in algorithm of this application. The data field length and data format of the same type of data are standardized. For any data segment to be analyzed, if its format is different from its corresponding standard format, it is judged as unqualified in quality. If the format is the same, it is judged by the field length. At the same time, for individual data, if it appears at the node of a large data segment, its length may be affected. Moreover, for some data, slight data loss has no effect on the overall data flow processing results. Therefore, certain restrictions are imposed on the field length. For field lengths that are inconsistent with the standard length, a one-size-fits-all approach is not adopted. Instead, the quality of the data segment is determined through comprehensive analysis, making the quality judgment more accurate.

[0075] If the field length ratio falls within the range of the field length ratio, it means that the deviation is relatively small. In this case, the corresponding data is judged as data of pending quality. If it does not fall within the range of the field length ratio, it means that the deviation is relatively large. When the overall stream data is processed and analyzed, the deviation has a greater impact on the overall analysis results. Therefore, the corresponding data of the field length ratio that does not fall within the range of the field length ratio is judged as data of unqualified quality, so that the quality judgment is more accurate.

[0076] For the same time node, the Flink application can analyze multi-threaded data. If the number of threads analyzed is n, the data quality assessment module integrates the total number of threads performing data analysis, the number of threads whose data results are determined to be of pending quality, and the number of threads whose data results are determined to be of unqualified quality, and re-evaluates the data whose quality is pending.

[0077] Assume that the total number of threads is n, the number of threads whose quality is pending is n1, the number of threads whose quality is unqualified is n2, and calculate the proportion of problem threads Np;

[0078] Set Np=(n1+n2)÷n, and set the problematic thread ratio evaluation parameter NQ in the data quality assessment module.

[0079] If Np≥NQ, the data quality corresponding to the thread whose quality is to be determined is determined to be unqualified.

[0080] For multi-threaded data processed simultaneously, the data quality of the processed stream data is determined by comprehensively analyzing the data processing results of multiple threads, making the quality determination more accurate.

[0081] The value of the problem thread ratio evaluation parameter NQ is related to the value of n2, and the larger the value of n2 is, the smaller the value of the problem thread ratio evaluation parameter NQ is.

[0082] When threads are processed simultaneously, the value of the evaluation parameter is adjusted according to the number of unqualified data threads. When there are quality problems with the data of multiple threads, it means that the proportion of data with weight problems is relatively large. At this time, the value of the evaluation parameter of the proportion of problem threads is reduced to make the data quality judgment more reasonable and the quality judgment more accurate.

[0083] The field consistency fluctuation includes that, for the same thread data, the data quality assessment module records the field length ratio determined as pending quality, and records the field length ratio of the data determined as pending quality within the first unit time length t1, generates a correlation curve Qt1 of the field length ratio for the first unit time length t1, and the data quality assessment module analyzes the correlation curve Qt1 and calculates the comprehensive deviation value Zq of the correlation curve Qt1,

[0084] Zq= ×p, Zkj is the field length ratio corresponding to time node j on the association curve Qt1, and p is the difference judgment parameter between Zkj and the comprehensive deviation value Zq. The data quality assessment module is provided with a comprehensive deviation value evaluation value Zp. If Zq>Zp, the data with pending quality within the first unit time length t1 is judged as unqualified data.

[0085] In this embodiment, Zp=1.2.

[0086] When there are problems with the data, the problems often have a certain degree of continuity. By collecting small-scale data analysis results on the data processed by the same thread and making a comprehensive judgment, the judgment on the single thread data can be more reasonable, and the data with pending quality can be judged more accurately, making the quality judgment more accurate.

[0087] For the difference judgment parameter p, its value is related to the value of Zkj, and it is set that p=|Zkj-1|×a+P1, where P1 is the basic value of the difference judgment parameter p, and a is the compensation value of Zkj to the difference judgment parameter p.

[0088] In this embodiment, P1=0.1, a=0.73.

[0089] For single-thread data, by associating the result of the field length ratio with the value of the corresponding difference judgment parameter, the influence of a single field length ratio on the comprehensive deviation value is increased, making the calculation amount judgment more accurate.

[0090] The data quality assessment module is provided with a second unit time length t2, t2=m×t1, m is a positive integer, and m≥3. The periodic analysis of the analysis results in the database includes that for any single thread, when the data analysis results stored in the database reach the second unit time length t2, the data quality assessment module divides the second unit time length t2 into a number of segmented time periods according to the first unit time length t1. For any segmented time period, the data quality assessment module compares the size relationship between the comprehensive deviation value of the corresponding correlation curve and the comprehensive deviation value evaluation value.

[0091] If all the comprehensive deviation values ​​are greater than the comprehensive deviation value evaluation value or all the comprehensive deviation values ​​are less than the comprehensive deviation value evaluation value, it is determined that the data difference fluctuations of the corresponding threads are consistent, and the data stored in the database are not analyzed;

[0092] If some comprehensive deviation values ​​are greater than the comprehensive deviation evaluation value, and some comprehensive deviation values ​​are less than the comprehensive deviation evaluation value, the stored data will be analyzed again according to the comprehensive deviation value.

[0093] When the comparison results of the comprehensive deviation value of a single thread and the comprehensive deviation value evaluation value are consistent, it means that the data calculated and analyzed by the corresponding thread is relatively stable. At this time, the data stored in the database is not analyzed, and the speed of real-time data processing is increased. When there are inconsistencies in the comparison results, it means that the analyzed data has a certain volatility. The stored data is analyzed again to determine the data quality. This application can not only perform quality analysis on real-time data, but also conduct retrospective analysis on the data that has been analyzed based on the analysis results, thereby further making the quality judgment more accurate.

[0094] In this embodiment, the expression parameter is a comprehensive deviation value of any divided time period.

[0095] The data quality assessment module calculates the comprehensive deviation value periodic change score Fz,

[0096] Fz= ×r, where Zqs is the comprehensive deviation value of any divided time period, r is the calculated adjustment value of the comprehensive deviation value to the comprehensive deviation value periodic change score, and the data quality assessment module is provided with a comprehensive deviation value periodic change evaluation value Fp. If Fz>Fp, the stored quality pending data is judged to be unqualified data.

[0097] set up,

[0098] When Zqs<Zp, r=0;

[0099] When Zqs=Zp, r=1;

[0100] When Zqs>Zp, r=1+(Zqs÷Zp).

[0101] By performing retrospective analysis on the data comparison results of multiple time periods and introducing variable calculation adjustment values ​​during the retrospective analysis, the calculation results are made more accurate.

[0102] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A streaming data quality monitoring method based on Flink SQL, characterized in that: include, Constrain the data in the data table using the preset field rule data and store it in the database in JSON format; Start the Flink task, read the field rule data in the database through the Flink application, parse the data stored in JSON format, and cache it in memory; After the Flink application completes data parsing, it starts consuming message queue data; While consuming the message queue data, the Flink application monitors the amount of data processed by the message queue and counts the amount of data processed in real time; Analyze real-time data using Flink SQL combined with custom operators; The Flink application collects statistics on the data situation analysis results and stores the statistical results in a database; When Flink SQL combines custom operators to analyze real-time data, it initially determines the quality of streaming data, and periodically analyzes the analysis results in the database to determine the quality of streaming data again. The initial determination of streaming data quality includes determining the consistency of fields of constrained data and determining the fluctuation of field consistency; The consistency determination of the field includes setting the difference determination parameters of each constraint item, integrating the single difference of each constraint item with the overall difference, and monitoring the streaming data quality; The consistency determination of the field also includes, if the data of multiple threads is monitored, analyzing the data with pending results in combination with the total number of threads and the number of threads with problems to determine the data quality; Assume that the total number of threads is n, the number of threads whose quality is pending is n1, the number of threads whose quality is unqualified is n2, and calculate the proportion of problem threads Np; Set Np=(n1+n2)÷n, and set the problematic thread ratio evaluation parameter NQ in the data quality assessment module. If Np ≥ NQ, the data quality corresponding to the thread whose quality is to be determined is judged to be unqualified; The value of the problem thread ratio evaluation parameter NQ is related to the value of n2, and the larger the value of n2, the smaller the value of the problem thread ratio evaluation parameter NQ; The determination of field consistency fluctuation includes monitoring streaming data quality by cumulatively analyzing the integration results of data that have undergone consistency determination; The periodic analysis of the analysis results in the database includes periodically performing cumulative analysis on the streaming data according to the data results of the initial determination of the streaming data quality, wherein the data results of the initial determination of the streaming data quality are presented through expression parameters; When performing the periodic analysis, a calculation adjustment value is set for any of the expression parameters, and the calculation adjustment values ​​of different expression parameters are different; The field consistency fluctuation includes that, for the same thread data, the data quality assessment module records the field length ratio determined as pending quality, and records the field length ratio of the data determined as pending quality within the first unit time length t1, generates a correlation curve Qt1 of the field length ratio for the first unit time length t1, and the data quality assessment module analyzes the correlation curve Qt1 and calculates the comprehensive deviation value Zq of the correlation curve Qt1; set, Zq= ×p, Zkj is the field length ratio corresponding to time node j on the association curve Qt1, and p is the difference judgment parameter of Zkj to the comprehensive deviation value Zq; the data quality assessment module is provided with a comprehensive deviation value evaluation value Zp, and if Zq>Zp, the data with pending quality within the first unit time length t1 is judged as unqualified data; for the difference judgment parameter p, its value is related to the value of Zkj, and it is set that p=|Zkj-1|×a+P1, wherein P1 is the basic value of the difference judgment parameter p, and a is the compensation value of Zkj to the difference judgment parameter p; The data quality assessment module is provided with a second unit time length t2, t2=m×t1, m is a positive integer, and m≥3. The periodic analysis of the analysis results in the database includes that for any single thread, when the data analysis results stored in the database reach the second unit time length t2, the data quality assessment module divides the second unit time length t2 into a number of segmented time periods according to the first unit time length t1. For any segmented time period, the data quality assessment module compares the size relationship between the comprehensive deviation value of the corresponding correlation curve and the comprehensive deviation value evaluation value. If all the comprehensive deviation values ​​are greater than the comprehensive deviation value evaluation value or all the comprehensive deviation values ​​are less than the comprehensive deviation value evaluation value, it is determined that the data difference fluctuations of the corresponding threads are consistent, and the data stored in the database are not analyzed; If there are both comprehensive deviation values ​​greater than the comprehensive deviation evaluation value and comprehensive deviation values ​​less than the comprehensive deviation evaluation value, the stored data are analyzed again according to the comprehensive deviation value; The data quality assessment module calculates the comprehensive deviation value periodic change score Fz, Fz= ×r, where Zqs is the comprehensive deviation value of any segmented time period, r is the calculated adjustment value of the comprehensive deviation value to the comprehensive deviation value periodic change score, and the data quality assessment module is provided with a comprehensive deviation value periodic change evaluation value Fp. If Fz>Fp, the stored quality pending data is judged to be unqualified data; set up, When Zqs<Zp, r=0; When Zqs=Zp, r=1; When Zqs>Zp, r=1+(Zqs÷Zp).

2. The streaming data quality monitoring method based on Flink SQL according to claim 1 is characterized in that: The constraints imposed by field rule data on the data in the data table include the type, length, and format of each field in the data table.

3. The streaming data quality monitoring method based on Flink SQL according to claim 2 is characterized in that: The consistency fluctuation determination of the field also includes integrating each segment of data that has passed the consistency determination. During the integration, a calculation compensation parameter for determining the consistency fluctuation is set for any segment of data that has passed the consistency determination, and the set calculation compensation parameter is related to the output content of the consistency determination.

4. The streaming data quality monitoring method based on Flink SQL according to claim 1 is characterized in that: The database is an unstructured database.

Images