Data quality monitoring method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data quality monitoring method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that data to be evaluated are extracted from service data of a distributed file system based on a preset extraction rule, the preset extraction rule is associated with dimension labels of the service data, and the distributed file system stores the service data of at least two different service systems; storing the to-be-evaluated data in the temporary table in a unified data format, traversing the to-be-evaluated data in the temporary table, and determining a data quality evaluation result of the business data; and displaying a data quality evaluation result, wherein the data quality evaluation result is used for performing exception processing on the business data. By adopting the method, the overall quality of the enterprise business data can be timely and comprehensively mastered.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data quality monitoring method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Data quality is crucial to enterprises and directly affects the accuracy of enterprise decision-making, the efficiency of business operations, etc. Data quality can be evaluated from multiple perspectives, including accuracy, timeliness, completeness, consistency, etc. Accuracy refers to the degree of closeness between the recorded data value and the true value, timeliness refers to whether the data is updated in a timely manner, completeness refers to whether the data is complete and intact and whether there are omissions, and consistency refers to whether data from different sources and at different time points remain consistent.

[0003] However, the amount of data that enterprises face is becoming increasingly large, the data sources are diverse, and the data flows through multiple links such as collection, storage, processing, analysis and application, making it difficult for enterprises to grasp the overall quality of data in a timely and comprehensive manner. Summary of the invention

[0004] Based on this, it is necessary to provide a data quality monitoring method, device, computer equipment, computer-readable storage medium and computer program product that can timely and comprehensively grasp the overall quality of enterprise business data in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for monitoring data quality, comprising:

[0006] Extracting the data to be evaluated from the business data of the distributed file system based on a preset extraction rule, wherein the preset extraction rule is associated with a dimension tag of the business data, and the distributed file system stores business data of at least two different business systems;

[0007] The data to be evaluated is stored in a temporary table in a unified data format, and the data to be evaluated in the temporary table is traversed to determine the data quality evaluation result of the business data;

[0008] The data quality assessment result is displayed, and the data quality assessment result is used to perform exception processing on the business data.

[0009] In one embodiment, before extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rule, the method includes:

[0010] Determine the dimension label of the business data based on the business scenario, flow stage and / or original data format of the business data corresponding to the business data.

[0011] In one embodiment, traversing the data to be evaluated in the temporary table to determine the data quality evaluation result of the business data includes:

[0012] Based on multiple first automated scripts, traverse the data to be evaluated in the temporary table to obtain multiple indicator data corresponding to the data to be evaluated respectively; the indicator data includes at least one of the following: data having a dependency relationship with the data to be evaluated, and attribute information of the data to be evaluated;

[0013] Based on the data to be evaluated and the indicator data, a data quality evaluation result of the business data is determined.

[0014] In one embodiment, before extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rule, the method includes:

[0015] Corresponding extraction rules are configured for multiple dimensional tags of business data respectively, and a corresponding second automation script is constructed based on the extraction rules; the second automation script is used to execute the steps of extracting the data to be evaluated from the business data stored in the database based on the preset extraction rules in response to business system initialization or configuration changes.

[0016] In one embodiment, presenting the data quality assessment result includes:

[0017] Synchronizing the data quality assessment results of the business data to a relational database;

[0018] The data quality assessment results in the relational database are displayed on a preset data display page.

[0019] In one embodiment, the method further comprises:

[0020] At each preset period, executing the step of extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rule;

[0021] The data quality assessment results of the business data are counted to obtain the data volume and abnormal type of the data to be processed in the business data, and an alarm is issued based on the data volume and abnormal type of the data to be processed.

[0022] In a second aspect, the present application also provides a data quality monitoring device, comprising:

[0023] A data extraction module, used to extract the data to be evaluated from the business data of the distributed file system based on a preset extraction rule, wherein the preset extraction rule is associated with a dimension tag of the business data, and the distributed file system stores business data of at least two different business systems;

[0024] A quality assessment module, used to store the data to be assessed in a temporary table in a unified data format, and traverse the data to be assessed in the temporary table to determine the data quality assessment result of the business data;

[0025] The data display module is used to display the data quality assessment results, and the data quality assessment results are used to perform exception processing on the business data.

[0026] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-described methods when executing the computer program.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the methods described above when executed by a processor.

[0028] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the methods described above when executed by a processor.

[0029] The above-mentioned data quality monitoring method, device, computer equipment, computer-readable storage medium and computer program product integrate the business data of multiple business systems through a distributed file system, which can facilitate the unified processing of business data of different dimensions and ensure the comprehensiveness of data quality assessment; by storing the data to be evaluated in a temporary table in a unified format, it is convenient to uniformly process the extracted data to be evaluated, and ensure that the data quality can be quickly assessed in the case of massive data, ensuring the timeliness of data quality assessment, and further, by displaying the assessment results, it is convenient for enterprises to pay attention to data quality and improve the enterprise's data quality management level. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0031] FIG1 is a diagram showing an application environment of a method for monitoring data quality in one embodiment;

[0032] FIG2 is a schematic flow chart of a method for monitoring data quality in one embodiment;

[0033] FIG3 is a flow chart of a method for monitoring data quality in another embodiment;

[0034] FIG4 is a flowchart of the steps of traversing the data to be evaluated in the temporary table and determining the data quality evaluation result of the business data in one embodiment;

[0035] FIG5 is a flow chart of a method for monitoring data quality in yet another embodiment;

[0036] FIG6 is a schematic diagram of a flow chart of step S206 in one embodiment;

[0037] FIG7 is a schematic flow chart of a method for monitoring data quality in yet another embodiment;

[0038] FIG8 is a schematic diagram of a service framework of a method for monitoring data quality in one embodiment;

[0039] FIG9 is a structural block diagram of a data quality monitoring device in one embodiment;

[0040] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The data quality monitoring method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. A Hadoop cluster can be run on the server 104. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0043] In an exemplary embodiment, as shown in FIG2 , a data quality monitoring method is provided, which is described by taking the method applied to the server 104 in FIG1 as an example, and includes the following steps S202 to S206 . Among them:

[0044] Step S202: extract the data to be evaluated from the business data of the distributed file system based on preset extraction rules, the preset extraction rules are associated with the dimension tags of the business data, and the distributed file system stores business data of at least two different business systems.

[0045] The distributed file system may include HDFS (Hadoop Distributed File System), which may be a part of the Hadoop cluster. The distributed file system is used to store business data of at least two different business systems, and the business data may include the original data produced by the business system. Furthermore, the business data may come from different business environments and different data environments. In this way, it is convenient to uniformly process business data of different dimensions.

[0046] Exemplarily, the dimension label of the business data can be determined according to the data source of the business data, or the dimension label of the business data can also be determined according to the data characteristics of the business data. The preset extraction rules can be used to extract the business data itself, or to extract some fields of the business data to obtain the data to be evaluated.

[0047] Step S204: store the data to be evaluated in a temporary table in a unified data format, and traverse the data to be evaluated in the temporary table to determine the data quality evaluation result of the business data.

[0048] Exemplarily, the data to be evaluated can be processed based on the template format, and the data to be evaluated can be converted into a specified data type and stored according to a specified storage structure. In this way, it is convenient to uniformly process the extracted data to be evaluated. The data to be evaluated obtained by the current session can be stored in a temporary table, which is used to delete when the current session ends. In this way, it is possible to cope with the situation where the business system produces massive business data, realize rapid evaluation of the data to be evaluated and avoid occupying resources for a long time. Furthermore, the data to be evaluated can be read from the temporary table for data quality assessment. In a possible implementation method, the method for performing data quality assessment on the data to be evaluated may include at least one of the following: determining accuracy by comparing with a trusted data source, using conditional statements to check whether there are missing fields, and using grouping operations to check whether the fields with dependencies are consistent.

[0049] Step S206, displaying the data quality assessment result, which is used to perform exception processing on the business data.

[0050] The data quality assessment results may include the abnormal business data to be processed and its abnormal type, as well as the statistical analysis results obtained through further analysis, processing suggestions, etc. In one possible implementation, data visualization tools may be used to generate charts, dashboards, etc. to present the data quality assessment results to relevant business personnel. In another possible implementation, a script or program may be constructed to regularly send the data quality assessment results to the terminals of relevant business personnel for display.

[0051] In the above data quality monitoring method, the business data of multiple business systems are integrated through a distributed file system, which can facilitate the unified processing of business data of different dimensions and ensure the comprehensiveness of data quality assessment; by storing the data to be evaluated in a temporary table in a unified format, it is convenient to uniformly process the extracted data to be evaluated, and ensure that the data quality can be quickly evaluated in the case of massive data, ensuring the timeliness of data quality assessment, and then, by displaying the evaluation results, it is convenient for enterprises to pay attention to data quality and improve the company's data quality management level.

[0052] In an exemplary embodiment, as shown in FIG3 , the data quality monitoring method may further include:

[0053] Step S2011, determining the dimension label of the business data based on the business scenario, flow stage and / or original data format of the business data corresponding to the business data.

[0054] Exemplarily, the functional modules and business processes of each business system can be collected and analyzed, different business scenarios can be determined, the key business data and the flow stages of business data involved in each business scenario can be identified, and the state changes of business data in different data flow stages can be recorded. Furthermore, the same first dimension label can be added to the business data under the same business scenario; the second dimension label can be added to the business data with dependencies, and the second dimension label can be used to distinguish the flow stages of business data. In one possible implementation, the first dimension label and the second dimension label can be used to determine whether the business data needs to be extracted and the storage location of the business data in the temporary table. Optionally, the third dimension label of the business data can be determined based on the original data format of the business data, and the third dimension label can be used to determine the extraction method of the business data.

[0055] In this embodiment, by pre-classifying the business data in different dimensions, it is possible to determine associated extraction rules based on the dimensional labels of the business data, thereby ensuring the consistency and efficiency of extracting the data to be evaluated from the business data.

[0056] In an exemplary embodiment, as shown in FIG4 , the step of traversing the data to be evaluated in the temporary table and determining the data quality evaluation result of the business data may include:

[0057] Step A1, based on multiple first automated scripts, traverse the data to be evaluated in the temporary table, and obtain multiple indicator data corresponding to the data to be evaluated respectively; the indicator data includes at least one of the following: data that has a dependency relationship with the data to be evaluated, and attribute information of the data to be evaluated.

[0058] Step A2: Determine the data quality assessment result of the business data based on the data to be assessed and the indicator data.

[0059] Exemplarily, the first automation script may be a Hive script, and multiple first automation scripts may be used to determine the data quality of business data from multiple perspectives (e.g., accuracy, timeliness, completeness, consistency, etc.). For example, data that has a dependency relationship with the data to be evaluated may include duplicate data, error-related data, etc., and the attribute information of the data to be evaluated may include data status, process processing time, etc. Error-related data may be used to determine the completeness of business data, data status may be used to determine the consistency of business data, and process processing time may be used to determine the timeliness of business data. Indicator data may also include invalid data.

[0060] Further, the Hive script may be executed by a Hive engine on the server 104 , and the determined data quality assessment results may be stored in a Hive table.

[0061] In this embodiment, by pre-building an automated script for determining the data quality assessment results of business data, it is possible to automatically execute the task of obtaining indicator data from different angles and achieve a comprehensive data quality assessment.

[0062] In an exemplary embodiment, as shown in FIG5 , the data quality monitoring method may further include:

[0063] Step S2012, respectively configure corresponding extraction rules for multiple dimension tags of business data, and construct a corresponding second automation script based on the extraction rules; the second automation script is used to respond to business system initialization or configuration changes, execute based on preset extraction rules, and extract the data to be evaluated from the business data stored in the database.

[0064] Exemplarily, the second automation script may be a Hive script, and the background management system may be used to configure the associated second automation script for different dimension tags. Furthermore, all second automation scripts may be assembled into a complete structured query language (SQL) for execution when the business system is initialized or the business line configuration is changed. In a possible implementation, the background management system may be implemented based on Vue+Springboot.

[0065] In this embodiment, by pre-building an automated script for extracting the data to be evaluated from the business data, it can be achieved that when the business system is initialized or the configuration is changed, the automated script can respond quickly and realize timely data extraction, thereby further improving the timeliness of data quality monitoring.

[0066] In an exemplary embodiment, as shown in FIG6 , the above step S206 may include:

[0067] Step S2061, synchronizing the data quality assessment results of the business data to the relational database.

[0068] Step S2062: Display the data quality assessment results in the relational database on a preset data display page.

[0069] For example, in response to determining the data quality assessment results, the heterogeneous data synchronization tool FlinkX can be used to synchronize the data quality assessment results of the business data from the Hive table to the relational database MySQL. Furthermore, the data visualization tool Superset can be used to access MySQL, and different types of charts can be configured on the front end for visualization, or relevant business personnel can directly view the data quality assessment results in MySQL through the front end.

[0070] In this embodiment, by synchronizing the data quality assessment results of business data to a relational database, it is possible to decouple data assessment and presentation of assessment results, thereby facilitating expansion of the presentation method of assessment results.

[0071] In an exemplary embodiment, as shown in FIG7 , the data quality monitoring method may further include:

[0072] Step S2071, at every preset period, executing a step of extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rule.

[0073] Step S2072, statistics the data quality assessment results of the business data, obtain the data volume and abnormal type of the data to be processed in the business data, and issue an alarm based on the data volume and abnormal type of the data to be processed.

[0074] Exemplarily, the task of extracting data to be evaluated can be triggered regularly, and the data to be evaluated can be extracted from the business data of the distributed file system based on the preset extraction rules; the data to be evaluated is stored in a temporary table in a unified data format, and the data to be evaluated in the temporary table is traversed to determine the data quality evaluation results of the business data. In a possible implementation, business data of different dimensions are stored in different tables, and the data quality evaluation results of the business data in each table can be counted to determine the total amount of abnormal data to be processed in each table, the abnormal type and distribution of the data to be processed, etc., and the alarm information including the statistical results is sent to the corresponding application group through the application robot according to the person in charge of each table.

[0075] In a possible implementation, a service framework of a data quality monitoring method is provided, as shown in FIG8 , and the server 104 may further process the business data to improve the monitoring effect of the data quality.

[0076] In this embodiment, by regularly evaluating the data quality of business data and establishing an alarm mechanism, it is possible to effectively monitor data quality issues and reduce the risks caused by data anomalies.

[0077] To sum up, in the above data quality monitoring method, the business data of multiple business systems are integrated through a distributed file system, which can facilitate the unified processing of business data of different dimensions and ensure the comprehensiveness of data quality assessment; by storing the data to be evaluated in a temporary table in a unified format, it is convenient to uniformly process the extracted data to be evaluated, and ensure that the data quality can be quickly evaluated in the case of massive data, ensuring the timeliness of data quality assessment, and then, by displaying the evaluation results, it is convenient for enterprises to pay attention to data quality and improve the enterprise's data quality management level.

[0078] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiment of the present application also provides a data quality monitoring device for implementing the data quality monitoring method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data quality monitoring device embodiments provided below can refer to the limitations of the data quality monitoring method above, and will not be repeated here.

[0080] In an exemplary embodiment, as shown in FIG9 , a data quality monitoring device 300 is provided, comprising: a data extraction module 301, a quality assessment module 302 and a data display module 303, wherein:

[0081] The data extraction module 301 is used to extract the data to be evaluated from the business data of the distributed file system based on preset extraction rules. The preset extraction rules are associated with the dimension tags of the business data. The distributed file system stores business data of at least two different business systems.

[0082] The quality assessment module 302 is used to store the data to be assessed in a temporary table in a unified data format, and traverse the data to be assessed in the temporary table to determine the data quality assessment result of the business data.

[0083] The data display module 303 is used to display the data quality assessment results, and the data quality assessment results are used to perform exception processing on business data.

[0084] In an exemplary embodiment, the data quality monitoring device 300 may further include:

[0085] The label determination module is used to determine the dimension labels of the business data based on the business scenario, flow stage and / or original data format of the business data corresponding to the business data.

[0086] In an exemplary embodiment, the quality assessment module 302 is further configured to:

[0087] Based on multiple first automated scripts, traverse the data to be evaluated in the temporary table to obtain multiple indicator data corresponding to the data to be evaluated respectively; the indicator data includes at least one of the following: data that has a dependency relationship with the data to be evaluated, and attribute information of the data to be evaluated;

[0088] Based on the data to be evaluated and the indicator data, determine the data quality assessment results of the business data.

[0089] In an exemplary embodiment, the data quality monitoring device 300 may further include:

[0090] The script construction module is used to configure corresponding extraction rules for multiple dimensional tags of business data, and to construct a corresponding second automation script based on the extraction rules; the second automation script is used to respond to business system initialization or configuration changes, and execute the steps of extracting the data to be evaluated from the business data stored in the database based on preset extraction rules.

[0091] In an exemplary embodiment, the data display module 303 is further used to:

[0092] Synchronize the data quality assessment results of business data to the relational database;

[0093] The data quality assessment results in the relational database are displayed on the preset data display page.

[0094] In an exemplary embodiment, the data quality monitoring device 300 may further include:

[0095] An alarm module, configured to execute, at each preset period, a step of extracting data to be evaluated from business data in the distributed file system based on a preset extraction rule;

[0096] The data quality assessment results of the business data are statistically analyzed to obtain the data volume and abnormal type of the data to be processed in the business data, and an alarm is issued based on the data volume and abnormal type of the data to be processed.

[0097] Each module in the above-mentioned data quality monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0098] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for monitoring data quality is implemented.

[0099] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0102] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0104] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0105] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0106] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for monitoring data quality, characterized in that: The method comprises: Extracting the data to be evaluated from the business data of the distributed file system based on a preset extraction rule, wherein the preset extraction rule is associated with a dimension tag of the business data, and the distributed file system stores business data of at least two different business systems; The data to be evaluated is stored in a temporary table in a unified data format, and the data to be evaluated in the temporary table is traversed to determine the data quality evaluation result of the business data; The data quality assessment result is displayed, and the data quality assessment result is used to perform exception processing on the business data.

2. The method according to claim 1, characterized in that: The method of extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rules includes: Determine the dimension label of the business data based on the business scenario, flow stage and / or original data format of the business data corresponding to the business data.

3. The method according to claim 1, characterized in that The traversing the data to be evaluated in the temporary table to determine the data quality evaluation result of the business data includes: Based on multiple first automated scripts, traverse the data to be evaluated in the temporary table to obtain multiple indicator data corresponding to the data to be evaluated respectively; the indicator data includes at least one of the following: data having a dependency relationship with the data to be evaluated, and attribute information of the data to be evaluated; Based on the data to be evaluated and the indicator data, a data quality evaluation result of the business data is determined.

4. The method according to claim 1, characterized in that: The method of extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rules includes: Corresponding extraction rules are configured for multiple dimensional tags of business data respectively, and a corresponding second automation script is constructed based on the extraction rules; the second automation script is used to execute the steps of extracting the data to be evaluated from the business data stored in the database based on the preset extraction rules in response to business system initialization or configuration changes.

5. The method according to claim 1, characterized in that The display of the data quality assessment result comprises: Synchronizing the data quality assessment results of the business data to a relational database; The data quality assessment results in the relational database are displayed on a preset data display page.

6. The method according to claim 1, characterized in that The method further comprises: At each preset period, executing the step of extracting the data to be evaluated from the business data of the distributed file system based on the preset extraction rule; The data quality assessment results of the business data are counted to obtain the data volume and abnormal type of the data to be processed in the business data, and an alarm is issued based on the data volume and abnormal type of the data to be processed.

7. A data quality monitoring device, characterized in that: The device comprises: A data extraction module, used to extract the data to be evaluated from the business data of the distributed file system based on a preset extraction rule, wherein the preset extraction rule is associated with a dimension tag of the business data, and the distributed file system stores business data of at least two different business systems; A quality assessment module, used to store the data to be assessed in a temporary table in a unified data format, and traverse the data to be assessed in the temporary table to determine the data quality assessment result of the business data; The data display module is used to display the data quality assessment results, and the data quality assessment results are used to perform exception processing on the business data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images