Data management method and device, computer equipment and storage medium

Abstract

The invention discloses a data management method and device, computer equipment and a storage medium, and relates to the technical field of big data processing. Through data link tracing analysis, a data source, a processing process and an inter-table access state are clearly displayed for user scene indexes, a user is helped to comprehensively understand a data forming process and timely discover a scene data quality problem, and the data consumption application quality is improved. Meanwhile, a financial data asset system is managed based on a business field line, a data source, a processing process and an application scene are traced, multiple business departments are covered, and data asset management is optimized. In addition, automatic data quality monitoring is achieved, the quality problems of the main data and the service data are displayed in a visual mode, problem nodes are rapidly positioned, and rectification is guided. In addition, the method supports real-time acquisition of a data asset application state, analysis of a user calling scene and use frequency, and construction of a data quality feedback and repair mechanism.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and specifically to a data governance method, apparatus, computer equipment and storage medium. Background Art

[0002] Data Middle Platform is a data architecture and management platform that aims to integrate, manage and share data resources inside and outside the enterprise to achieve centralized management and efficient use of data. Data Middle Platform usually includes core components such as data collection, data storage, data processing, data analysis and data services. Its technical architecture usually includes multiple layers such as data collection layer, data storage layer, data processing layer, data analysis layer and data service layer.

[0003] At present, many companies have used data middle-end technology to build a variety of data consumption applications and accumulated a large amount of data. As the content of construction scenarios deepens and the scope and frequency of construction scenario applications continue to increase, the reliability of data quality becomes increasingly important, and data quality issues become increasingly prominent, specifically manifested in the unstable quality of data results in the built data consumption application scenarios. When data quality problems occur, they cannot be discovered by the project team of the data consumption application construction manufacturer in the first place. They can only be discovered and reported by end users and can only be solved after layers of screening and analysis. The way to solve the problem is relatively passive and lagging. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data governance method, apparatus, computer equipment and storage medium to solve the relatively passive and lagging technical problems of the data quality detection solutions of the existing data middle platform.

[0005] In order to solve the above technical problems, the present application embodiment provides a data governance method, which adopts the following technical solutions:

[0006] A data governance approach, comprising:

[0007] Determine the data consumption scenario of the data to be processed, and obtain data consumption scenario indicator information based on the data consumption scenario;

[0008] Determine the data lineage relationship based on the data consumption scenario indicator information, and generate the full data link based on the data lineage relationship;

[0009] Perform data integration on the data to be processed according to the full data link to obtain a full data table;

[0010] Configure the data quality rule engine based on data consumption scenarios, data lineage, and full data links;

[0011] Use the data quality rule engine to perform data quality detection on all data tables to obtain data quality detection results.

[0012] Furthermore, the steps of determining the data consumption scenario of the data to be processed and obtaining the data consumption scenario indicator information according to the data consumption scenario specifically include:

[0013] Obtaining scenario description information related to the business scenario in the data to be processed, wherein the scenario description information includes scenario category and scenario level;

[0014] Identify the association between scenario description information and preset data indicators, and construct data consumption scenario indicator information based on the association.

[0015] Furthermore, the data to be processed is stored in a preset database, which includes several data warehouse layers, each of which is configured with a data model. After identifying the association between the scene description information and the preset data indicators, and constructing the data consumption scene indicator information according to the association, it also includes:

[0016] Construct data retrieval keywords and data screening conditions based on the association relationships and the characteristics and classification of data models in each data warehouse layer.

[0017] Furthermore, the steps of determining the data lineage relationship based on the data consumption scenario indicator information and generating the full data link according to the data lineage relationship specifically include:

[0018] Determine the data flow relationship between various data tables for the data to be processed based on the data consumption scenario indicator information;

[0019] Determine the data lineage relationship based on the data flow relationship, and draw the data link based on the data lineage relationship;

[0020] Integrate all data links to get the full data link.

[0021] Furthermore, before the step of integrating the data to be processed according to the full data link to obtain the full data table, the method further includes:

[0022] Identify the data architecture in each data table, where the data tables include indicator data tables, business data wide tables, master data association tables, and basic data tables;

[0023] Standardize the data schema across data tables.

[0024] Furthermore, after the step of standardizing the data architecture in each data table, the method further includes:

[0025] Build the data asset map, scenario indicator map, and data API map in sequence.

[0026] Furthermore, the steps of configuring the data quality rule engine according to the data consumption scenario, data lineage and full data link specifically include:

[0027] Selecting appropriate data quality monitoring rules based on data lineage and full data links, where data quality monitoring rules include quality monitoring rules and data maintenance rules;

[0028] Obtain the appropriate rule engine template based on the data consumption scenario;

[0029] Import the data quality monitoring rules into the rule engine template to obtain the data quality rule engine.

[0030] In order to solve the above technical problems, the embodiment of the present application also provides a data management device, which adopts the following technical solution:

[0031] A data management device, comprising:

[0032] A scenario matching module is used to determine the data consumption scenario of the data to be processed and obtain data consumption scenario indicator information according to the data consumption scenario;

[0033] The lineage acquisition module is used to determine the data lineage relationship based on the data consumption scenario indicator information and generate a full data link based on the data lineage relationship;

[0034] A data integration module is used to integrate the data to be processed according to the full data link to obtain a full data table;

[0035] The engine building module is used to configure the data quality rule engine based on data consumption scenarios, data lineage relationships, and the entire data link;

[0036] The quality detection module is used to use the data quality rule engine to perform data quality detection on the entire data table to obtain data quality detection results.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0038] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data governance method as described in any one of the above items when executing the computer-readable instructions.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0040] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the data governance method as described in any one of the above.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0042] 1) This application can perform data link traceability analysis for user scenario indicators, track the formation process of scenario indicators, display the data source of scenario indicators, the processing and calculation process, and the data acquisition status of which result tables, dimension tables and related tables are involved, so that end users can promptly discover data quality problems in corresponding scenarios when conducting data analysis in daily use of specific scenarios, allowing users to see the data link clearly, understand the data formation process, and improve the quality of data consumption applications.

[0043] 2) This application can manage the existing financial data asset system according to business areas and lines, and trace the formation process of financial data assets, including data sources, data processes, data processing results, data application scenarios, and the business departments and offices involved.

[0044] 3) This application can automatically monitor the data quality of the data asset system processing chain, including the timeliness, completeness, uniqueness and other data quality issues of master data and business data. It can intuitively display the data quality monitoring statistical indicators on a visual page, support the rapid location of problem nodes in the data link and rectify data problems.

[0045] 4) This application can obtain the latest application status of data assets in real time, understand the user call scenarios of financial data assets, understand the frequency of data use, establish a user scenario data problem feedback and reporting repair mechanism, and evaluate the quality of monitoring data, analyze the potential for financial data sharing, and provide guidance for future data consumption application construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0048] Figure 2 A flow chart showing an embodiment of a data governance method according to the present application is shown;

[0049] Figure 3 A schematic diagram showing the structure of an embodiment of a data management device according to the present application is shown;

[0050] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0055] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0056] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0057] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0058] It should be noted that the data management method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data management device is generally set in the server / terminal device.

[0059] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0060] In view of the relatively passive and lagging technical problems of the data quality detection scheme of the existing data center, this application provides the following solutions:

[0061] 1. Connect the data link of the middle platform, use the data blood relationship information of the data middle platform to build the relationship between the source table, model table, indicator table and data consumption scenario in the financial domain, create the data traceability link of "consumption scenario-scenario indicator data-middle platform wide table data-source business table data", establish a data asset system, integrate scattered data, and lay a solid foundation for the aggregation, retrieval, analysis and display of consumption scenario data asset results and the tracing of abnormal data;

[0062] 2. Implement data quality rule configuration and calculation engine, and realize automatic monitoring and visual display of data quality in each link through automation technology. Sort out the data quality monitoring rules in each link, form a full-link monitoring rule list of "consumption scenario-scenario indicator-middle platform wide table-source business table", build the monitoring rules into the data middle platform automation program, automatically issue a data quality anomaly list, analyze data problems, and quickly discover and locate abnormal problems;

[0063] 3. Establish a closed-loop mechanism for eliminating defects, supervising and rectifying data quality issues. The system will automatically generate supervision orders for discovered data problems and push them to relevant responsible persons for rectification to ensure that problems are handled in a timely and effective manner. The system will also directly connect to the business system through single sign-on to quickly locate problematic data to ensure that problems can be quickly located and rectified.

[0064] 4. Based on the data consumption scenarios, continuously sort out the scenario data links during the data integration and service stages, and abstract, refine and share the data architecture models for indicator data, business wide table data, master data associations, and basic data to achieve unified data architecture standards, avoid problems such as repeated traceability of data indicators and multiple sources, and further improve the effective utilization rate of data assets; make full use of the data middle platform capabilities, build data asset maps, scenario indicator maps, data API map analysis and other data quality analysis functions to improve data quality, and provide businesses with reliable, available, and easy-to-use data for normalized data quality control.

[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for data processing according to the present application. The data governance method comprises the following steps:

[0066] S201, determining a data consumption scenario of the data to be processed, and obtaining data consumption scenario indicator information according to the data consumption scenario.

[0067] In this step, we first need to clarify the specific business scenarios or analysis targets involved in the data to be processed, that is, the data consumption scenarios. Through in-depth analysis of business needs, we extract scenario description information, including scenario categories, scenario levels, and target output results of the scenarios. At the same time, based on scenario requirements, we further obtain data indicator information associated with the scenarios, such as the definition, source, calculation logic, and specific application methods of the indicators in the scenarios. These indicator information can be extracted from the data asset system already built in the data center, or automatically generated through the association with the scenario description information.

[0068] S202, determine the data lineage relationship based on the data consumption scenario indicator information, and generate a full data link based on the data lineage relationship.

[0069] By analyzing the data consumption scenario indicator information, we gradually sort out the data flow paths of the data to be processed at each level of the middle platform, and clarify the processing logic and flow of data between different nodes. Based on these flow relationships, we generate data lineage relationships covering all levels, and describe in detail all the associated tables and operation steps for each indicator from the source data to the final consumer application. At the same time, we generate a visual full data link diagram based on the lineage relationship to intuitively present the data source, processing process and usage scenario.

[0070] S203, integrating the data to be processed according to the full data link to obtain a full data table.

[0071] Based on the full data link, data from different sources and structures are integrated to form a complete full data table. During the integration process, the architecture of the data table needs to be sorted out and standardized, including the unification of field names, the normalization of data types, and the clarification of the relationship between primary keys and foreign keys to ensure the compatibility of multi-source data. Then, according to the guidance of the data link diagram, the relationship between each data table is logically integrated and physically merged to eliminate duplicate data and redundant fields, and optimize data storage and query efficiency.

[0072] S204, configure the data quality rule engine according to the data consumption scenario, data lineage relationship and the entire data link.

[0073] In this step, the data quality rule engine is designed and configured by analyzing the specific requirements of the data consumption scenario for data quality, as well as the data lineage and structural characteristics of the entire data link. Specifically, it is necessary to define data quality rules applicable to different scenarios, including integrity rules (such as non-null fields, complete primary and foreign key associations), consistency rules (such as data value range verification, field relationship constraints), and timeliness rules (such as data update frequency monitoring). In addition, the rule engine also needs to support quality monitoring strategies for different data levels, such as raw data verification at the source table layer, aggregated data verification at the wide table layer, and calculation logic verification at the indicator layer.

[0074] S205: Use the data quality rule engine to perform data quality detection on the entire data table to obtain data quality detection results.

[0075] In this step, the configured data quality rule engine is used to conduct a comprehensive test on all data tables to evaluate the accuracy, completeness, and consistency of the data. The detection process includes the automatic execution of rules and the automatic capture of abnormal data, such as identifying empty value fields, duplicate records, values ​​that do not meet the expected range, and breakpoints in the data lineage chain. At the same time, the test results are displayed in a visual form, including data problem distribution, abnormal rate statistics, and trend analysis. In addition, for abnormal data problems found during the detection, a detailed report needs to be generated to record the specific manifestations of the problem, the node where it occurred, and the scope of impact, and provide rectification suggestions.

[0076] Furthermore, the step of determining the data consumption scenario of the data to be processed and obtaining the data consumption scenario indicator information according to the data consumption scenario specifically includes:

[0077] Obtaining scenario description information related to the business scenario in the data to be processed, wherein the scenario description information includes scenario category and scenario level;

[0078] Identify the association between scenario description information and preset data indicators, and construct data consumption scenario indicator information based on the association.

[0079] In the embodiment of the present application, the data consumption scenario of the data to be processed is determined, and the scenario description information related to the business scenario in the data to be processed is obtained, wherein the scenario description information includes the scenario category and the scenario level. The correlation analysis is performed to analyze the correlation between the data consumption scenario and different indicators, and the correlation analysis is performed layer by layer based on the scenario category and the scenario level, and finally to the corresponding scenario indicator situation, and the indicator information penetration analysis is supported to visualize the analysis results.

[0080] By classifying and scenario-based analyzing the data to be processed, the application scope of the data in different business consumption scenarios can be accurately locked. First, by parsing the data to be processed, the scenario description information related to the business is extracted, including scenario categories (such as finance, sales, operations, etc.) and scenario levels (such as core scenarios, general scenarios). Based on the scenario description information, the built-in blood relationship and indicator system library of the data center is used to automatically identify the relationship between the data and the preset key indicators, and build a complete data consumption scenario indicator system. At the same time, the process supports dynamic adjustment and expansion. When the business scenario or data indicator is updated, the association relationship can be adjusted in real time to ensure the accuracy and timeliness of the data link. In addition, the method further associates the data table model with the business logic to form an intuitive data link view for users, which is convenient for users to quickly locate the source of the scene data, the processing logic and the final output results during the data analysis process.

[0081] Furthermore, the data to be processed is stored in a preset database, which includes several data warehouse layers, each of which is configured with a data model. After identifying the association between the scene description information and the preset data indicators, and constructing the data consumption scene indicator information according to the association, it also includes:

[0082] Construct data retrieval keywords and data screening conditions based on the association relationships and the characteristics and classification of data models in each data warehouse layer.

[0083] In an embodiment of the present application, corresponding search keywords and screening conditions are established based on the indicators associated with the data consumption scenario and the characteristics and classification of data models at different levels of the data warehouse layer, supporting keyword search, search result display and access to the entire link of the scenario and its data traceability, displaying the basic information, blood relationship, governance and change record information of the corresponding indicators and data table models, so as to guide users to quickly locate relevant data table model information when analyzing related issues.

[0084] Furthermore, the steps of determining the data lineage relationship based on the data consumption scenario indicator information and generating the full data link according to the data lineage relationship specifically include:

[0085] Determine the data flow relationship between various data tables for the data to be processed based on the data consumption scenario indicator information;

[0086] Determine the data lineage relationship based on the data flow relationship, and draw the data link based on the data lineage relationship;

[0087] Integrate all data links to get the full data link.

[0088] In this embodiment, the data consumption scenario indicator information is further used to comprehensively sort out the data flow paths between various data tables, thereby constructing a complete data link diagram. Specifically, firstly, based on the data consumption scenario indicator information, the flow relationship between the various levels (such as the source layer, the sharing layer, and the analysis layer) of the data to be processed in the data center is analyzed to clarify how the data reaches the final consumption scene from the source data table through the intermediate processing table. Subsequently, an accurate data lineage relationship is generated based on the data flow relationship, and the source, flow direction, and processing process of each link of the data are clearly presented. Through further abstraction and integration of data lineage relationships, a full data link view across businesses and data tables is formed.

[0089] Through the above steps, this solution can help users quickly understand the entire life cycle of data, intuitively present the data source, processing and application path, and ensure the accuracy, completeness and traceability of the data.

[0090] Furthermore, before the step of integrating the data to be processed according to the full data link to obtain the full data table, the method further includes:

[0091] Identify the data architecture in each data table, where the data tables include indicator data tables, business data wide tables, master data association tables, and basic data tables;

[0092] Standardize the data schema across data tables.

[0093] In this embodiment, in order to ensure the accuracy and consistency of the integration of the entire data link, the identification and standardization processing of the data table data architecture is further introduced. First, by analyzing the structure of each data table, the data architecture in the indicator data table, business data wide table, master data association table and basic data table is identified, and the core fields, associations and supporting functions of each table for data consumption scenarios are clarified. Then, based on the standardization requirements of the data architecture, data tables from different sources or formats are uniformly processed, including field naming standardization, data type unification, and clarification of primary key and foreign key relationships, etc., to eliminate data redundancy and conflicts. At the same time, the standardized data architecture is compared with the existing data middle platform data asset model to ensure that it is consistent with the construction requirements of the entire data link. Through this standardized processing link, the compatibility and integration efficiency between data tables can be effectively improved.

[0094] Through the above steps, this solution can improve the matching degree and fusion efficiency of different types of data tables in data integration, provide technical support for building a high-quality and highly consistent data asset system, and further improve the accuracy and reliability of data governance and analysis.

[0095] Furthermore, after the step of standardizing the data architecture in each data table, the method further includes:

[0096] Build the data asset map, scenario indicator map, and data API map in sequence.

[0097] In the embodiment of the present application, based on the data consumption scenario, the scenario data link is continuously sorted out during the data integration and service stage, and the data architecture model is abstracted, refined and shared for the indicator data table, business data wide table, master data association table and basic data table, so as to achieve the standardization of data architecture, avoid the problems of repeated tracing of data indicators and multiple sources, and further improve the effective utilization rate of data assets. And make full use of the data middle platform capabilities, build data asset map / scenario indicator map / data API map analysis and other data quality analysis functions, improve data quality, and provide reliable, available and easy-to-use data for business to carry out normalized data quality control.

[0098] A data asset map is built based on the data middle platform, providing comprehensive, multi-level, and multi-dimensional data retrieval capabilities. Data assets can be layered and penetrated step by step according to their life cycle, and multi-level resource association queries can be performed on data resources shared and aggregated by various manufacturers and business departments.

[0099] Build a scenario indicator map maintenance function. On the maintenance interface, manually build a tree structure of the scenario map through a mind map-like method and associate it with the scenario indicators. At the same time, it has a quick retrieval function for scenario indicators to solve the problem of lack of intuitive link tracking capabilities for the source of scenario indicator data. The problem of abnormal scenario indicator data can be solved by looking at the data source through the code, and the long path and time-consuming problem can be solved. This makes it convenient to quickly locate when maintaining scenario indicators or checking the traceability of existing indicators.

[0100] Building a data API map provides access to data service management, ensuring the orderly and stable operation of data services, and will provide easy-to-use and intuitive service monitoring function support to improve the efficiency of data resource sharing and application. Unified data service monitoring can achieve unified scheduling and monitoring of data sharing, conduct comprehensive monitoring of data service usage, monitor the status of each API service application, data sharing performance, etc., to ensure the efficiency and reliability of data sharing, and achieve data security, integrity, availability, verifiability, and reliable and stable operation for data sharing services.

[0101] Furthermore, the solution of this application also includes establishing a closed-loop defect elimination, supervision and rectification mechanism for data quality issues, with the goal of efficiently locating problems and quickly carrying out rectification, realizing the traceability of supervision orders, and automatically generating supervision orders for discovered data problems through the system, pushing them to relevant responsible persons for rectification, ensuring that problems are handled in a timely and effective manner, and quickly locating problematic data through single-point sign-on direct connection to the business system, ensuring that problems can be quickly located and rectified.

[0102] First, the system automatically identifies abnormal data problems and automatically issues supervision orders to the corresponding responsible persons for the data in each data link. It also supports manual generation of rectification supervision orders. Supervision orders are managed uniformly, including "My To-Do" and "My Done", and the rectification status of supervision orders under the current user's name is displayed, including the number of problems, rectification progress, and planned rectification completion time.

[0103] Secondly, the responsible party implements rectification measures according to the rectification time limit requirements, determines the source of data rectification based on rule-related indicators, and the tool uses single sign-on to directly connect to the business source system, quickly locates problem data for analysis and rectification, and monitors the rectification progress in real time to ensure the smooth progress of the rectification work and timely and effective handling and feedback of problems.

[0104] If rectification is difficult or there are special problems, the responsible person can apply for exemption from rectification for data that meets the characteristics of the audit rules. After approval, this problem does not need to be rectified. It supports automatic SMS reminders to the person in charge of rectification, and also supports manual reminders to ensure the quality and effectiveness of rectification work.

[0105] This application provides a data traceability method that clearly displays the data link from the perspective of the end user to help the end user see the data link clearly, quickly locate the problem link in the data link, understand the data formation process and the person responsible for problem rectification, and promptly discover data quality problems in the corresponding scenario, so that users can see the data link clearly, understand the data formation process, and ensure the traceability of the source of the data and the transparency of the data flow. Through the traceable automated monitoring and visual statistical analysis display of the data processing link, the data quality problem nodes such as the timeliness, integrity, and uniqueness of the data can be quickly located and data problem rectification monitoring and analysis can be performed, effectively improving the timeliness and work efficiency of data problem solving, and ensuring the smooth operation of data consumption applications.

[0106] Furthermore, the steps of configuring the data quality rule engine according to the data consumption scenario, data lineage and full data link specifically include:

[0107] Selecting appropriate data quality monitoring rules based on data lineage and full data links, where data quality monitoring rules include quality monitoring rules and data maintenance rules;

[0108] Obtain the appropriate rule engine template based on the data consumption scenario;

[0109] Import the data quality monitoring rules into the rule engine template to obtain the data quality rule engine.

[0110] In the embodiment of the present application, data quality rule configuration and calculation engine are implemented, and automatic monitoring and visual display of data quality in each link are realized through automation technology. According to the data lineage relationship and the whole data link, the data quality monitoring rules of each link are sorted out to form a full-link monitoring rule list of "consumption scenario-scenario indicator-middle platform wide table-source business table", and the monitoring rules are built into the data middle platform automation program, and a data quality anomaly list is automatically issued to analyze data problems and quickly discover and locate abnormal problems;

[0111] According to the actual consumption scenario needs, that is, according to the data lineage and the entire data link, the monitoring rule configuration management is completed, including quality monitoring rules and data maintenance rules, and the association of rule indicators and monitoring data table models is supported. And the adapted rule engine template is obtained according to the data consumption scenario.

[0112] Compose monitoring rules for monitoring strategies in different data consumption scenarios. Support functions such as alarm strategy maintenance, classification, policy task scheduling monitoring, policy query, rule range configuration, level setting, and recipient setting to ensure that alarm information can be communicated in a timely and effective manner, helping relevant personnel better understand and identify problems.

[0113] Monitoring is automatically performed according to data quality monitoring rules, and the details of the automatic monitoring rule results including occurrence time, processing status, responsible person and other related information are displayed in detail. Manual whitelist exemption operations for abnormal monitoring results are supported to meet the needs of users to check and confirm quality monitoring results.

[0114] Use visualization to present the data quality of key indicators in data consumption scenarios. Centrally display the distribution of quality monitoring issues in related scenarios, systematically grasp the key data situation, display and analyze the execution results of data traceability quality monitoring rules from different angles and trend analysis, and guide users to locate and analyze data quality issues.

[0115] In the above embodiments, the present application discloses a data governance method, which relates to the field of big data processing technology. Through the data link tracing analysis for user scenario indicators, the formation process of scenario indicators can be clearly presented, including the data source, processing and calculation process, and the status of the table data involved, so that the end user can quickly find data quality problems when using specific scenarios to carry out data analysis, and have a deep understanding of the data link and its formation process, thereby improving the quality and reliability of data consumption applications. At the same time, the present application can manage the existing financial data asset system according to the business field line, comprehensively trace the formation process of data assets, covering the data source, processing process, result application scenario and related business department information, and provide full-process transparent support for financial management. In addition, through the automated data quality monitoring mechanism, the present application can identify the timeliness, integrity and uniqueness problems in master data and business data in real time, and intuitively display the quality monitoring results in a visual form, quickly locate the problem node and repair it. Finally, the present application can also obtain the latest application status of data assets in real time, analyze user call scenarios and data usage frequency, establish a data quality feedback mechanism, and evaluate the data sharing potential and consumption application status.

[0116] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0117] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0118] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0120] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0121] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data management device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 3 As shown, the data management device 300 described in this embodiment includes:

[0123] A scene matching module 301 is used to determine the data consumption scene of the data to be processed, and obtain data consumption scene indicator information according to the data consumption scene;

[0124] A lineage acquisition module 302 is used to determine the data lineage relationship based on the data consumption scenario indicator information, and generate a full data link according to the data lineage relationship;

[0125] A data integration module 303 is used to integrate the data to be processed according to the full data link to obtain a full data table;

[0126] The engine construction module 304 is used to configure the data quality rule engine according to the data consumption scenario, data lineage relationship and the full data link;

[0127] The quality detection module 305 is used to use the data quality rule engine to perform data quality detection on the entire data table to obtain a data quality detection result.

[0128] Furthermore, the scene matching module 301 specifically includes:

[0129] A scenario description unit, used to obtain scenario description information related to the business scenario in the data to be processed, wherein the scenario description information includes a scenario category and a scenario level;

[0130] The indicator information unit is used to identify the association between the scenario description information and the preset data indicators, and to construct data consumption scenario indicator information based on the association.

[0131] Furthermore, the data to be processed is stored in a preset database, which includes several data warehouse layers, each of which is configured with a data model. The data governance device 300 also includes:

[0132] The data screening module is used to construct data retrieval keywords and data screening conditions based on the association relationships and the characteristics and classifications of the data models in each data warehouse layer.

[0133] Furthermore, the blood relationship acquisition module 302 specifically includes:

[0134] A flow relationship identification unit, used to determine the data flow relationship between various data tables of the data to be processed according to the data consumption scenario indicator information;

[0135] A blood relationship determination unit, used to determine the data blood relationship according to the data flow relationship, and draw a data link according to the data blood relationship;

[0136] The data link integration unit is used to integrate all data links to obtain a full data link.

[0137] Furthermore, the data management device 300 also includes:

[0138] A data architecture identification module is used to identify the data architecture in each data table, wherein the data table includes an indicator data table, a business data wide table, a master data association table, and a basic data table;

[0139] The data architecture standardization module is used to standardize the data architecture in each data table.

[0140] Furthermore, the data management device 300 also includes:

[0141] The map construction module is used to sequentially construct the data asset map, scenario indicator map, and data API map.

[0142] Furthermore, the engine construction module 304 specifically includes:

[0143] A rule adaptation unit, used for selecting adapted data quality monitoring rules according to data lineage and full data link, wherein the data quality monitoring rules include quality monitoring rules and data maintenance rules;

[0144] The engine template adaptation unit is used to obtain the adapted rule engine template according to the data consumption scenario;

[0145] The engine building unit is used to import the data quality monitoring rules into the rule engine template to obtain the data quality rule engine.

[0146] In the above embodiments, the present application discloses a data governance device, which relates to the field of big data processing technology. Through the data link tracing analysis for user scenario indicators, the formation process of scenario indicators can be clearly presented, including the data source, processing and calculation process, and the status of the table data involved, so that the end user can quickly find data quality problems when using specific scenarios to carry out data analysis, and have a deep understanding of the data link and its formation process, thereby improving the quality and reliability of data consumption applications. At the same time, the present application can manage the existing financial data asset system according to the business field line, comprehensively trace the formation process of data assets, covering the data source, processing process, result application scenario and related business department information, and provide full-process transparent support for financial management. In addition, through the automated data quality monitoring mechanism, the present application can identify the timeliness, integrity and uniqueness problems in master data and business data in real time, and intuitively display the quality monitoring results in a visual form, quickly locate the problem node and repair it. Finally, the present application can also obtain the latest application status of data assets in real time, analyze user call scenarios and data usage frequency, establish a data quality feedback mechanism, and evaluate the data sharing potential and consumption application status.

[0147] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4This is a basic structural block diagram of the computer device in this embodiment.

[0148] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0149] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0150] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the data governance method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0151] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the data governance method.

[0152] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0153] In the above embodiment, the present application discloses a computer device, which relates to the field of big data processing technology. Through the data link tracing analysis for user scenario indicators, the formation process of scenario indicators can be clearly presented, including data sources, processing and calculation processes, and the status of table data involved, so that end users can quickly discover data quality problems when using specific scenarios to carry out data analysis, and have a deep understanding of data links and their formation process, thereby improving the quality and reliability of data consumption applications. At the same time, the present application can manage the existing financial data asset system according to the business field line, comprehensively trace the formation process of data assets, covering data sources, processing processes, result application scenarios and related business department information, and provide full-process transparent support for financial management. In addition, through the automated data quality monitoring mechanism, the present application can identify the timeliness, integrity and uniqueness problems in master data and business data in real time, and intuitively display the quality monitoring results in a visual form, quickly locate the problem nodes and repair them. Finally, the present application can also obtain the latest application status of data assets in real time, analyze user call scenarios and data usage frequency, establish a data quality feedback mechanism, and evaluate data sharing potential and consumer application status.

[0154] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data governance method as described above.

[0155] In the above embodiments, the present application discloses a computer-readable storage medium, which relates to the field of big data processing technology. Through the data link tracing analysis for user scenario indicators, the formation process of scenario indicators can be clearly presented, including data sources, processing and calculation processes, and the status of table data involved, so that end users can quickly discover data quality problems when using specific scenarios to carry out data analysis, and have a deep understanding of data links and their formation process, thereby improving the quality and reliability of data consumption applications. At the same time, the present application can manage the existing financial data asset system according to the business field line, comprehensively trace the formation process of data assets, covering data sources, processing processes, result application scenarios and related business department information, and provide full-process transparent support for financial management. In addition, through the automated data quality monitoring mechanism, the present application can identify the timeliness, integrity and uniqueness problems in master data and business data in real time, and intuitively display the quality monitoring results in a visual form, quickly locate the problem nodes and repair them. Finally, the present application can also obtain the latest application status of data assets in real time, analyze user call scenarios and data usage frequency, establish a data quality feedback mechanism, and evaluate the data sharing potential and consumer application status.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0157] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0158] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data governance method, characterized in that: include: Determine a data consumption scenario of the data to be processed, and obtain data consumption scenario indicator information according to the data consumption scenario; Determine the data lineage relationship based on the data consumption scenario indicator information, and generate a full data link according to the data lineage relationship; Performing data integration on the data to be processed according to the full data link to obtain a full data table; Configure a data quality rule engine according to the data consumption scenario, the data lineage relationship and the full data link; The data quality rule engine is used to perform data quality detection on the full data table to obtain a data quality detection result.

2. The data governance method according to claim 1, characterized in that: The step of determining the data consumption scenario of the data to be processed and obtaining the data consumption scenario indicator information according to the data consumption scenario specifically includes: Acquire scenario description information related to the business scenario in the data to be processed, wherein the scenario description information includes a scenario category and a scenario level; Identify the association relationship between the scenario description information and the preset data indicators, and construct the data consumption scenario indicator information based on the association relationship.

3. The data governance method according to claim 2, characterized in that: The data to be processed is stored in a preset database, the database includes a plurality of data warehouse layers, each of the data warehouse layers is configured with a data model, and after the steps of identifying the association between the scenario description information and the preset data indicators, and constructing the data consumption scenario indicator information according to the association, it also includes: According to the association relationship and the characteristics and classification of the data models in each data warehouse layer, data retrieval keywords and data screening conditions are constructed.

4. The data governance method according to claim 1, characterized in that: The step of determining the data lineage relationship based on the data consumption scenario indicator information and generating a full data link according to the data lineage relationship specifically includes: Determine the data flow relationship between the data tables for the data to be processed according to the data consumption scenario indicator information; Determine the data lineage relationship according to the data flow relationship, and draw a data link according to the data lineage relationship; All data links are integrated to obtain the full data link.

5. The data governance method according to claim 4, characterized in that: Before the step of integrating the data to be processed according to the full data link to obtain a full data table, the step further includes: Identify the data architecture in each of the data tables, wherein the data tables include an indicator data table, a business data wide table, a master data association table, and a basic data table; The data structures in each of the data tables are standardized.

6. The data governance method according to claim 5, characterized in that: After the step of standardizing the data architecture in each of the data tables, the method further includes: Build the data asset map, scenario indicator map, and data API map in sequence.

7. The data governance method according to claim 1, characterized in that: The step of configuring the data quality rule engine according to the data consumption scenario, the data lineage relationship and the full data link specifically includes: Selecting adapted data quality monitoring rules according to the data lineage relationship and the full data link, wherein the data quality monitoring rules include quality monitoring rules and data maintenance rules; Obtaining an adapted rule engine template according to the data consumption scenario; The data quality monitoring rules are imported into the rule engine template to obtain a data quality rule engine.

8. A data management device, characterized in that: include: A scenario matching module is used to determine the data consumption scenario of the data to be processed, and obtain data consumption scenario indicator information according to the data consumption scenario; A lineage acquisition module, used to determine the data lineage relationship based on the data consumption scenario indicator information, and generate a full data link according to the data lineage relationship; A data integration module, used for integrating the data to be processed according to the full data link to obtain a full data table; An engine construction module, used to configure a data quality rule engine according to the data consumption scenario, the data lineage relationship and the full data link; The quality detection module is used to use the data quality rule engine to perform data quality detection on the full data table to obtain a data quality detection result.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data governance method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data governance method as described in any one of claims 1 to 7.

Images