Data treatment method, terminal equipment and storage medium

Abstract

The invention is suitable for the field of data processing, and discloses a data treatment method, terminal equipment and a storage medium. The method is applied to a data facility, the data facility comprises a data platform, a data system and a data tool, and the method comprises the steps of accessing or collecting to-be-governed data; executing a preset operation corresponding to the full life cycle stage on the to-be-treated data, wherein the full life cycle stage comprises a data production stage, a data processing stage, a data circulation stage, a data application stage, a data maintenance stage, a data display stage and a data destruction stage; multi-dimensional data assessment corresponding to the full life cycle stage is executed on the to-be-governed data, assessment results are obtained, and the assessment results comprise a data value assessment report, a data quality assessment report, a data standardization assessment report and a data safety assessment report; and constructing a matrix type responsibility system according to an evaluation result, wherein the responsibility system maps a data engineer, a data analyst, a data manager, a data security teacher and a data quality evaluator to a corresponding life cycle stage in the full life cycle stage. According to the method, the problems of full-flow fault, evaluation subjectivity, role fuzzification and facility islanding in datamation treatment are solved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a data governance method, a terminal device, and a storage medium. Background Art

[0002] With the rapid development of the digital economy, people's needs for aspects such as the value of data assets, data quality, data signing rights, data space, and data transactions are becoming increasingly urgent.

[0003] Although traditional relational databases and non-relational databases provide reliable data storage solutions, they do not have the ability to provide a complete data full life cycle and management, and there are problems such as full process breaks, subjective evaluation, blurred roles, and facility islets. A new technical means is needed to solve the above technical problems. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data governance method, a terminal device, and a storage medium, which can solve the problems of full process breaks, subjective evaluation, blurred roles, and facility islets in related technologies.

[0005] A first aspect of the present invention provides a data governance method, which is applied to a data facility. The data facility includes a data platform, a data system, and a data tool. The method includes:

[0006] Accessing or collecting data to be governed;

[0007] Performing a preset operation corresponding to a full life cycle stage on the data to be governed. The full life cycle stage includes a data production stage, a data processing stage, a data circulation stage, a data application stage, a data maintenance stage, a data display stage, and a data destruction stage;

[0008] Performing a multi-dimensional data evaluation corresponding to the full life cycle stage on the data to be governed to obtain an evaluation result. The evaluation result includes a data value evaluation report, a data quality evaluation report, a data normalization evaluation report, and a data security evaluation report;

[0009] Constructing a matrix-style responsibility system according to the evaluation result. The responsibility system maps data engineers, data analysts, data managers, data security engineers, and data quality evaluators to corresponding life cycle stages in the full life cycle stage.

[0010] Optionally, in a first implementation manner of the first aspect of the present invention, the step of performing a preset operation corresponding to a full life cycle stage on the data to be governed includes:

[0011] Perform metadata annotation operations corresponding to data production on the data to be governed to obtain original data, where the data annotation operations include constructing a data knowledge graph corresponding to the original data;

[0012] Perform data cleaning operations on the data to be governed to obtain cleaned data, where the data annotation includes constructing a data knowledge graph corresponding to the cleaned data;

[0013] Perform structured processing operations corresponding to the data processing stage on the cleaned data to obtain structured intermediate data;

[0014] Perform circulation operations corresponding to the data circulation stage on the structured intermediate data to obtain circulated data;

[0015] Perform business logic operations corresponding to the data application stage on the circulated data to obtain business application results;

[0016] Perform version control management operations corresponding to the data maintenance stage based on the business application results to obtain versioned data;

[0017] Perform visual modeling analysis operations corresponding to the data display stage based on the versioned data to obtain an interactive visualization report;

[0018] If the interactive visualization report expires, destroy the interactive visualization report to complete the destruction operations corresponding to the data destruction stage.

[0019] Optionally, in the second implementation manner of the first aspect of the present invention, the step of performing structured processing operations corresponding to the data processing stage on the cleaned data to obtain structured intermediate data includes:

[0020] Perform master data operations, generate data directory operations, generate data dictionary operations, and generate data model operations on the cleaned data in sequence to obtain the structured intermediate data.

[0021] Optionally, in the third implementation manner of the first aspect of the present invention, the step of performing circulation operations corresponding to the data circulation stage on the structured intermediate data to obtain circulated data includes:

[0022] Perform column-level permission control on the structured intermediate data according to the role and scenario mapping table in the dynamic permission policy to obtain controlled data;

[0023] Perform encrypted transmission on the controlled data to obtain the circulated data.

[0024] Optionally, in the fourth implementation manner of the first aspect of the present invention, the step of performing a visual modeling analysis operation corresponding to the data display phase according to the versioned data to obtain an interactive visual report includes:

[0025] Performing a modeling analysis on the versioned data according to the analysis dimensions in the business requirements to obtain a visual data set;

[0026] Generating an interactive governance dashboard including a drill-down analysis component according to the node relationships in the data knowledge graph to obtain an interactive visual report.

[0027] Optionally, in the fifth implementation manner of the first aspect of the present invention, the step of accessing or collecting the data to be governed includes:

[0028] Performing data extraction to obtain the data to be governed.

[0029] Optionally, in the sixth implementation manner of the first aspect of the present invention, the step of constructing a matrix-based responsibility system according to the evaluation results includes:

[0030] Generating responsibility cells including life cycle stages and quality defect types according to the null value rate and duplication rate indicators in the data quality assessment report;

[0031] Assigning roles to the responsibility cells to construct the matrix-based responsibility system.

[0032] Optionally, in the seventh implementation manner of the first aspect of the present invention, the step of assigning roles to the responsibility cells to construct the matrix-based responsibility system includes:

[0033] Assigning roles to the responsibility cells according to the position ability matrix in the preset RBAC permission model to construct the matrix-based responsibility system.

[0034] In a second aspect, an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above data governance method are implemented.

[0035] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above data governance method are implemented.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer program product, which when running on a terminal device causes the terminal device to execute the above data governance method.

[0037] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: by defining data production, processing, circulation, application, maintenance, display, and destruction, data silos and process breakpoints caused by stage separation in traditional databases are eliminated. Multidimensional data evaluation reports are generated, and quantitative indicators are used instead of manual experience judgments to improve the objectivity and reproducibility of the evaluation. A matrix responsibility system is constructed to bind roles such as data engineers and security officers to life cycle stages, and clarify the boundaries of rights and responsibilities. Multi-source heterogeneous data are integrated based on unified data facilities (platforms, systems, tools), breaking the closed architecture of traditional databases and supporting cross-system data governance operations. The problems of full-process faults, subjective evaluations, blurred roles, and isolated facilities in data governance are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A schematic diagram of an embodiment of a data governance method in an embodiment of the present invention;

[0040] Figure 2 A reference diagram of an embodiment of the present invention;

[0041] Figure 3 A reference diagram of an embodiment of the present invention;

[0042] Figure 4 A reference diagram of an embodiment of the present invention;

[0043] Figure 5 The figure is a schematic diagram of an embodiment of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are protected by the present invention.

[0045] It should be noted that the terms "including", "comprising" and "having" and any variations thereof in the specification, claims and drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, terminal, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include unlisted steps or units, or may optionally further include other steps or units inherent to these processes, methods, products or devices. In the claims, specification and drawings of the present invention, relational terms such as "first" and "second" are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects.

[0046] Referring to "embodiments" herein means that the specific features, structures or characteristics described in connection with the embodiments may be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] With the rapid development of the digital economy, people's needs for aspects such as the value of data assets, data quality, data signing rights, data space, and data transactions are becoming increasingly urgent.

[0048] Although traditional relational databases and non-relational databases provide reliable data storage solutions, they are unable to provide a complete data full life cycle and management, and there are problems such as full process discontinuity, subjective evaluation, blurred roles, and facility isolation. A new technical means is needed to solve the above technical problems.

[0049] In view of this, the embodiments of the present invention provide a data governance method, terminal device and storage medium. By defining data production, processing, circulation, application, maintenance, display, and destruction, the data islands and process breakpoints caused by stage fragmentation in traditional databases are eliminated. A multi-dimensional data evaluation report is generated, and quantitative indicators are used to replace manual experience judgment, improving the objectivity and reproducibility of evaluation. A matrix-style responsibility system is constructed, binding roles such as data engineers and security engineers to the life cycle stages, and clarifying the boundaries of rights and responsibilities. Based on a unified data facility (platform, system, tool), multi-source heterogeneous data is integrated, breaking the closed architecture of traditional databases and supporting cross-system data governance operations. The problems of full process discontinuity, subjective evaluation, blurred roles, and facility isolation in data governance are solved.

[0050] In order to illustrate the technical solutions of the present invention, specific embodiments will be used for illustration below.

[0051] Figure 1 The schematic diagram of the implementation process of a data governance method provided by an embodiment of the present invention is shown. This method can be applied to a terminal device. The terminal device can be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, etc.

[0052] Specifically, the above data governance method may include the following steps S101 to S104.

[0053] Step S101, access or collect the data to be governed.

[0054] Automatically discover multi-source heterogeneous data (including databases, log files, API interfaces, etc.) through a metadata scanning tool (such as Apache Atlas).

[0055] Call Informatica or Apache NiFi to perform data extraction, for example, extract patient visit records from the hospital HIS system and convert them into JSON / Parquet format.

[0056] Temporarily store the original data in the / raw directory of Kafka or Hadoop HDFS to obtain the data to be governed.

[0057] Optionally, before step S101, there is also step S100:

[0058] Step S100, determine the size and selection of the data governance infrastructure according to the scale of data governance. The data facilities include a data platform, a data system, and data tools. The data facilities also include computer software and hardware, computers, storage hardware, application support, etc.

[0059] Optionally, call an ETL tool to perform data extraction to obtain the data to be governed.

[0060] Step S102, perform preset operations corresponding to the full life cycle stage on the data to be governed. The full life cycle stage includes a data production stage, a data processing stage, a data circulation stage, a data application stage, a data maintenance stage, a data display stage, and a data destruction stage.

[0061] Among them, referring to Figure 2 , Figure 2 is the functional architecture of the data governance system, including life cycle management, data facility support, a data evaluation system, and data role and responsibility division.

[0062] Among them, for the data life cycle management module:

[0063] Functional description: Covers the full - process management of data from generation to destruction, including the following core stages:

[0064] Data production: Data collection, metadata annotation, knowledge graph construction.

[0065] Data processing: Cleaning, structuring, directory management, metadata management, master data management, storage, analysis, and integration.

[0066] Data circulation: Ownership authentication, encrypted transmission, transactions driven by smart contracts.

[0067] Data application: Business logic execution (such as prediction model training, report generation).

[0068] Data maintenance: Version control, backup and recovery, metadata update.

[0069] Data presentation: Visualization modeling, interactive report generation.

[0070] Data destruction: Compliance verification, blockchain - based record destruction of evidence.

[0071] Involves:

[0072] Data knowledge graph: Represents the semantic associations between data entities through a graph structure (nodes - relationships - attributes).

[0073] Master data management: Unified management of core business entities.

[0074] Smart contract: An automated protocol based on blockchain that executes preset operations when the triggering conditions are met.

[0075] Among them, for the data facility support module:

[0076] Functional description: Provides the underlying technical infrastructure, which is divided into 3 types of core components:

[0077] Data platform:

[0078] Big data platforms (Hadoop, Spark);

[0079] Cloud platforms (AWS, Azure).

[0080] Data system:

[0081] Databases (MySQL, MongoDB);

[0082] Data warehouses (Snowflake, Redshift);

[0083] Data lakes (HDFS, S3);

[0084] Data middle - platforms (Alibaba Cloud, Tencent).

[0085] Data tools:

[0086] Governance tools (Apache Atlas, Collibra);

[0087] Analysis tools (Tableau, Power BI);

[0088] Security tools (OneTrust, AWS Lake Formation).

[0089] Involve:

[0090] Data middle platform: An enterprise-level data capability reuse platform that provides data integration, development, and service capabilities.

[0091] Data lake: A distributed system that supports low-cost storage of raw data (structured / unstructured).

[0092] ETL tool: A process tool that completes the processes of data extraction (Extract), transformation (Transform), and loading (Load).

[0093] Optionally, the data to be governed can be accessed, collected, or updated at the current moment. Specifically, in the process of data governance, the data to be governed can be accessed, collected, or updated at any current moment (real-time or near real-time), rather than just at a fixed time point. It is easy to understand that data governance is a dynamic and continuous process that can respond to data changes in real-time, ensuring the effectiveness, quality, and security of data at each life cycle stage. This approach makes data governance more flexible and efficient, adapting to the rapidly changing data environment to ensure the timeliness and accuracy of data management.

[0094] In step S103, perform multi-dimensional data evaluation corresponding to the full life cycle stage on the data to be governed to obtain an evaluation result, where the evaluation result includes a data value evaluation report, a data quality evaluation report, a data normalization evaluation report, and a data security evaluation report.

[0095] Among them, the data evaluation system module:

[0096] Normalization evaluation: Check whether the data conforms to industry standards (such as the HL7 medical data standard).

[0097] Data requirement matching degree: Verify whether the data meets business requirements (such as field integrity).

[0098] Security evaluation: Detect vulnerabilities (such as unauthorized access risks), compliance (such as GDPR).

[0099] Quality assessment: Calculate metrics such as accuracy, integrity, and consistency.

[0100] Value assessment: Estimate the economic value of data assets based on the cost approach, market approach, and income approach.

[0101] Involved in:

[0102] CVSS score: Common Vulnerability Scoring System, quantifying the security risk level (0 - 10 points).

[0103] Monte Carlo simulation: A statistical method for predicting the future income of data through a probability model.

[0104] Optionally, the data to be governed can be accessed, collected, or updated at the current moment.

[0105] Step S104, construct a matrix - type responsibility system according to the evaluation results. The responsibility system maps data engineers, data analysts, data managers, data security officers, and data quality assessors to the corresponding life - cycle stages in the full life - cycle stage.

[0106] Among them, data roles and responsibility modules:

[0107] Function description: Divide into 8 full - time roles according to the governance stage and evaluation requirements:

[0108] Data administrator: Responsible for data storage, backup, and permission allocation.

[0109] Data analyst: Perform data modeling and business insight extraction.

[0110] Data trader: Manage data circulation contracts and transaction settlements.

[0111] Data security officer: Implement encryption, desensitization, and audit tracking.

[0112] Data quality assessor: Define quality rules and generate evaluation reports.

[0113] Data value assessor: Construct a valuation model and conduct market benchmarking analysis.

[0114] Data engineer: Develop ETL processes and maintain data pipelines.

[0115] Data specification assessor: Review the degree of data standardization (such as field naming specifications).

[0116] Involved in:

[0117] Dynamic desensitization: Mask sensitive data in real - time according to user roles (such as only showing the last four digits of the ID card).

[0118] ABAC model (Attribute - Based Access Control): Dynamically authorize data operation permissions based on user attributes (department, position).

[0119] In the embodiments of the present invention, it is applied to data facilities, which include a data platform, a data system, and data tools.

[0120] The data facilities in this embodiment provide full-stack capability support for data governance from the infrastructure to the upper-layer applications by integrating multi-dimensional technical components.

[0121] Optionally, refer to Figure 3 , Figure 3 which is the business architecture diagram of the data governance system in this embodiment.

[0122] The governance objectives include:

[0123] Compliance (meeting regulatory requirements such as GDPR, Data Security Law, etc.).

[0124] Data assetization (enhancing data value and supporting business decisions).

[0125] Risk control (preventing risks such as data leakage and abuse).

[0126] The governance plan includes:

[0127] Formulating a data governance roadmap.

[0128] Defining a data governance maturity assessment model (such as the DCMM framework).

[0129] The execution layer includes:

[0130] Data quality management: Defining data quality standards (integrity, consistency), and establishing a process for repairing problem data.

[0131] Metadata management: Unifying the data catalog to support business personnel in quickly retrieving data assets.

[0132] Master data management: Maintaining the "single trusted source" of core business entities (customers, products).

[0133] Data security management: Implementing data classification and grading (such as public, confidential), and access control (RBAC model).

[0134] Data serviceization: Opening data capabilities in the form of APIs, data products, etc. (such as customer portrait API).

[0135] The support layer includes:

[0136] A data governance platform (integrating a data catalog, quality monitoring, and lineage analysis tools).

[0137] Automation tools (such as a data quality rule engine, a privacy computing platform).

[0138] Optionally, refer toFigure 4 , Figure 4 are the horizontal (data life cycle stage) and vertical (governance support capabilities) dimensions of the model architecture in this embodiment.

[0139] Horizontal dimension:

[0140] Data production includes data collection, metadata annotation, and knowledge graph construction.

[0141] Data processing includes data cleaning (duplicate removal, format standardization), data structuring, data catalog management (metadata indexing), metadata management (lineage tracking), master data management (core entity standardization), data storage (database / data lake), data analysis (modeling, aggregation), and data integration (ETL, API docking).

[0142] Data usage includes business logic execution (such as risk control model training).

[0143] Data circulation includes encrypted transmission and smart contract-driven transactions.

[0144] Data maintenance includes version control and backup recovery.

[0145] Data destruction includes compliance verification and blockchain evidence storage.

[0146] Vertical dimension:

[0147] Data governance tools: Technical tools to support stage tasks (such as using Great Expectations for data cleaning).

[0148] Data visualization: Visualization of stage results (such as generating reports with Tableau).

[0149] Data quality assessment: Quantifying the data health in the stage (such as integrity and accuracy).

[0150] Data value assessment: Calculating the economic value of data (such as valuation using the income approach).

[0151] Data security assessment: Detecting security risks (such as vulnerability scanning).

[0152] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: By defining data production, processing, circulation, application, maintenance, display, and destruction, the data islands and process breakpoints caused by stage fragmentation in traditional databases are eliminated. A multi-dimensional data evaluation report is generated, replacing manual experience judgment with quantitative indicators, which improves the objectivity and reproducibility of evaluation. A matrix-style responsibility system is constructed, binding roles such as data engineers and security engineers to the life cycle stages, and clarifying the boundaries of rights and responsibilities. Based on a unified data facility (platform, system, tool), multi-source heterogeneous data is integrated, breaking the closed architecture of traditional databases and supporting cross-system data governance operations. The problems of full-process faults, subjective evaluation, blurred roles, and facility islands in data governance are solved.

[0153] Traditional databases only provide storage functions and cannot cover the entire data life cycle. Based on this, an alternative embodiment of the present invention is proposed.

[0154] Step S102 also includes the following specific implementation manners.

[0155] Step S1021, perform a metadata annotation operation corresponding to data production on the data to be governed to obtain raw data. The metadata annotation operation includes constructing a data knowledge graph corresponding to the raw data;

[0156] Step S1022, perform a structured processing operation corresponding to the data processing stage on the raw data to obtain structured intermediate data;

[0157] Step S1023, perform a circulation operation corresponding to the data circulation stage on the structured intermediate data to obtain circulated data;

[0158] Step S1024, perform a business logic operation corresponding to the data application stage on the circulated data to obtain a business application result;

[0159] Step S1025, perform a version control management operation corresponding to the data maintenance stage according to the business application result to obtain versioned data;

[0160] Step S1026, perform a visual modeling analysis operation corresponding to the data display stage according to the versioned data to obtain an interactive visualization report;

[0161] Step S1027, if the interactive visualization report expires, destroy the interactive visualization report to complete the destruction operation corresponding to the data destruction stage.

[0162] In the embodiment of the present invention, in the data production stage, Apache Atlas can be used to perform metadata annotation on raw data (such as user logs), annotating the field meanings (such as user_id representing the unique user identifier) and data sources (log collection system).

[0163] Build a data knowledge graph, associate user logs with the customer master data table, and form a "user-behavior-device" relationship network.

[0164] In the data processing stage, duplicate removal (such as deleting duplicate order records) and missing field completion (such as filling in the default delivery address) can be performed through Spark SQL.

[0165] Structured processing can convert the cleaned unstructured logs (such as in JSON format) into structured tables and define the Schema (such as field types, constraints).

[0166] In the data circulation stage, data services can be published through the API gateway and access rate limiting can be configured.

[0167] In the data application stage, data services can be called in the risk control system to calculate the user credit score in real time.

[0168] In the data maintenance stage, Git can be used to manage data versions and record data snapshots after each ETL task.

[0169] In the data display stage, interactive dashboards can be generated through Tableau to support drill-down analysis.

[0170] In the data destruction stage, a visualization report expiration policy (such as automatic deletion after 30 days) can be set to perform secure erasure.

[0171] Optionally, the data that can be labeled by metadata annotation operations includes perception data, control data, mechanism data, model data, identification data, source data, training data, description data, algorithm data, structured data, semi-structured data, unstructured data, visualization data, trustworthy data, privacy data, monitoring data, spatio-temporal data, spatial data, environmental data, security data, qualified data, unqualified data, open data, open source data, industry data, enterprise data, personal data, public data, private data, resource data, etc.

[0172] Optionally, for the Internet of Things data classification system:

[0173] 1. Classified by industry: industrial data (device sensor data, production line logs), agricultural data (soil humidity, meteorological monitoring), scientific and technological data (R & D experiment data), financial data (payment transaction records), transportation data (vehicle trajectories, road condition monitoring), environmental data (air quality, water quality indicators), forestry data (forest coverage monitoring), government affairs data (public service records), smart city data (smart meters, security camera data).

[0174] Governance significance: Develop differentiated governance strategies according to the characteristics of different industries (such as strong encryption for financial data and real-time guarantee for environmental data).

[0175] 2. Classified by resource type: knowledge resource data (patent library, technical documents), water resource data (reservoir capacity, irrigation records), expert resource data (consulting reports, industry insights), and industry resource data (standardized data sets shared across enterprises).

[0176] Governance significance: clarify data ownership (e.g., water resources belong to the government), and standardize the boundaries of resource sharing (e.g., expert data needs to be de - sensitized before opening).

[0177] Optionally, in the data production stage, collect data by industry classification (such as industrial equipment sensor data), and define metadata tags (such as "equipment ID - industry - temperature").

[0178] Optionally, in the data processing stage, construct an industry knowledge graph (such as the association rule of "soil humidity → crop yield" in agricultural data).

[0179] Optionally, in the data circulation stage, set sharing permissions according to resource type (such as water resource data is only open to environmental protection departments).

[0180] In the embodiments of the present invention, by performing operations such as metadata annotation, cleaning, structuring, circulation, and maintenance in stages, the full - link control of data from production to destruction is realized. By constructing a data knowledge graph to explicitly express data relationships (such as field dependencies, business entity associations), the traceability of the governance process is improved. Automatically trigger destruction according to the timeliness of the visualization report, avoiding storage and compliance risks caused by the accumulation of redundant data.

[0181] The storage structure of the traditional solution is chaotic and difficult to support efficient analysis. Based on this, the present invention proposes an optional embodiment.

[0182] Refer to Figure 3 , Figure 3 is a schematic diagram of a specific embodiment of step S102 of the data governance method in the embodiments of the present invention, and step S1022 further includes the following specific implementation manners.

[0183] Step S10221, perform master data operations, generate data catalog operations, generate data dictionary operations, and generate data model operations on the cleaned data in sequence to obtain the structured intermediate data.

[0184] Among them, the master data operation can use Informatica MDM to integrate scattered customer information, generate a unique customer ID, and eliminate duplicate records.

[0185] Generate a data catalog to create a data catalog in Alation and classify data assets by business theme (such as "finance", "users").

[0186] The generated data dictionary can define the business meanings and compliance requirements of fields (e.g., the phone field needs to comply with GDPR privacy protection).

[0187] The generated data model can use ERWin to design a 3NF relational model and define foreign key constraints between tables.

[0188] Optionally, the obtained structured intermediate data may include metadata, master data, data catalog, data dictionary, data entity, data model, data mechanism, data space, data format, data specification, data characteristics, data type, data structure, data format, data type, data relationship, data dependency, data paradigm, data structure, data algorithm, data identifier, data encoding, source data, data source, raw data, data interface, classified data, etc.

[0189] In the embodiments of the present invention, the redundancy and ambiguity of core business entities (such as customers, products) are eliminated through master data operations, which can improve cross-system consistency. The generation of data catalogs and dictionaries enables business personnel to quickly retrieve data assets and reduce communication costs. The definition of data models (such as ER models) can make data processing conform to business logic and reduce the generation of dirty data.

[0190] Refer to Figure 4 , Figure 4 FIG.

[0191] Step S10231: Perform column-level permission control on the structured intermediate data according to the role and scenario mapping table in the dynamic permission policy to obtain controlled data.

[0192] Specifically, according to the role and scenario mapping table in the dynamic permission policy, call Apache Ranger to perform column-level permission control on the structured intermediate data to obtain controlled data.

[0193] Optionally, configure role permissions based on Apache Ranger. For example, the "data analyst" role can only access the user_id and age fields, and the phone and email fields are masked.

[0194] Step S10232: Perform encrypted transmission on the controlled data to obtain the circulated data.

[0195] Specifically, according to the encryption standard of GB / T 39786-2021, call the SM4 algorithm to perform encrypted transmission on the controlled data to obtain the circulated data.

[0196] Among them, the SM4 algorithm (CBC mode) is selected according to the GB / T 39786-2021 standard to generate a key and the key is managed by a key management system (KMS).

[0197] In the embodiment of the present invention, column-level permission management based on roles and scenarios (such as hiding the mobile phone number field) can prevent unauthorized access to sensitive data. Encrypted transmission (such as TLS / cryptography algorithm of China) can ensure data anti-theft and anti-tampering during data sharing.

[0198] Traditional solutions lack the ability of interactive analysis and insufficiently mine data value. Based on this, an alternative embodiment of the present invention is proposed.

[0199] Step S1026 further includes the following specific embodiments.

[0200] Step S10261: Perform modeling analysis on the versioned data according to the analysis dimensions in the business requirements to obtain a visual dataset.

[0201] Among them, import the versioned data into Tableau and establish a real-time connection with the data source. Select the analysis dimensions according to the business requirements and construct a basic analysis model (such as a sales trend chart, a user distribution heat map).

[0202] Perform aggregation calculation on the versioned data to generate a structured dataset that can be directly used for visualization.

[0203] Step S10262: Generate an interactive governance dashboard containing a drill-down analysis component according to the node relationships in the data knowledge graph to obtain an interactive visual report.

[0204] Among them, based on the data knowledge graph (such as Apache Atlas records), mark the data source in the Superset dashboard (such as the field comes from the Hive table dwd.sales).

[0205] Set the drill-down level in Superset, and click on the high-level chart to automatically link and display the lower-level detailed data.

[0206] Embed the interactive governance dashboard into the enterprise internal data portal to support multi-role access according to permissions (such as management viewing summary data and analysts viewing details).

[0207] In the embodiment of the present invention, through the drill-down analysis component, the root cause of data problems can be quickly located. Based on the visualization of data relationships, it can support the optimization of business decisions.

[0208] Traditional methods have a single evaluation dimension and cannot cover the entire life cycle. Based on this, an alternative embodiment of the present invention is proposed.

[0209] Step S104 further includes the following specific embodiments.

[0210] Step S1041: Generate responsibility cells including life cycle stages and quality defect types according to the null value rate and duplication rate indicators in the data quality assessment report.

[0211] Among them, parse the data quality assessment report and extract key indicators. For example, if it is found during the data processing stage that the null value rate of a certain field exceeds the standard (>5%), it is marked as "integrity defect".

[0212] Map the quality defect types to the corresponding life cycle stages.

[0213] Generate two-dimensional matrix cells, with the horizontal axis being the life cycle stages and the vertical axis being the defect types. Example cell: data processing stage - integrity defect.

[0214] Step S1042: Assign roles to the responsibility cells to construct a matrix-based responsibility system.

[0215] Among them, according to the preset post ability matrix, assign responsible persons to the responsibility cells. For example, the data engineer is responsible for fixing the null value problem during the data processing stage, and the data quality assessor is responsible for monitoring and improving the effect.

[0216] Configure role permissions through the RBAC model.

[0217] In the embodiment of the present invention, the matrix cells display the distribution of governance problems across stages and dimensions. The data engineer and the quality assessor jointly optimize the cleaning rules, which can reduce the null value rate to.

[0218] Such as Figure 5 As shown, it is a schematic diagram of a terminal device provided by an embodiment of the present invention. The terminal device 5 may include: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a data governance program. When the processor 501 executes the computer program 503, the steps in the above-mentioned various data governance embodiments are implemented.

[0219] The computer program can be divided into one or more modules / units. One or more modules / units are stored in the memory 502 and executed by the processor 501 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device.

[0220] The terminal device may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art can understand that Figure 5These are merely examples of terminal devices and do not constitute limitations on terminal devices. They may include more or fewer components than shown in the figures, or combine certain components, or have different components. For example, a terminal device may also include input / output devices, network access devices, buses, etc.

[0221] The so-called processor 501 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0222] The memory 502 may be an internal storage unit of the terminal device, such as the hard disk or memory of the terminal device. The memory 502 may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the terminal device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 502 may also include both an internal storage unit and an external storage device of the terminal device. The memory 502 is used to store computer programs and other programs and data required by the terminal device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0223] It should be noted that for the convenience and brevity of description, the structure of the above terminal device may also refer to the specific description of the structure in the method embodiments, which will not be elaborated here.

[0224] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above data governance method can be implemented.

[0225] An embodiment of the present invention provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can execute the steps in the above data governance method.

[0226] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0227] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0228] In the embodiments provided by the present invention, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. Another point is that the couplings or direct couplings or communication connections shown or discussed among each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0231] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0232] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features. And these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data governance method, characterized in that, Applied to a data facility, the data facility includes a data platform, a data system, and data tools. The method includes: Accessing or collecting the data to be governed; Performing preset operations corresponding to the full life cycle stages on the data to be governed. The full life cycle stages include a data production stage, a data processing stage, a data circulation stage, a data application stage, a data maintenance stage, a data display stage, and a data destruction stage; Performing multi-dimensional data evaluation corresponding to the full life cycle stages on the data to be governed to obtain an evaluation result. The evaluation result includes a data value evaluation report, a data quality evaluation report, a data standardization evaluation report, and a data security evaluation report; Constructing a matrix-style responsibility system according to the evaluation result. The responsibility system maps data engineers, data analysts, data managers, data security engineers, and data quality evaluators to the corresponding life cycle stages in the full life cycle stage.

2. The data governance method according to claim 1, wherein The step of performing preset operations corresponding to the full life cycle stages on the data to be governed includes: Performing a metadata annotation operation corresponding to data production on the data to be governed to obtain raw data. The data annotation operation includes constructing a data knowledge graph corresponding to the raw data; Performing a data cleaning operation on the data to be governed to obtain cleaned data. The data annotation includes constructing a data knowledge graph corresponding to the cleaned data; Performing a structured processing operation corresponding to the data processing stage on the cleaned data to obtain structured intermediate data; Performing a circulation operation corresponding to the data circulation stage on the structured intermediate data to obtain circulated data; Performing a business logic operation corresponding to the data application stage on the circulated data to obtain a business application result; Performing a version control management operation corresponding to the data maintenance stage according to the business application result to obtain versioned data; Performing a visual modeling analysis operation corresponding to the data display stage according to the versioned data to obtain an interactive visual report; If the interactive visual report expires, destroying the interactive visual report to complete the destruction operation corresponding to the data destruction stage.

3. The data governance method according to claim 2, wherein The step of performing a structured processing operation corresponding to the data processing stage on the cleaned data to obtain structured intermediate data includes: Performing a master data operation, a data catalog generation operation, a data dictionary generation operation, and a data model generation operation on the cleaned data in sequence to obtain the structured intermediate data.

4. The data governance method according to claim 2, wherein The step of performing a circulation operation corresponding to the data circulation stage on the structured intermediate data to obtain circulated data includes: Performing column-level permission control on the structured intermediate data according to the role and scenario mapping table in the dynamic permission policy to obtain controlled data; Performing encrypted transmission on the controlled data to obtain the circulated data.

5. The data governance method according to claim 2, wherein The step of performing a visual modeling analysis operation corresponding to the data display stage according to the versioned data to obtain an interactive visual report includes: Performing modeling analysis on the versioned data according to the analysis dimensions in the business requirements to obtain a visual data set; Generate an interactive governance dashboard containing a drill-down analysis component based on the node relationships in the data knowledge graph to obtain an interactive visualization report.

6. The data governance method according to claim 1, wherein The step of accessing or collecting the data to be governed includes: Execute data extraction to obtain the data to be governed.

7. The data governance method according to claim 1, wherein The step of constructing a matrix-based responsibility system according to the evaluation result includes: Generate responsibility cells containing life cycle stages and quality defect types based on the null value rate and duplication rate indicators in the data quality assessment report; Assign roles to the responsibility cells to construct the matrix-based responsibility system.

8. The data governance method according to claim 7, wherein The step of assigning roles to the responsibility cells to construct the matrix-based responsibility system includes: Assign roles to the responsibility cells according to the position ability matrix in the preset RBAC permission model to construct the matrix-based responsibility system.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data governance method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the data governance method according to any one of claims 1 to 8 are implemented.

Images