A data governance method, apparatus, device and medium

Abstract

The application provides a data governance method, device, equipment and medium, comprising defining a field block and a data domain under the field block; defining a business process under the data domain; developing a physical model and data of the business process according to business process information; business process quality inspection; confirming dimensions according to the business process information, and defining the confirmed dimensions; developing a physical model and data of the dimensions according to the dimension information; dimension quality inspection; defining atomic indicators and business limits according to the business process information; defining business indicators according to the business process information, the dimension information, the atomic indicator information and the business limit information, and automatically generating a logical model of the business indicators; developing a physical model and data of the business indicators according to the business indicator information and the logical model of the business indicators; and business indicator quality inspection. The application can reduce repeated development of data, reduce data quality problems, reduce false reports of data quality, and improve data governance efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data governance method, device, equipment and medium. BACKGROUND

[0002] Big data governance is mainly realized based on related technologies such as data collection, data storage, metadata, data standards, data quality, data processing and data sharing. The general process of data governance is as follows: starting from data collection, first collecting business data, then formulating data standards, and then connecting the data standards and the business data model through the collection, registration, scanning and publishing of metadata management; finally, the business data model is audited through the data quality audit rules associated with the data standards, and if quality problems are found, quality reports are formed to promote the corresponding business departments to rectify.

[0003] For example, a government data governance method, device, storage medium and electronic equipment disclosed in Chinese patent application No. CN115878592A include obtaining government data governance requirements and obtaining government data sets according to the government data governance requirements; based on the theme of government data in the government data sets, a data warehouse is established through a large-scale parallel processing architecture, and a data access model is established based on the data warehouse; based on the data access model, the government data sets are accessed through a preset data access method; the government data sets are processed to obtain target government data sets; based on the target government data sets, the target government data sets are output through a data sharing method; government data governance operations are completed based on the target government data sets; the obtained government data sets are processed to improve data quality; the scheme provides a complete government data specification governance system, efficiently manages the whole process life cycle from data access to data sharing, and achieves the effect of government data governance. For example, a data governance method, device, computer equipment, storage medium and computer program product disclosed in Chinese patent application No. CN115858513A include database statement analysis processing of a table creation statement of a business data table to be governed to obtain various data table information contained in the business data table, after which the business data table to be governed is taken as an entity, and based on the data table information, the business data table to be governed is subjected to library table correlation relationship mining processing to obtain data table correlation relationship; then, based on the partitioned business data table and the data table correlation relationship, the business data table is subjected to business theme merging processing to obtain business data tables under each business theme, and the business data relationship is used to provide data support for business theme merging, and then the business data tables under each business theme are subjected to data standardization governance to obtain data governance results, thereby realizing metadata governance and standardization processing between different business modules

[0004] However, the prior art lacks unified data definition from the perspective of business when governing big data, the calculation caliber of business data is not unified, the technical personnel is easy to have inconsistent understanding of the business meaning of data, which makes the audit condition of data processing easy to have errors, thereby leading to repeated development of data, frequent data quality problems and low efficiency of governance. In view of the above problems, the present inventors have conducted in-depth research on the problem, and thus the present case arises. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a data governance method, device, equipment and medium, which solves the problem of lack of unified data definition from the perspective of business when governing big data in the prior art, the calculation caliber of business data is not unified, the technical personnel is easy to have inconsistent understanding of the business meaning of data, which makes the audit condition of data processing easy to have errors, thereby leading to repeated development of data, frequent data quality problems and low efficiency of governance.

[0006] In a first aspect, the present application provides a data governance method, comprising the following steps:

[0007] Step S1, defining a field block and data domains belonging to the field block to obtain field block information and data domain information;

[0008] Step S2, defining a business process belonging to the data domain to obtain business process information, the business process information at least including a Chinese name of the business process, an English name of the business process, a description of the business process and a logic model of the business process;

[0009] Step S3, developing a physical model and data of the business process according to the business process information, the physical model and data development script of the business process being automatically generated through the logic model defined by the business process;

[0010] Step S4, after the development of the business process is completed, configuring a business process quality inspection rule, and performing quality inspection on the business process information according to the business process quality inspection rule;

[0011] Step S5, defining the dimensions according to the business process information, obtaining dimension information, the dimension information at least including a Chinese name of the dimension, an English name of the dimension, a description of the dimension and a logic model of the dimension;

[0012] Step S6, developing a physical model and data of the dimension according to the dimension information, the physical model and data development script of the dimension being automatically generated through the logic model defined by the dimension;

[0013] Step S7, after the development of the dimension is completed, configuring a dimension quality inspection rule, and performing quality inspection on the dimension information according to the dimension quality inspection rule;

[0014] Step S8, defining atomic indicators and business limits according to the business process information to obtain atomic indicator information and business limit information; defining business indicators according to the business process information, the dimension information, the atomic indicator information and the business limit information to obtain business indicator information, and automatically generating a logic model of the business indicators according to the business indicator information;

[0015] Step S9, developing a physical model of the business indicators and data according to the business indicator information and the logic model of the business indicators, and automatically generating the physical model of the business indicators and data development scripts through the logic model of the business indicators;

[0016] Step S10, after the development of the business indicators is completed, configuring business indicator quality inspection rules, and performing quality inspection on the business indicator information according to the business indicator quality inspection rules.

[0017] Further, in step S2, the logic model of the business process at least includes a Chinese name of the logic model, an English name of the logic model, an explanation of the logic model and an attribute field of the logic model; the attribute field at least includes a Chinese name of the attribute field, an English name of the attribute field, a data type of the attribute field, a data length of the attribute field, a value range of the attribute field, a business scope of the attribute field and a mapping relationship between the attribute field and a data standard;

[0018] In step S3, the physical model of the business process and the data development scripts are automatically generated through the logic model defined by the business process, and specifically include:

[0019] The table name of the physical model of the business process is automatically generated through the English name of the logic model defined by the business process; the field information of the physical model of the business process is automatically generated through the attribute field of the logic model defined by the business process, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model;

[0020] The data development scripts of the business process are automatically generated through the business scope of the attribute field defined by the business process.

[0021] Further, in step S4, the business process quality inspection rules are automatically generated through the mapping relationship between the attribute field defined by the business process and the data standard, and specifically include:

[0022] The name, type, length or value range of the data element of the business process is defined in the data standard in advance to obtain data element information, and the attribute field defined by the business process is corresponded to the corresponding data element to form a mapping relationship; the name specification, data type consistency, data length integrity or value range accuracy inspection rules are automatically generated according to the data element information.

[0023] Further, in step S5, the logical model of the dimension at least includes the Chinese name of the dimension, the English name of the dimension, the description of the dimension and the attribute field of the dimension; the attribute field at least includes the Chinese name of the attribute field, the English name of the attribute field, the data type of the attribute field, the data length of the attribute field, the value range of the attribute field, the business scope of the attribute field and the mapping relationship between the attribute field and the data standard;

[0024] In step S6, the physical model of the dimension and the data development script are automatically generated through the logical model defined by the dimension, specifically including: the table name of the physical model of the dimension is automatically generated through the English name of the logical model defined by the dimension; the field information of the physical model of the dimension is automatically generated through the attribute field of the logical model defined by the dimension, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model;

[0025] The data development script of the dimension is automatically generated through the business scope of the attribute field defined by the dimension.

[0026] Further, in step S7, the dimension inspection rule is automatically generated through the mapping relationship between the attribute field defined by the dimension and the data standard, specifically including:

[0027] The name, type, length or value range of the data element of the dimension is defined in the data standard in advance to obtain data element information, and the attribute field defined by the dimension is corresponded to the corresponding data element to form a mapping relationship; the name specification, data type consistency, data length integrity or value range accuracy inspection rules are automatically generated according to the data element information.

[0028] Further, in step S8, the atomic index is a reusable minimum granularity statistical unit set in the business process, the business limit is a reusable business rule condition set in the business process, and the business index information includes a Chinese name of the business index, an English name of the business index, a description of the business index, a business process to which the business index belongs, an atomic index used by the business index, a business limit used by the business index, a dimension involved in the business index, a statistical period of the business index, or a statistical time of the business index.

[0029] The logic model of the business index at least includes a Chinese name of the logic model, an English name of the logic model, a description of the logic model, and an attribute field of the logic model; wherein the Chinese name of the logic model is automatically generated through the Chinese name of the business index, the English name of the logic model is automatically generated through the English name of the business index, the description of the logic model is automatically generated through the description of the business index, and the attribute field of the logic model is composed of the business process, the atomic index, the business limit, the dimension, the statistical period, or the statistical time corresponding to the business index, and at least includes a Chinese name of the attribute field, an English name of the attribute field, a data type of the attribute field, a data length of the attribute field, a value range of the attribute field, a business scope of the attribute field, and a mapping relationship between the attribute field and a data standard.

[0030] Further, in step S9, the physical model of the business index and the data development script are automatically generated through the logic model of the business index and specifically include:

[0031] The table name of the physical model of the business index is automatically generated through the English name of the logic model of the business index, and the field information of the physical model of the business index is automatically generated through the attribute field of the logic model of the business index, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model.

[0032] The data development script of the business index is automatically generated through the business scope of the attribute field of the business index, and specifically includes: the sql script of the business index is automatically generated through the combination of the business process, the atomic index, the business limit, the dimension, the statistical period, and the statistical time; wherein the sql source table of the business index is obtained through the business process, the sql statistical field of the business index is obtained through the atomic index, the sql filtering condition of the business index is obtained through the business limit, the sql grouping condition of the business index is obtained through the dimension, the scheduling period of the business index is obtained through the statistical period, and the sql query time condition of the business index is obtained through the statistical time.

[0033] In step S10, the business index quality inspection rule is automatically generated through the mapping relationship between the attribute field of the business index and the data standard; wherein the mapping relationship between the attribute field of the business index and the data standard includes the mapping relationship between the attribute field and the data standard in the business process and the mapping relationship between the attribute field and the data standard in the dimension.

[0034] In a second aspect, the present application provides a data governance device, comprising a field and data domain definition module, a business process definition module, a business process development module, a business process quality inspection module, a dimension definition module, a dimension development module, a dimension quality inspection module, a business index definition module, a business index development module and a business index quality inspection module.

[0035] The field and data domain definition module is configured to define field plates and data domains belonging to the field plates, and obtain field plate information and data domain information.

[0036] The business process definition module is configured to define business processes belonging to the data domains, and obtain business process information, which at least includes the Chinese name of the business process, the English name of the business process, the description of the business process and the logic model of the business process.

[0037] The business process development module is configured to develop the physical model and data of the business process according to the business process information, and the physical model and data development script of the business process are automatically generated through the logic model defined by the business process.

[0038] The business process quality inspection module is configured to configure business process quality inspection rules after the development of the business process is completed, and to perform quality inspection on the business process information according to the business process quality inspection rules.

[0039] The dimension definition module is configured to confirm dimensions according to the business process information, define the confirmed dimensions, and obtain dimension information, which at least includes the Chinese name of the dimension, the English name of the dimension, the description of the dimension and the logic model of the dimension.

[0040] The dimension development module is configured to develop the physical model and data of the dimension according to the dimension information, and the physical model and data development script of the dimension are automatically generated through the logic model defined by the dimension.

[0041] The dimension quality inspection module is configured to configure dimension quality inspection rules after the development of the dimension is completed, and to perform quality inspection on the dimension information according to the dimension quality inspection rules.

[0042] The business indicator definition module is configured to define atomic indicators and business limits according to business process information, to obtain atomic indicator information and business limit information, to define business indicators according to the business process information, dimension information, atomic indicator information and business limit information, and to obtain business indicator information, and to automatically generate a logical model of the business indicators according to the business indicator information.

[0043] The business indicator development module is configured to develop a physical model and data of the business indicators according to the business indicator information and the logical model of the business indicators, and to automatically generate a development script of the physical model and data of the business indicators according to the logical model of the business indicators.

[0044] The business indicator quality inspection module is configured to configure a business indicator quality inspection rule after the development of the business indicators, and to perform quality inspection on the business indicator information according to the business indicator quality inspection rule.

[0045] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect when executing the program.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the method of the first aspect.

[0047] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: a new method of big data governance is established based on the dimension modeling theory, taking the business perspective as the main line, the business data is uniformly defined through the domain plate, data domain, business process, dimension, atomic indicator, business limit, and business indicator, a unified data definition is provided from the business perspective, the calculation caliber of the business data is unified, the data development script and the quality inspection rule are automatically generated based on the definition information, the technical personnel can reach a consistent understanding of the business meaning of the data, the repeated development of the data can be effectively reduced, the data quality problems can be reduced, the false positives of the data quality can be reduced, and the data governance efficiency can be improved.

[0048] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the embodiments of the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0049] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0050] Figure 1 The figure is an execution flowchart of the data governance method in the first embodiment of the present application.

[0051] Figure 2 Fig. 1 is a structural schematic diagram of a data governance device according to an embodiment of the present application;

[0052] Figure 3 Fig. 2 is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0053] Figure 4 Fig. 3 is a structural schematic diagram of a medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to better understand the technical solutions of the present application, the technical solutions of the present application will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0055] Embodiment 1

[0056] The present embodiment provides a data governance method, as shown in Figure 1 The method comprises the following steps:

[0057] Step S1, defining a field block to obtain field block information, the field block specifically refers to a business field facing the data governance; by developing business perspective management on the business data of the same field block, subsequent data governance work on the business data of the same field block can be facilitated; when defining the field block, the field block information at least includes the name of the field block, the number of the field block and the description of the field block, of course, in specific implementation, the field block information can also increase other information according to actual needs;

[0058] After defining the field block, define the data domain belonging to the field block to obtain data domain information, the data domain is a further classification of the business data under the field block, which is a collection of subsequent business data; when defining the data domain, the data domain information at least includes the name of the data domain, the number of the data domain and the description of the data domain, of course, in specific implementation, the data domain information can also increase other information according to actual needs.

[0059] Step S2, defining the business process belonging to the data domain to obtain business process information, the business process is a specific business activity under the data domain; the business process information at least includes the Chinese name of the business process, the English name of the business process, the description of the business process and the logic model of the business process;

[0060] In the step S2, the logic model of the business process at least includes a Chinese name of the logic model, an English name of the logic model, a description of the logic model, and an attribute field of the logic model; the attribute field at least includes a Chinese name of the attribute field, an English name of the attribute field, a data type of the attribute field, a data length of the attribute field, a value range of the attribute field, a business caliber of the attribute field, and a mapping relationship between the attribute field and a data standard; in the specific implementation, the data source table and the source field corresponding to the attribute field in the business process can be parsed through the business caliber of the attribute field, and the attribute field defined in the business process can be automatically checked through the mapping relationship between the attribute field and the data standard.

[0061] In step S3, the physical model and the data development script of the business process are automatically generated according to the business process information defined by the logic model of the business process.

[0062] In step S3, the physical model and the data development script of the business process are automatically generated according to the business process information defined by the logic model of the business process.

[0063] The table name of the physical model of the business process is automatically generated through the English name of the logic model defined by the business process; the field information of the physical model of the business process is automatically generated through the attribute field of the logic model defined by the business process, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model.

[0064] The data development script of the business process is automatically generated through the business caliber of the attribute field defined by the business process; in the specific implementation, the data source table and the source field corresponding to the attribute field in the business process can be parsed through the business caliber of the attribute field defined by the business process, so as to determine the association relationship between the physical model table and the data source table, and automatically generate the sql script for the development of the business process, thereby realizing the standardized and automatic development of the business process.

[0065] In step S4, after the development of the business process is completed, the business process quality inspection rules are configured, and the business process information is inspected according to the business process quality inspection rules, including the checks of standardization, integrity, accuracy, consistency, etc.

[0066] In step S4, the business process quality inspection rules are automatically generated through the mapping relationship between the attribute field and the data standard defined by the business process, and specifically include:

[0067] The name, type, length or value range of the data element of the business process is defined in the data standard in advance, data element information is obtained, the attribute field defined by the business process is corresponded to the corresponding data element to form a mapping relationship, that is, the mapping relationship between the attribute field defined by the business process and the data standard is obtained; the name specification quality inspection rule, the data type consistency quality inspection rule, the data length integrity quality inspection rule or the value range accuracy quality inspection rule is automatically generated according to the data element information, so as to realize the automatic standard specification inspection of the data quality of the business process, reduce manual intervention, reduce the false positive rate of data quality, and improve the standardization compliance of data.

[0068] In specific implementation, after the name specification quality inspection rule, the data type consistency quality inspection rule, the data length integrity quality inspection rule or the value range accuracy quality inspection rule is generated, whether the name of the attribute field defined by the business process is standardized is automatically checked according to the name specification quality inspection rule, whether the data type of the attribute field defined by the business process is consistent is automatically checked according to the data type consistency quality inspection rule, whether the data length of the attribute field defined by the business process is complete is automatically checked according to the data length integrity quality inspection rule, or whether the value range of the attribute field defined by the business process is accurate is automatically checked according to the value range accuracy quality inspection rule; of course, the present application is not limited to this, and other quality inspection rules can also be set according to actual needs in specific implementation, for example, the quality inspection rules can also include timeliness.

[0069] Step S5, confirming the dimension according to the business process information, the dimension refers to the related business object participating in the business activity, such as person, time, place, thing and organization; by confirming the business object involved in the business process, the related business objects such as person, time, place, thing and organization involved in the business process are determined, which can facilitate subsequent index analysis of the business process based on the dimension;

[0070] The confirmed dimension is defined to obtain dimension information, the dimension information at least includes the Chinese name of the dimension, the English name of the dimension, the description of the dimension and the logical model of the dimension; the logical model of the dimension at least includes the Chinese name of the dimension, the English name of the dimension, the description of the dimension and the attribute field of the dimension; the attribute field at least includes the Chinese name of the attribute field, the English name of the attribute field, the data type of the attribute field, the data length of the attribute field, the value range of the attribute field, the business scope of the attribute field and the mapping relationship between the attribute field and the data standard; in specific implementation, the data source table and the source field corresponding to the attribute field in the dimension can be parsed through the business scope of the attribute field, and the attribute field defined by the dimension can be automatically checked through the mapping relationship between the attribute field and the data standard.

[0071] Step S6: Based on the dimension information, the physical model and data development work of the dimension is carried out. The physical model and data development script of the dimension are automatically generated based on the logical model defined by the dimension.

[0072] In step S6, the physical model and data development script of the dimension are automatically generated from the logical model defined by the dimension, specifically including:

[0073] The table name of the dimension's physical model is automatically generated based on the English name of the logical model defined by the dimension. The field information of the dimension's physical model is automatically generated based on the attribute fields of the logical model defined by the dimension. The Chinese names of the attribute fields correspond to the field descriptions of the physical model, the English names of the attribute fields correspond to the field names of the physical model, the data types of the attribute fields correspond to the field types of the physical model, and the data lengths of the attribute fields correspond to the field lengths of the physical model.

[0074] The data development script of the dimension is automatically generated through the business scope of the attribute fields defined by the dimension. During specific implementation, the business scope of the attribute fields defined by the dimension can be used to parse the data source table and source field corresponding to the attribute fields in the dimension, thereby determining the association between the physical model table and the data source table, and automatically generating the SQL script for dimension development, realizing the standardized and automated development of the dimension.

[0075] Step S7: After the dimension development is completed, configure the dimension quality inspection rules and perform quality inspection on the dimension information according to the dimension quality inspection rules;

[0076] In step S7, dimension quality inspection rules are automatically generated based on the mapping relationship between the attribute fields defined in the dimension and the data standards, specifically including:

[0077] The name, type, length, or value range of the dimension's data element are pre-defined in the data standard to obtain data element information. The attribute fields defined in the dimension are then mapped to the corresponding data elements to form a mapping relationship. Based on the data element information, quality inspection rules for name standardization, data type consistency, data length integrity, or value range accuracy are automatically generated to achieve automated standard inspection of dimension data quality, reduce manual intervention, reduce the false positive rate of data quality, and improve data standardization compliance.

[0078] In the embodiment of the present application, after the name specification property inspection rule, the data type consistency property inspection rule, the data length integrity property inspection rule or the value range accuracy property inspection rule is generated, the name of the attribute field defined by the dimension is automatically inspected according to the name specification property inspection rule, the data type of the attribute field defined by the dimension is automatically inspected according to the data type consistency property inspection rule, the data length of the attribute field defined by the dimension is automatically inspected according to the data length integrity property inspection rule or the value range of the attribute field defined by the dimension is automatically inspected according to the value range accuracy property inspection rule. Of course, the present application is not limited to this, and other property inspection rules can be set according to actual needs in the embodiment, such as the timeliness property inspection rule.

[0079] In step S8, the atomic index and the business limit are defined according to the business process information, and the atomic index information and the business limit information are obtained; the business index is defined according to the business process information, the dimension information, the atomic index information and the business limit information, that is, the business index is formed by combining the business process information, the dimension information, the atomic index information and the business limit information to form a statistical index meeting a specific business analysis requirement, the business index information is obtained, and the logic model of the business index is automatically generated according to the business index information.

[0080] In this step S8, the atomic index is a reusable minimum granularity statistical unit set in the business process, and when the atomic index is defined, the atomic index information includes the name of the atomic index, the business process to which the atomic index belongs, the statistical attribute of the atomic index or the statistical range of the atomic index. Of course, in the embodiment, the atomic index information can also be increased with other information according to actual needs. The present application sets the reusable minimum granularity statistical unit (i.e. the atomic index) in the business process, which can facilitate the reuse of the atomic index to reduce data duplication development.

[0081] In this step S8, the business limit is a reusable business rule condition set in the business process, and when the business limit is defined, the business limit information includes the name of the business limit, the business process to which the business limit belongs, the attribute of the business limit or the technical range of the business limit. Of course, in the embodiment, the business limit information can also be increased with other information according to actual needs. The present application sets the reusable business rule condition (i.e. the business limit) in the business process, which can facilitate the reuse of the business limit to reduce data duplication development.

[0082] In this step S8, the business index information includes the Chinese name of the business index, the English name of the business index, the description of the business index, the business process to which the business index belongs, the atomic index used by the business index, the business limit used by the business index, the dimension involved by the business index, the statistical period of the business index or the statistical time of the business index.

[0083] In this step S8, the logic model of the business indicator at least includes the Chinese name of the logic model, the English name of the logic model, the description of the logic model and the attribute field of the logic model; wherein the Chinese name of the logic model is automatically generated by the Chinese name of the business indicator, the English name of the logic model is automatically generated by the English name of the business indicator, the description of the logic model is automatically generated by the description of the business indicator, the attribute field of the logic model is composed of the business process, the atomic indicator, the business limit, the dimension, the statistical period or the statistical time corresponding to the business indicator, and the attribute field at least includes the Chinese name of the attribute field, the English name of the attribute field, the data type of the attribute field, the data length of the attribute field, the value range of the attribute field, the business scope of the attribute field and the mapping relationship between the attribute field and the data standard.

[0084] Step S9, according to the business indicator information and the logic model of the business indicator, the physical model of the business indicator and the data development work are carried out, and the physical model of the business indicator and the data development script are automatically generated by the logic model of the business indicator;

[0085] In this step S9, the physical model of the business indicator and the data development script are automatically generated by the logic model of the business indicator, which specifically includes:

[0086] The table name of the physical model of the business indicator is automatically generated by the English name of the logic model of the business indicator, and the field information of the physical model of the business indicator is automatically generated by the attribute field of the logic model of the business indicator, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model;

[0087] The data development script of the business index is automatically generated through the business scope of the attribute field of the business index, and specifically includes: automatically generating the sql script of the business index through the combination of the business process, the atomic index, the business limit, the dimension, the statistical period and the statistical time; wherein, the sql source table of the data acquisition of the business index is obtained through the business process (as the business scope of the attribute field defined through the business process can be parsed to obtain the corresponding data source table, thus the sql source table of the data acquisition corresponding to the business index can be known through the business process), the sql statistical field of the business index is obtained through the atomic index, the sql filtering condition of the business index is obtained through the business limit, the sql grouping condition of the business index is obtained through the dimension, the scheduling period of the business index is obtained through the statistical period, and the sql query time condition of the business index is obtained through the statistical time; thus, the standardized and automated development of the business index can be well realized through the technical scheme of the application; it should be noted that: as the attribute field of the logical model of the business index is composed of the business process, the atomic index, the business limit, the dimension, the statistical period or the statistical time corresponding to the business index, and the attribute field of the atomic index, the business limit, the statistical time and the statistical period comes from the business process, so the attribute field of the business index mainly comes from the attribute field of the business process and the dimension.

[0088] In step S10, after the development of the business index is completed, the business index quality inspection rule is configured, and the business index information is inspected according to the business index quality inspection rule.

[0089] In this step S10, the business index quality inspection rule is automatically generated through the mapping relationship between the attribute field of the business index and the data standard; wherein, the mapping relationship between the attribute field of the business index and the data standard includes the mapping relationship between the attribute field in the business process and the data standard and the mapping relationship between the attribute field in the dimension and the data standard.

[0090] As the attribute field of the logical model of the business index is composed of the business process, the atomic index, the business limit, the dimension, the statistical period or the statistical time corresponding to the business index, and the attribute field of the atomic index, the business limit, the statistical time and the statistical period comes from the business process, so the attribute field of the business index mainly comes from the attribute field of the business process and the dimension, thus the attribute field of the business index can inherit the mapping relationship between the attribute field in the business process and the data standard and the mapping relationship between the attribute field in the dimension and the data standard, and the name specification quality inspection rule, the data type consistency quality inspection rule, the data length integrity quality inspection rule or the value range accuracy quality inspection rule can be automatically generated according to the data element information in the business process and the data element information in the dimension, so as to realize the automatic standard specification inspection of the data quality of the business index, reduce the manual intervention, reduce the false positive rate of the data quality, and improve the standardization compliance of the data.

[0091] The data governance method of the application is realized based on a dimension modeling theory, the dimension modeling theory is proposed by a data warehouse master Ralph Kimball, and is a classical warehouse modeling in a data warehouse engineering field; the dimension modeling constructs a model starting from an analysis decision demand, a data model constructed serves the analysis demand, and therefore it focuses on how the user completes the analysis demand more quickly, and meanwhile has better response performance of large-scale complex query; the dimension modeling is analysis-oriented, in order to improve the query performance, a design technology of data redundancy and denormalization can be added; in general, the dimension modeling is an abstracted method system of data organization and management in an enterprise data warehouse, is various business data tables in the organization and management data warehouse, how to organize and create the table based on a business scene, a business process and a business demand, and finally provides efficient query performance.

[0092] In summary, the application establishes a new big data governance method with a business perspective as a main line based on the dimension modeling theory, uniformly defines business data through a field plate, a data field, a business process, a dimension, an atomic index, a business limit and a business index, provides a unified data definition with a business perspective, unifies a calculation caliber of the business data, and automatically generates data development scripts and quality inspection rules based on definition information, so that technical personnel reach a consistent understanding of the business meaning of the data, can effectively reduce data repeated development, reduce data quality problems, reduce data quality false positives, and improve data governance efficiency.

[0093] Based on the same inventive concept, the application also provides a device corresponding to the method in embodiment one, which is specifically shown in embodiment two.

[0094] Embodiment two

[0095] In this embodiment, a data governance device is provided, as shown in Figure 2 which includes a field and data field definition module, a business process definition module, a business process development module, a business process quality inspection module, a dimension definition module, a dimension development module, a dimension quality inspection module, a business index definition module, a business index development module and a business index quality inspection module.

[0096] The field and data field definition module is used for defining a field plate and a data field belonging to the field plate, and obtaining field plate information and data field information.

[0097] The business process definition module is used for defining a business process belonging to the data field, and obtaining business process information, the business process information at least including a Chinese name of the business process, an English name of the business process, a description of the business process and a logic model of the business process.

[0098] The business process development module is configured to develop a physical model and data of the business process according to the business process information, and a development script of the physical model and data of the business process is automatically generated according to a logical model of the business process definition;

[0099] The business process quality inspection module is configured to configure a business process quality inspection rule after the development of the business process is completed, and to perform quality inspection on the business process information according to the business process quality inspection rule;

[0100] The dimension definition module is configured to confirm dimensions according to the business process information, to define the confirmed dimensions, and to obtain dimension information, which at least includes a Chinese name of the dimension, an English name of the dimension, a description of the dimension, and a logical model of the dimension;

[0101] The dimension development module is configured to develop a physical model and data of the dimension according to the dimension information, and a development script of the physical model and data of the dimension is automatically generated according to a logical model of the dimension definition;

[0102] The dimension quality inspection module is configured to configure a dimension quality inspection rule after the development of the dimension is completed, and to perform quality inspection on the dimension information according to the dimension quality inspection rule;

[0103] The business indicator definition module is configured to define an atomic indicator and a business limit according to the business process information, to obtain atomic indicator information and business limit information, to define a business indicator according to the business process information, the dimension information, the atomic indicator information, and the business limit information, to obtain business indicator information, and to automatically generate a logical model of the business indicator according to the business indicator information;

[0104] The business indicator development module is configured to develop a physical model and data of the business indicator according to the business indicator information and the logical model of the business indicator, and a development script of the physical model and data of the business indicator is automatically generated according to the logical model of the business indicator;

[0105] The business indicator quality inspection module is configured to configure a business indicator quality inspection rule after the development of the business indicator is completed, and to perform quality inspection on the business indicator information according to the business indicator quality inspection rule.

[0106] It should be noted that in the second embodiment, the specific functions implemented by the field and data domain definition module are completely the same as those of step S1 of the first embodiment, the specific functions implemented by the business process definition module are completely the same as those of step S2 of the first embodiment, the specific functions implemented by the business process development module are completely the same as those of step S3 of the first embodiment, the specific functions implemented by the business process quality inspection module are completely the same as those of step S4 of the first embodiment, the specific functions implemented by the dimension definition module are completely the same as those of step S5 of the first embodiment, the specific functions implemented by the dimension development module are completely the same as those of step S6 of the first embodiment, the specific functions implemented by the dimension quality inspection module are completely the same as those of step S7 of the first embodiment, the specific functions implemented by the business indicator definition module are completely the same as those of step S8 of the first embodiment, the specific functions implemented by the business indicator development module are completely the same as those of step S9 of the first embodiment, and the specific functions implemented by the business indicator quality inspection module are completely the same as those of step S10 of the first embodiment. Therefore, the field and data domain definition module, the business process definition module, the business process development module, the business process quality inspection module, the dimension definition module, the dimension development module, the dimension quality inspection module, the business indicator definition module, the business indicator development module, and the business indicator quality inspection module will not be described here, and please refer to the detailed description of the first embodiment.

[0107] The application establishes a new method of big data management based on the dimension modeling theory, and uniformly defines the business data through the field plate, the data domain, the business process, the dimension, the atomic indicator, the business limit, and the business indicator, thereby providing a unified data definition from the business perspective, unifying the calculation range of the business data, and automatically generating the data development script and the quality inspection rule based on the definition information, so that the technical personnel can reach a consistent understanding of the business meaning of the data, and effectively reduce the repeated development of the data, the data quality problems, and the false positives of the data quality, and improve the data management efficiency.

[0108] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, which is described in detail in the third embodiment.

[0109] Embodiment three

[0110] The present embodiment provides an electronic device, as shown in the following formula (I), comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, any of the embodiments of the first embodiment can be implemented. Figure 3

[0111] ​Since the electronic device introduced in the embodiment is the device used for implementing the method in Embodiment One of the present application, the specific implementation of the electronic device of the present embodiment and its various forms can be understood by those skilled in the art based on the method introduced in Embodiment One of the present application, and therefore the implementation of the method in the present embodiment by the electronic device will not be described in detail. As long as the device used for implementing the method in the present embodiment is implemented by those skilled in the art, it belongs to the scope of the present application.

[0112] Based on the same inventive concept, the present application provides a storage medium corresponding to Embodiment One, which is described in detail in Embodiment Four.

[0113] Embodiment Four

[0114] The present embodiment provides a computer-readable storage medium, such as Figure 4 The computer program stored thereon can implement any of the embodiments in Embodiment One when executed by a processor.

[0115] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0116] The present application is described with reference to flowcharts and / or block diagrams of the method, device, and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0117] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product that includes instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0118] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0119] Although the specific embodiments of the present application are described above, it should be understood by those skilled in the art that the specific embodiments described are merely illustrative, and not intended to limit the scope of the present application, and equivalent modifications and variations made in accordance with the spirit of the present application should be covered by the scope of the claims of the present application.

Claims

1. A data governance method, characterized by: The method comprises the following steps: Step S1, defining a field plate and a data field belonging to the field plate, obtaining field plate information and data field information; Step S2, defining a business process belonging to the data field, obtaining business process information, the business process information at least comprising a Chinese name of the business process, an English name of the business process, a description of the business process and a logical model of the business process; Step S3, developing a physical model and data of the business process according to the business process information, the physical model and data development script of the business process being automatically generated through the logical model defined by the business process; Step S4, after the development of the business process, configuring a business process quality inspection rule, and performing quality inspection on the business process information according to the business process quality inspection rule; Step S5, confirming a dimension according to the business process information, defining the confirmed dimension, obtaining dimension information, the dimension information at least comprising a Chinese name of the dimension, an English name of the dimension, a description of the dimension and a logical model of the dimension; Step S6, developing a physical model and data of the dimension according to the dimension information, the physical model and data development script of the dimension being automatically generated through the logical model defined by the dimension; Step S7, after the development of the dimension, configuring a dimension quality inspection rule, and performing quality inspection on the dimension information according to the dimension quality inspection rule; Step S8, defining an atomic index and a business limit according to the business process information, obtaining atomic index information and business limit information; defining a business index according to the business process information, the dimension information, the atomic index information and the business limit information, obtaining business index information, and automatically generating a logical model of the business index according to the business index information; Step S9, developing a physical model and data of the business index according to the business index information and the logical model of the business index, the physical model and data development script of the business index being automatically generated through the logical model of the business index; Step S10, after the development of the business index, configuring a business index quality inspection rule, and performing quality inspection on the business index information according to the business index quality inspection rule.

2. The data governance method of claim 1, wherein: In step S2, the logical model of the business process at least comprises a Chinese name of the logical model, an English name of the logical model, an explanation of the logical model and an attribute field of the logical model; The attribute field at least comprises a Chinese name of the attribute field, an English name of the attribute field, a data type of the attribute field, a data length of the attribute field, a value range of the attribute field, a business scope of the attribute field and a mapping relationship between the attribute field and a data standard; In step S3, the physical model and data development script of the business process are automatically generated through the logical model defined by the business process, specifically comprising: The table name of the physical model of the business process is automatically generated through the English name of the logical model defined by the business process; the field information of the physical model of the business process is automatically generated through the attribute field of the logical model defined by the business process, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model; The data development script of the business process is automatically generated through the business scope of the attribute field defined by the business process.

3. The data governance method of claim 2, wherein: In step S4, the business process quality inspection rule is automatically generated through the mapping relationship between the attribute field defined by the business process and the data standard, specifically including: In the data standard, the name, type, length or value range of the data element of the business process is defined in advance to obtain data element information, and the attribute field defined by the business process is corresponded to the corresponding data element to form a mapping relationship; the name specification quality inspection rule, the data type consistency quality inspection rule, the data length integrity quality inspection rule or the value range accuracy quality inspection rule is automatically generated according to the data element information.

4. The data governance method of claim 1, wherein: In step S5, the logical model of the dimension at least includes the Chinese name of the dimension, the English name of the dimension, the description of the dimension and the attribute field of the dimension; the attribute field at least includes the Chinese name of the attribute field, the English name of the attribute field, the data type of the attribute field, the data length of the attribute field, the value range of the attribute field, the business scope of the attribute field and the mapping relationship between the attribute field and the data standard; In step S6, the physical model of the dimension and the data development script are automatically generated through the logical model defined by the dimension, specifically including: the table name of the physical model of the dimension is automatically generated through the English name of the logical model defined by the dimension; the field information of the physical model of the dimension is automatically generated through the attribute field of the logical model defined by the dimension, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model; The data development script of the dimension is automatically generated through the business scope of the attribute field defined by the dimension.

5. The data governance method of claim 4, wherein: In step S7, the dimension quality inspection rule is automatically generated through the mapping relationship between the attribute field defined by the dimension and the data standard, specifically including: In the data standard, the name, type, length or value range of the data element of the dimension is defined in advance to obtain data element information, and the attribute field defined by the dimension is corresponded to the corresponding data element to form a mapping relationship; the name specification quality inspection rule, the data type consistency quality inspection rule, the data length integrity quality inspection rule or the value range accuracy quality inspection rule is automatically generated according to the data element information.

6. The data governance method of claim 1, wherein: In step S8, the atomic index is a reusable minimum granularity statistical unit set in the business process, the business limit is a reusable business rule condition set in the business process; the business index information includes the Chinese name of the business index, the English name of the business index, the description of the business index, the business process to which the business index belongs, the atomic index used by the business index, the business limit used by the business index, the dimension involved by the business index, the statistical period of the business index or the statistical time of the business index; The logical model of the business index at least includes a Chinese name of the logical model, an English name of the logical model, a description of the logical model, and an attribute field of the logical model; wherein the Chinese name of the logical model is automatically generated by the Chinese name of the business index, the English name of the logical model is automatically generated by the English name of the business index, the description of the logical model is automatically generated by the description of the business index, the attribute field of the logical model is composed of a business process, an atomic index, a business limit, a dimension, a statistical period or a statistical time corresponding to the business index, and at least includes a Chinese name of the attribute field, an English name of the attribute field, a data type of the attribute field, a data length of the attribute field, a value range of the attribute field, a business scope of the attribute field, and a mapping relationship between the attribute field and a data standard.

7. The data governance method of claim 6, wherein: In step S9, the physical model of the business index and the data development script are automatically generated by the logical model of the business index, and specifically include: The table name of the physical model of the business index is automatically generated by the English name of the logical model of the business index, and the field information of the physical model of the business index is automatically generated by the attribute field of the logical model of the business index, wherein the Chinese name of the attribute field corresponds to the field description of the physical model, the English name of the attribute field corresponds to the field name of the physical model, the data type of the attribute field corresponds to the field type of the physical model, and the data length of the attribute field corresponds to the field length of the physical model; The data development script of the business index is automatically generated by the business scope of the attribute field of the business index, and specifically includes: the sql script of the business index is automatically generated by the combination of the business process, the atomic index, the business limit, the dimension, the statistical period and the statistical time; wherein the sql source table of the business index is obtained by the business process, the sql statistical field of the business index is obtained by the atomic index, the sql filtering condition of the business index is obtained by the business limit, the sql grouping condition of the business index is obtained by the dimension, the scheduling period of the business index is obtained by the statistical period, and the sql query time condition of the business index is obtained by the statistical time; In step S10, the business index quality inspection rule is automatically generated by the mapping relationship between the attribute field of the business index and the data standard; wherein the mapping relationship between the attribute field of the business index and the data standard includes the mapping relationship between the attribute field and the data standard in the business process and the mapping relationship between the attribute field and the data standard in the dimension.

8. A data governance apparatus, characterized by: The system comprises a domain and data field definition module, a business process definition module, a business process development module, a business process quality inspection module, a dimension definition module, a dimension development module, a dimension quality inspection module, a business index definition module, a business index development module, and a business index quality inspection module; The domain and data field definition module is configured to define a domain board and a data field belonging to the domain board, and obtain domain board information and data field information. The business process definition module is configured to define a business process belonging to the data field, and obtain business process information, which at least includes a Chinese name of the business process, an English name of the business process, a description of the business process, and a logical model of the business process. The business process development module is configured to develop a physical model and data of the business process according to the business process information, and a development script of the physical model and data of the business process is automatically generated according to a logical model of the business process definition; The business process quality inspection module is configured to configure a business process quality inspection rule after the development of the business process, and to perform quality inspection on the business process information according to the business process quality inspection rule; The dimension definition module is configured to confirm dimensions according to the business process information, to define the confirmed dimensions, and to obtain dimension information, which at least includes a Chinese name of the dimension, an English name of the dimension, a description of the dimension, and a logical model of the dimension; The dimension development module is configured to develop a physical model and data of the dimension according to the dimension information, and a development script of the physical model and data of the dimension is automatically generated according to a logical model of the dimension definition; The dimension quality inspection module is configured to configure a dimension quality inspection rule after the development of the dimension, and to perform quality inspection on the dimension information according to the dimension quality inspection rule; The business indicator definition module is configured to define an atomic indicator and a business limit according to the business process information, and to obtain atomic indicator information and business limit information; A business indicator is defined according to the business process information, the dimension information, the atomic indicator information, and the business limit information, and business indicator information is obtained, and a logical model of the business indicator is automatically generated according to the business indicator information; The business indicator development module is configured to develop a physical model and data of the business indicator according to the business indicator information and the logical model of the business indicator, and a development script of the physical model and data of the business indicator is automatically generated according to the logical model of the business indicator; The business indicator quality inspection module is configured to configure a business indicator quality inspection rule after the development of the business indicator, and to perform quality inspection on the business indicator information according to the business indicator quality inspection rule.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 7.

Images