A data management system and method

Abstract

This invention provides a data management system and method, belonging to the field of data management. The system includes: a data source management module, a database management module, a data model management module, a data standard management module, and a data quality management module. The data source management module is used to obtain multiple initial metadata and data types from multiple preset business systems, and construct a database to be processed using these initial metadata and data types. The database management module is used to update the database to obtain an updated database. The data model management module is used to construct a target model based on the updated database. This invention solves the problems of low online data rate, low structured data rate, and multiple data sources caused by the increasing amount of data in business systems. It establishes a unified data asset management standard, laying a solid data management foundation for building intelligent pipelines.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically to a data management system and method. Background Technology

[0002] Currently, as the amount of data accumulated in business systems grows, many companies have not yet established unified data asset management standards for data collection, unified data storage, data application, and data management. This results in problems such as low online data rates, low data structuring rates, and multiple data sources, failing to meet the requirements for building digital twins of pipelines, intelligent pipelines, and smart pipe networks. Therefore, establishing a unified data standard system to achieve unified data sources and solve data silos has become a crucial step in the transformation from traditional pipeline management to intelligent management. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data management system and method that addresses the shortcomings of the prior art.

[0004] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A data management system, comprising: a data source management module, a database management module, a data model management module, a data standard management module, and a data quality management module.

[0005] The data source management module is used to obtain multiple initial metadata and the data type corresponding to each of the multiple preset business systems from multiple preset business systems, and to construct the database to be processed for each preset business system through the multiple initial metadata and the data type corresponding to each initial metadata.

[0006] The database management module is used to update each of the databases to be processed to obtain the updated databases of each of the preset business systems.

[0007] The data model management module is used to construct target models for each of the preset business systems based on each of the updated databases.

[0008] The data standard management module is used to import standard management information and update the standard management information to obtain updated standard management information.

[0009] The data quality management module is used to check and analyze all target models based on the updated standard management information and obtain a data quality check result report.

[0010] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A data management method, comprising the following steps:

[0011] Obtain multiple initial metadata and the data types corresponding to each of the multiple preset business systems;

[0012] The database to be processed for each of the preset business systems is constructed by using multiple initial metadata of each preset business system and the data types corresponding to each initial metadata.

[0013] Each of the databases to be processed is updated to obtain the updated databases of each of the preset business systems;

[0014] Each target model of the preset business system is constructed based on the updated database.

[0015] Import standard management information and update the standard management information to obtain updated standard management information;

[0016] Based on the updated standard management information, all target models are checked and analyzed to obtain a data quality check result report.

[0017] The beneficial effects of this invention are as follows: a database to be processed is constructed by pre-setting multiple initial metadata and data types of the business system; the database to be processed is updated to obtain an updated database; a target model is constructed based on the updated database; the information of the standard management information is updated to obtain updated standard management information; and a data quality inspection result report is obtained by checking and analyzing all target models based on the updated standard management information. This solves the problems of low online data rate, low structured data rate, and multiple data sources caused by the increasing amount of data in the business system. It establishes a unified data asset management standard and lays a good data management foundation for building a smart pipeline. Attached Figure Description

[0018] Figure 1 This is a block diagram of a data management system provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating a data management method provided in an embodiment of the present invention. Detailed Implementation

[0020] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0021] Figure 1 This is a block diagram of a data management system provided in an embodiment of the present invention.

[0022] like Figure 1As shown, a data management system includes: a data source management module, a database management module, a data model management module, a data standard management module, and a data quality management module.

[0023] The data source management module is used to obtain multiple initial metadata and the data type corresponding to each of the multiple preset business systems from multiple preset business systems, and to construct the database to be processed for each preset business system through the multiple initial metadata and the data type corresponding to each initial metadata.

[0024] The database management module is used to update each of the databases to be processed to obtain the updated databases of each of the preset business systems.

[0025] The data model management module is used to construct target models for each of the preset business systems based on each of the updated databases.

[0026] The data standard management module is used to import standard management information and update the standard management information to obtain updated standard management information.

[0027] The data quality management module is used to check and analyze all target models based on the updated standard management information and obtain a data quality check result report.

[0028] Preferably, the preset business system can be an operation and management system, a production and operation system, a comprehensive support system, or other information systems.

[0029] In the above embodiments, a database to be processed is constructed by pre-setting multiple initial metadata and data types of the business system. The database to be processed is updated to obtain an updated database. A target model is constructed based on the updated database. The information of the standard management information is updated to obtain updated standard management information. The data quality inspection result report is obtained by checking and analyzing all target models based on the updated standard management information. This solves the problems of low data online rate, low structured rate and multiple data sources caused by the increasing amount of data in the business system. It establishes a unified data asset management standard and lays a good data management foundation for building a smart pipeline.

[0030] Optionally, as an embodiment of the present invention, the data source management module includes a data source acquisition unit, a database discrimination unit, a first data conversion unit, a second data conversion unit, a third data conversion unit, and a database integration unit.

[0031] The data source acquisition unit is used to obtain multiple initial metadata of each of the multiple preset business systems and the data type corresponding to each of the initial metadata from multiple preset business systems;

[0032] The database discrimination unit is used to calculate the data capacity of multiple initial metadata of each preset business system, obtain the data capacity of each preset business system, and determine whether each preset business system meets the following conditions: the data type corresponding to all initial metadata in the preset business system is any data type among multiple preset data types, and the data capacity of the preset business system is greater than or equal to the preset data capacity; if yes, a first database is created, and all initial metadata of the preset business systems that meet the conditions is stored in the first database; if no, a second database is created, and all initial metadata of the preset business systems that do not meet the conditions is stored in the second database.

[0033] The first data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a first preset discrimination data type into a first converted metadata, so as to update the first database or the second database.

[0034] The second data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a second preset discrimination data type into a second converted metadata, so as to update the first database or the second database.

[0035] The third data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a third preset discrimination data type into a third converted metadata, so as to update the first database or the second database.

[0036] The database integration unit is used to take the first database or the second database updated by the first data conversion unit, the second data conversion unit and the third data conversion unit of each preset business system as the database to be processed for each preset business system.

[0037] Preferably, the first database can be a relational database, the second database can be a non-relational database, the first preset data type can be a text file and / or an XML file, the second preset data type can be an EXECL file, and the third preset data type can be a PDM file and an HML file.

[0038] It should be understood that data source management is provided, allowing for the definition and collection of data sources for the data to be managed based on business systems (i.e., the preset business systems), management departments, etc. This mainly includes data source support, data source exploration, and physical data modeling.

[0039] Specifically, the data source management module supports data access from relational databases such as Oracle, MySQL, and SQL Server, as well as non-relational databases such as HBase and MongoDB, based on database acquisition adapter technology. It also supports importing data from PDM model relationships.

[0040] Specifically, the text file and XML file (i.e., the first preset discriminant data type) filled according to the template are imported through the interface, and the file (i.e., the first preset discriminant data type) is parsed according to the parsing strategy preset by the data source management module, so that the interface file is serialized.

[0041] It should be understood that the data source management module has a pre-set parsing strategy to parse the imported EXECL file (i.e., the second preset data type) according to the template, thereby serializing the interface file. The EXCEL acquisition adapter only needs to support the specified template. If it is found that the existing template cannot meet the business requirements or the format is incompatible, it can provide parsing and acquisition of EXCEL files by specifying a new template through the data source management module (i.e., the second data conversion unit).

[0042] Specifically, it can use existing model parsing technology to parse the collected and stored data model's PDM file and XML file (i.e., the third preset discrimination data type). This achieves the ability to parse entities, entity attributes, and relationships between entities in the data model.

[0043] In the above embodiments, a database to be processed is constructed by pre-setting multiple initial metadata and data types of the business system, which can support the import of data from the PDM model relationship and unify the data format, thereby realizing the management of the data source.

[0044] Optionally, as an embodiment of the present invention, the metadata to be processed in the database to be processed includes multiple attribute information, and the metadata to be processed is any one of the initial metadata, the first converted metadata, the second converted metadata, and the third converted metadata; the database management module includes an attribute management unit and a database association unit;

[0045] The attribute management unit is used to import multiple department terminals that correspond one-to-one with the multiple attribute information, and send each of the attribute information to the corresponding department terminal, and obtain updated attribute information from the corresponding department terminal, update the corresponding metadata to be processed according to the multiple updated attribute information corresponding to each of the metadata to be processed, to obtain the updated metadata corresponding to each of the metadata to be processed, and update the corresponding database to be processed according to the multiple updated metadata corresponding to each database to be processed, to obtain the updated database to be processed;

[0046] The database association unit is used to associate each updated database to be processed with at least one of the remaining updated databases to be processed, to obtain the associated databases of each of the preset business systems, and to use the associated databases as the updated databases.

[0047] The database management module also includes any or a combination of multiple units from the catalog generation unit, data view generation unit, business system management unit, and asset map generation unit.

[0048] The directory generation unit is used to generate a data directory table based on all the updated attribute information of all department terminals and all the databases to be processed.

[0049] The data view generation unit is used to perform visualization processing on each of the updated metadata, obtain visualized metadata of each of the updated metadata, and access and / or query each of the visualized metadata.

[0050] The business system management unit is used to add and / or modify and / or delete each of the preset business systems;

[0051] The asset mapping generation unit is used to draw maps of all updated databases to obtain asset maps.

[0052] It should be understood that the remaining updated pending database can be understood as all updated pending databases except for the updated pending database of the currently described preset business system.

[0053] It should be understood that after each attribute information is sent to the corresponding department terminal, the department terminal adds or modifies the corresponding attribute information to obtain the updated attribute information.

[0054] It should be understood that data asset management focuses on building a data asset management system. Through data asset management, data standardization management and data processing can be organically integrated to form data standards, realize metadata description of specific resource data, support the use of standardized data interfaces and rich chart display tools to quickly customize various data asset applications, and, together with the comprehensive evaluation of data assets, gradually realize the standardization capability of data assets and the improvement of the openness and application capability of data assets.

[0055] Specifically, data asset management based on metadata-driven comprehensive profiling enables full lifecycle management and monitoring of data, full-process record-keeping for traceability, panoramic asset visualization, and provides a full-scenario view of data assets to meet the needs of different user application scenarios. It caters to managers with overall planning, users focused on detailed definition, and developers involved in processing and maintenance, providing multi-level graphical displays to meet the graphical query and auxiliary analysis needs of application scenarios.

[0056] A big data asset management system is built based on data catalog technology. This system enables the management of data assets, achieving a "logically unified, physically decentralized" data management approach adapted to the current state of the field. Data entering the data resource pool is classified and its dimensional attributes are labeled, ensuring data traceability and facilitating use and statistical analysis. The metadata attributes managed by the data catalog can include nine categories, as shown in Table 1. Table 1 shows the metadata attribute classification.

[0057] Table 1:

[0058]

[0059] It should be understood that the attribute management unit configures the affiliation between departments (i.e., the department terminals) and their managed business systems based on the departmental affiliation relationships configured in departmental management. This reflects the departmental management hierarchy and the business systems managed by the department. Furthermore, it allows for the addition and modification of business systems and their subordinate departments (i.e., the department terminals). After departmental configuration, unified management of departmental business matters is achieved through organizational business list management. This mainly includes the business responsibilities, service recipients, required materials, and supporting information systems for each business matter. Exporting business matters is supported in standard Excel format. Information list management enables unified management of information resource needs across departments, primarily including information resource name, source department, acquisition method, and data update frequency.

[0060] It should be understood that the directory generation unit can configure the system information to which a department (i.e., the department terminal) belongs, reflecting the current management status of the system by the department (i.e., the department terminal). A brief description of the relevant system functions is provided.

[0061] It should be understood that the data view generation unit mainly provides access to view metadata information such as tables and fields in the business system, and provides data sampling queries to facilitate user understanding of the data.

[0062] Specifically, the business system management unit can be understood as adding, modifying, and deleting relevant business domain information on the platform based on the business domain definition, and providing the results to other modules.

[0063] Specifically, the database association unit provides a lineage analysis function to enable data tracing and source tracking. By selecting a specified table (i.e., the updated database to be processed) and performing data tracing and source tracking operations on the specified table (i.e., the updated database to be processed), the analysis of the data source can be achieved.

[0064] Specifically, the asset map generation unit provides panoramic management based on the "application scenario" dimension. It is based on a full-scenario view of metadata assets from various business systems. From the perspective of application scenarios, it includes managers who plan globally, users who focus on detailed definitions, and developers who process and maintain the data. It provides multi-level graphical displays to meet the graphical query and auxiliary analysis needs of application scenarios.

[0065] It should be understood that the asset map allows viewing of collected metadata from different perspectives (system dimension, subject domain dimension, business tag), supporting the display of metadata and relationships, and then expanding, drilling down, or tracing back to each level of metadata and relationships. For example, from pipeline integrity management to line integrity to PIS system to cathodic protection cables to cable corrosion protection material entity attributes, metadata can be displayed for item-by-item searching, supporting advanced search and intelligent recommendation of results.

[0066] In the above embodiments, the database update of the database to be processed obtains the updated database, realizes the metadata description of specific resource data, can quickly customize various data asset applications, and, together with the comprehensive evaluation of data assets, gradually realizes the standardization capability of data assets and the improvement of the openness and application capability of data assets. At the same time, it also meets the needs of different users' application scenarios and the graphical query and auxiliary analysis of application scenarios, and provides multi-level graphical display.

[0067] Optionally, as an embodiment of the present invention, the data model management module includes a logical model management unit, a physical model management unit, and a mapping relationship management unit.

[0068] The logical model management unit is used to construct the original logical model of each preset business system through each updated database, and modify each original logical model to obtain the target logical model of each preset business system.

[0069] The physical model management unit is used to construct the original physical model of each preset business system through each updated database, and modify each original physical model to obtain the target physical model of each preset business system.

[0070] The mapping relationship management unit is used to perform mapping processing on the target logical model and the target physical model of each of the preset business systems to obtain the target model of each of the preset business systems.

[0071] It should be understood that the modifications mentioned can all be the creation, editing, deletion, etc. of subject areas.

[0072] It should be understood that metadata management enables the definition and storage of metadata models, which are then packaged into various metadata functions at the functional layer and ultimately provided for external application and display; metadata classification and modeling, lineage and impact analysis are provided to facilitate data tracking and retrospection.

[0073] Specifically, a data model is an abstraction of the characteristics of real-world data, used to describe the concepts and definitions of a set of data. A data model is the way data is stored in a database and is the foundation of a database system. In a database, the physical structure of data, also known as the data storage structure, is the representation and configuration of data elements in computer memory; the logical structure of data refers to the logical relationships between data elements. The mapping relationship between the logical model and the physical model facilitates the retrieval of a data attribute from the conceptual model. Information from multiple physical models can also be used to determine data fusion rules for the data warehouse by combining data quality and data time points generated by business processes.

[0074] It should be understood that the logical model management unit provides data modeling capabilities, visually defines data models, defines relationships between models, and supports the design of tables, fields, relationships, views, indexes, and partitions. It provides model category management, including the creation, editing, and deletion of subject areas. It supports visual ER diagrams to display data models. It supports importing Excel files to define logical models (i.e., the target logical model).

[0075] Specifically, the physical model management unit provides data modeling capabilities, allowing for the visual definition of data models and the definition of relationships between models. It supports the design of tables, fields, relationships, views, indexes, and partitions. It offers model category management, including the creation, editing, and deletion of subject areas. It supports visual ER diagrams to display data models. It also supports reverse data modeling. For created data models, they can be materialized into corresponding table structures based on the selected database type.

[0076] It should be understood that the mapping relationship management unit provides a mapping relationship definition between the physical model (i.e., the target physical model) and the logical model (i.e., the target logical model), so as to reflect the mapping relationship (field level) between the business system and the logical model evolved from the standard, and provides corresponding data interfaces for the data standard module (i.e., the data standard management module) and the data quality management module, so that the data standard can generate data quality detection rules based on the mapping relationship.

[0077] In the above embodiments, the target model is constructed based on the updated database, which facilitates data tracking and backtracking, and enables data standards to generate data quality detection rules based on the mapping relationship.

[0078] Optionally, as an embodiment of the present invention, the standard management information includes original standard system information, original standard document information, and original standard specification information; the data standard management module includes a standard specification management unit, a standard system management unit, and a standard document management unit.

[0079] The standard system management unit is used to import the original standard system information and add and / or modify and / or delete the original standard system information to obtain the updated standard system information;

[0080] The standard file management unit is used to import the original standard file information and add and / or modify and / or delete the original standard file information to obtain updated standard file information;

[0081] The standard specification management unit is used to import the original standard specification information, and to add and / or modify and / or delete the original standard specification information to obtain updated standard specification information. The updated standard system information, the updated standard document information, and the updated standard specification information are all used as updated standard management information.

[0082] It should be understood that the standard system management unit is used to add corresponding standard system information under the first-level business domain directory, so that users can formulate different standard systems according to business domain classification, search for the standard name according to the business domain to which the standard belongs, and realize the addition, modification and deletion operations of the standard system (i.e. the original standard system information).

[0083] Specifically, the standard file management unit is used to add corresponding standard file information in the standard system directory of the first-level business domain, so that users can formulate different standard files according to the standard system classification, search for the standard name according to the business domain to which the standard belongs, and realize the import, addition, modification and deletion operations of standard files (i.e. the original standard file information).

[0084] It should be understood that the standard specification management unit is used to add corresponding standard specification information under standard files, so that users can formulate different standard specifications according to the standard file classification, search for the standard name according to the business domain to which the standard belongs, and realize the import, addition, modification and deletion operations of standard specifications (i.e. the original standard specification information).

[0085] In the above embodiments, updating the standard management information to obtain updated standard management information facilitates the customization of standard management information settings.

[0086] Optionally, as an embodiment of the present invention, the data quality management module includes a quality rule management unit, a quality inspection unit, a data modification unit, and a report generation unit.

[0087] The quality rule management unit is used to view the updated standard management information and establish quality inspection rules based on the updated standard specification information.

[0088] The quality inspection unit is used to perform quality inspections on each of the target models according to the quality inspection rules, and to obtain the initial inspection results of each of the preset business systems and the initial inspection parameter information of each of the preset business systems.

[0089] The data modification unit is used to modify the data of each target model to obtain the modified target model of each preset business system, and to perform quality checks on each modified target model again according to the quality check rules to obtain the target check results of each preset business system.

[0090] The report generation unit is used to generate a quality inspection result report based on all initial metadata, the initial inspection results, the modified target model, and the target inspection results.

[0091] The data quality management module also includes any one or a combination of multiple units from the inspection parameter management unit and the inspection result sending unit;

[0092] The inspection parameter management unit is used to send all initial inspection parameter information to a preset server;

[0093] The inspection result sending unit is used to send all initial inspection results to the designated terminal.

[0094] It should be understood that the data quality management module implements quality management throughout the entire data lifecycle. It can visually configure data quality inspection strategies according to standard rules, conduct data quality checks through a scheduling center, identify problematic data, and assign the problematic data to relevant personnel for correction according to the owner's system. It can also generate data quality assessment reports and problem handling reports as needed.

[0095] Specifically, to mitigate the impact on information system databases, data quality checks employ data flow inspection technology. The data quality inspection methods and calculations run within the engine rather than relying on database SQL. It not only performs preliminary database and table probing and checks, and data quality checks during data processing, but also provides post-processing, refined data quality checks. Furthermore, it can filter, investigate, and track problematic data, providing data quality reports, analyzing problematic data, and handling anomalies.

[0096] It should be understood that the quality rule management unit is used to view the corresponding standard value range and quality rule information under the standard specification (i.e., the updated standard management information), so that users can formulate different standard value ranges and rule information under the standard specification (i.e., the updated standard management information).

[0097] Specifically, the quality inspection unit can provide data quality accuracy checks, facilitating refined data quality analysis of a given table. It provides data quality inspection services to perform specified rule checks on database tables (i.e., the target model), logical expression checks, composite checks, a visual definition interface, and a data quality inspection method interface, making it easy to add data quality inspection methods.

[0098] Specifically, the quality inspection unit performs specified rule checks on the database table (i.e., the target model), including format checks, range checks, missing record checks, precision checks, logical expression checks, and compound rule checks. The data quality inspection service can be configured visually to perform single-field multi-rule checks, multi-field rule checks, and correlation checks between multiple fields.

[0099] The format checking rules include time format checking, number checking, ID card checking, regular expression checking, etc., and different input interfaces are provided according to the characteristics of different rules.

[0100] Range checking rules, including checking not in the table and not in a custom range, are provided with a visual definition interface.

[0101] Logical expression checks include logical checks and string logical checks. Logical checks define logical comparison rules between one or more fields, including equal to, not equal to, less than, less than or equal to, greater than, greater than or equal to, greater than and less than, greater than and less than, greater than and less than, greater than and less than equal to, greater than and less than equal to, true, false, etc. The comparison values ​​for logical checks can come from different fields in the check table. String logical checks include equal to, in a list, contain, start as, end as, string length comparison, etc. A visual definition interface is provided.

[0102] Composite quality inspection refers to the combination of AND and OR rules for multiple data quality inspections, and can be configured visually.

[0103] The data quality inspection rules are open, and an interface for data quality inspection methods is provided to facilitate the addition of data quality inspection features via Java extensions.

[0104] It should be understood that the inspection parameter management unit is used to configure the inspection model defined by the quality inspection service, including relevant information such as execution time, frequency, and execution node (i.e., the initial inspection parameter information). It also generates the inspection service and sends it to the corresponding node server (i.e., the preset server).

[0105] It should be understood that the inspection result sending unit is used to assign the detected problem data to the corresponding owner business users for problem data query, modification and other operations.

[0106] Specifically, the data modification unit can perform quality checks on the processed data, such as multiplying the amount field value by parameters before checking, and modify and improve problematic data. For each service, after each run, the unit can specifically examine the problem data and make modifications. After modification, the data is sent back to the source to achieve the goal of improving the problematic data. Simultaneously, after receiving the data quality issue assignment, the business system administrator (i.e., the designated terminal) rectifys the data problems in the business system according to the data quality report error information and verification rules. After rectification, the system provides feedback on the data quality rectification status.

[0107] It should be understood that the report generation unit provides a data quality inspection result report (i.e., the quality inspection result report) for each data quality inspection service, including abnormal data, a description of the rules for abnormal data inspection, and the ability to perform problem data statistics, modification statistics, and inspection rule statistics. It is also required to provide a display of problem data and the causes of the problems, as well as to provide assignment for modification, a modification interface, and analysis of problem data change trends.

[0108] In the above embodiments, the data quality inspection result report is obtained by checking and analyzing all target models through updated standard management information, realizing the quality management of the entire data lifecycle. The data quality inspection strategy can be visualized and configured according to standard rules, which solves the problems of low data online rate, low structured rate and multiple data sources caused by the accumulation of data in business systems.

[0109] Optionally, as an embodiment of the present invention, the data management system further includes a data acquisition task scheduling unit, the data acquisition task scheduling unit being used for:

[0110] Import scheduling task instructions, obtain the scheduling task start time according to the scheduling task instructions, and obtain the scheduling status of each of the initial metadata from multiple preset business systems;

[0111] Each time, determine whether the scheduling status is a successful scheduling status. If it is, store the scheduling status of the initial data. If not, generate a scheduling failure reason based on the scheduling status and store the scheduling failure reason.

[0112] Import the scheduling task stop instruction, obtain the scheduling task shutdown time based on the scheduling task stop instruction, and calculate the difference between the scheduling task start time and the scheduling task shutdown time to obtain the scheduling time.

[0113] It should be understood that the data source management module (i.e., the data acquisition task scheduling unit) is used to configure work tasks, realize timed scheduling of work tasks, and record the operation status of work tasks. The operation status includes information such as scheduling success, scheduling failure, failure reason, and scheduling time. The failure reason is given by the program.

[0114] In the above embodiments, the start time and status of the scheduling task are obtained through the scheduling task instruction, and it is determined whether the scheduling status is a successful scheduling status. The shutdown time of the scheduling task is obtained through the scheduling task stop instruction, and the scheduling time is calculated based on the difference between the shutdown time and the start time of the scheduling task. This realizes the timed scheduling of the task and can record the running status of the task.

[0115] Optionally, as an embodiment of the present invention, the data management system further includes a system management module, which includes any one or more of the following units: a department terminal management unit, a user management unit, a role management unit, a menu management unit, a permission management unit, and an attribute information management unit.

[0116] The department terminal management unit is used to add new department terminals, modify terminal information, and modify their affiliation.

[0117] The user management unit is used to create users and add, modify, and delete users' login names, passwords, departments, job responsibilities, and user groups.

[0118] The role management unit is used to create multiple roles and assign at least one of the roles to the user;

[0119] The menu management unit is used to create menus and to add, modify, and delete menus.

[0120] The permission management unit is used to create, modify, and delete permissions, and to assign the permissions to the user;

[0121] The attribute information management unit is used to add, modify, and delete attribute information.

[0122] It should be understood that the department terminal management unit is configured based on the hierarchical relationship of departments. After the departments (i.e., the department terminals) are configured, they are displayed in a tree structure. After the departments (i.e., the department terminals) are adjusted, it is very convenient to add, modify, and modify the affiliation of departments (i.e., the department terminals).

[0123] It should be understood that the user management unit is used to configure and manage information such as the user's login name, department, job responsibilities, and user group.

[0124] Specifically, a role is a set of permissions, and a user group refers to a group of users with the same characteristics. A user group can be assigned multiple roles, and users belonging to that user group inherit the roles possessed by that user group. Roles or permissions can also be assigned to individual users. A built-in system super administrator account is included. After successful login, this account has full system permissions without requiring permission checks. This account is for backup, and its password can be modified, and the account can be enabled / disabled. The system also includes a built-in super administrator user group; users belonging to this group have full system permissions without requiring permission checks.

[0125] It should be understood that the menu management unit implements the functions of adding, modifying, and deleting menu items.

[0126] Specifically, the permissions are divided into data permissions and functional permissions. Data permissions refer to the scope of data that a user can view, including: metadata viewing permissions, data model viewing permissions, and data asset viewing permissions. Functional permissions are divided into menu permissions and operation permissions. Menu permissions refer to a permission corresponding to each menu item, while operation permissions refer to each sub-item on the menu page, such as "Create". Functional permissions need to be granularized down to a specific operation, such as adding a user or deleting an organization. The granularity of permission settings should conform to business needs and operational habits, and should not be too cumbersome or too simple.

[0127] It should be understood that the attribute information management unit adds, modifies, and deletes data attribute definitions used by the system for maintaining menu items and drop-down lists in the system.

[0128] The above embodiments enable customized management of the system, increasing the flexibility of system management.

[0129] Optionally, as an embodiment of the present invention, the data management system further includes a comprehensive management module, which includes any or a combination of multiple units such as an authentication unit and a data retrieval unit.

[0130] The authentication unit is used to authenticate users using the login name and password, and to store the IP address, login time, and cumulative login count of successfully authenticated users.

[0131] The data retrieval unit is used to retrieve all initial metadata, the attribute information, the newly added and / or modified attribute information, and the updated metadata.

[0132] It should be understood that the data retrieval unit, including the data display page, must provide data retrieval functionality. This facilitates quick retrieval of the required data and allows users to quickly obtain relevant information.

[0133] Specifically, this system requires login. Users log in using their username and password through the authentication unit. To ensure password security, the system encrypts the password with a random number using base64 encoding and records the IP address, login time, and cumulative login count of successfully logged-in users. Upon successful login, the system reads user information (user ID, username, company name, etc.), organization information (organization ID, organization number, organization name, parent organization ID, organization number, organization name, etc.), user group information (user group ID, user group code, user group name, etc.), and permission information (organization permissions, user group permissions, user permissions, user permissions, etc.). This information is stored in the SESSION file, and all SESSION information is destroyed upon user logout.

[0134] The above embodiments can ensure the information security of users using this system, and at the same time, facilitate the quick retrieval of the required data content, making it easy for users to quickly obtain relevant information.

[0135] Optionally, as an embodiment of the present invention, the system management module further includes a node management unit, which is specifically used for:

[0136] When providing a multi-server environment, the system allocates the server nodes where system tasks are executed, achieving high availability and avoiding system bottlenecks and performance degradation caused by resource contention.

[0137] Optionally, as an embodiment of the present invention, the integrated management module further includes any one or a combination of multiple units selected from the following: a notification and announcement display unit, an approval and pending task release unit, a task statistics unit, and an operation monitoring unit.

[0138] The notification and announcement display unit is used to display notification information pages;

[0139] The approval pending release unit is used for the approval process of data view release in this system;

[0140] The task statistics unit is used to provide statistics on data inspection tasks, data quality problem handling, etc., through reports.

[0141] The operation monitoring unit is a page used to view the running status of scheduled tasks in this system, allowing system administrators to understand the status of currently executing and historical scheduled tasks.

[0142] Optionally, as another embodiment of the present invention, the present invention addresses the problem that as the amount of data in business systems accumulates, the company has not yet established a unified data asset management standard in terms of data collection, unified data storage, data application, and data management, resulting in problems such as low data online rate, low structuring rate, and multiple data sources. The present invention realizes the implementation of "data standard system" and "data asset map" on the data asset management platform, laying a good data management foundation for the company to build a smart pipeline.

[0143] Optionally, as another embodiment of the present invention, based on the platform construction goals, the platform needs to have the ability to collect metadata and check data quality. It also needs to reflect the standards in the standard system within the platform and establish a correlation between the standards and the logical and physical models.

[0144] Figure 2 This is a flowchart illustrating a data management method provided in an embodiment of the present invention.

[0145] Alternatively, as another embodiment of the present invention, such as Figure 2 As shown, a data management method includes the following steps:

[0146] Obtain multiple initial metadata and the data types corresponding to each of the multiple preset business systems;

[0147] The database to be processed for each of the preset business systems is constructed by using multiple initial metadata of each preset business system and the data types corresponding to each initial metadata.

[0148] Each of the databases to be processed is updated to obtain the updated databases of each of the preset business systems;

[0149] Each target model of the preset business system is constructed based on the updated database.

[0150] Import standard management information and update the standard management information to obtain updated standard management information;

[0151] Based on the updated standard management information, all target models are checked and analyzed to obtain a data quality check result report.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0155] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This is understood to mean that the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data management system, characterized in that, include: The module includes a data source management module, a database management module, a data model management module, a data standard management module, and a data quality management module. The data source management module is used to obtain multiple initial metadata and the data type corresponding to each of the multiple preset business systems from multiple preset business systems, and to construct the database to be processed for each preset business system through the multiple initial metadata and the data type corresponding to each initial metadata. The database management module is used to update each of the databases to be processed to obtain the updated databases of each of the preset business systems. The data model management module is used to construct target models for each of the preset business systems based on each of the updated databases. The data standard management module is used to import standard management information and update the standard management information to obtain updated standard management information. The data quality management module is used to check and analyze all target models according to the updated standard management information and obtain a data quality check result report; The data source management module includes a data source acquisition unit, a database discrimination unit, a first data conversion unit, a second data conversion unit, a third data conversion unit, and a database integration unit. The data source acquisition unit is used to obtain multiple initial metadata of each of the multiple preset business systems and the data type corresponding to each of the initial metadata from multiple preset business systems; The database discrimination unit is used to calculate the data capacity of multiple initial metadata of each preset business system, obtain the data capacity of each preset business system, and determine whether each preset business system meets the condition that the data type corresponding to all the initial metadata in the preset business system is any data type among multiple preset data types, and the data capacity of the preset business system is greater than or equal to the preset data capacity. If so, a first database is created, and all the initial metadata in the preset business system that meets the conditions is stored in the first database; If not, a second database is created, and all the initial metadata in the preset business system that does not meet the conditions is stored in the second database. The first database is a relational database, and the second database is a non-relational database.

2. The data management system according to claim 1, characterized in that, The first data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a first preset discrimination data type into a first converted metadata, so as to update the first database or the second database. The second data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a second preset discrimination data type into a second converted metadata, so as to update the first database or the second database. The third data conversion unit is used to convert the initial metadata in the first database or the second database whose data type is a third preset discrimination data type into a third converted metadata, so as to update the first database or the second database. The database integration unit is used to take the first database or the second database updated by the first data conversion unit, the second data conversion unit and the third data conversion unit of each preset business system as the database to be processed for each preset business system.

3. The data management system according to claim 2, characterized in that, The metadata to be processed in the database includes multiple attribute information, and the metadata to be processed is any one of the initial metadata, the first converted metadata, the second converted metadata, and the third converted metadata; the database management module includes an attribute management unit and a database association unit; The attribute management unit is used to import multiple department terminals that correspond one-to-one with the multiple attribute information, and send each of the attribute information to the corresponding department terminal, and obtain updated attribute information from the corresponding department terminal, update the corresponding metadata to be processed according to the multiple updated attribute information corresponding to each of the metadata to be processed, to obtain the updated metadata corresponding to each of the metadata to be processed, and update the corresponding database to be processed according to the multiple updated metadata corresponding to each database to be processed, to obtain the updated database to be processed; The database association unit is used to associate each updated database to be processed with at least one of the remaining updated databases to be processed, to obtain the associated databases of each of the preset business systems, and to use the associated databases as the updated databases. The database management module also includes any one or a combination of multiple units from the following: a catalog generation unit, a data view generation unit, a business system management unit, and an asset map generation unit. The directory generation unit is used to generate a data directory table based on all the updated attribute information of all department terminals and all the databases to be processed. The data view generation unit is used to perform visualization processing on each of the updated metadata, obtain visualized metadata of each of the updated metadata, and access and / or query each of the visualized metadata. The business system management unit is used to add and / or modify and / or delete each of the preset business systems; The asset mapping generation unit is used to draw maps of all updated databases to obtain asset maps.

4. The data management system according to claim 1, characterized in that, The data model management module includes a logical model management unit, a physical model management unit, and a mapping relationship management unit. The logical model management unit is used to construct the original logical model of each preset business system through each updated database, and modify each original logical model to obtain the target logical model of each preset business system. The physical model management unit is used to construct the original physical model of each preset business system through each updated database, and modify each original physical model to obtain the target physical model of each preset business system. The mapping relationship management unit is used to perform mapping processing on the target logical model and the target physical model of each of the preset business systems to obtain the target model of each of the preset business systems.

5. The data management system according to claim 1, characterized in that, The standard management information includes original standard system information, original standard document information, and original standard specification information. The data standard management module includes a standard specification management unit, a standard system management unit, and a standard document management unit. The standard system management unit is used to import the original standard system information and add and / or modify and / or delete the original standard system information to obtain the updated standard system information; The standard file management unit is used to import the original standard file information and add and / or modify and / or delete the original standard file information to obtain updated standard file information; The standard specification management unit is used to import the original standard specification information, and to add and / or modify and / or delete the original standard specification information to obtain updated standard specification information. The updated standard system information, the updated standard document information, and the updated standard specification information are all used as updated standard management information.

6. The data management system according to claim 1, characterized in that, The data quality management module includes a quality rule management unit, a quality inspection unit, a data modification unit, and a report generation unit. The quality rule management unit is used to view the updated standard management information and establish quality inspection rules based on the updated standard specification information. The quality inspection unit is used to perform quality inspections on each of the target models according to the quality inspection rules, and to obtain the initial inspection results of each of the preset business systems and the initial inspection parameter information of each of the preset business systems. The data modification unit is used to modify the data of each target model to obtain the modified target model of each preset business system, and to perform quality checks on each modified target model again according to the quality check rules to obtain the target check results of each preset business system. The report generation unit is used to generate a quality inspection result report based on all initial metadata, the initial inspection results, the modified target model, and the target inspection results. The data quality management module also includes any one or a combination of multiple units from the inspection parameter management unit and the inspection result sending unit; The inspection parameter management unit is used to send all initial inspection parameter information to a preset server; The inspection result sending unit is used to send all initial inspection results to the designated terminal.

7. The data management system according to claim 1, characterized in that, The data management system further includes a data acquisition task scheduling unit, which is used for: Import scheduling task instructions, obtain the scheduling task start time according to the scheduling task instructions, and obtain the scheduling status of each of the initial metadata from multiple preset business systems; Each time, determine whether the scheduling status is a successful scheduling status. If it is, store the scheduling status of the initial data. If not, generate a scheduling failure reason based on the scheduling status and store the scheduling failure reason. Import the scheduling task stop instruction, obtain the scheduling task shutdown time based on the scheduling task stop instruction, and calculate the difference between the scheduling task start time and the scheduling task shutdown time to obtain the scheduling time.

8. The data management system according to claim 3, characterized in that, The data management system further includes a system management module, which comprises any one or more of the following units: department terminal management unit, user management unit, role management unit, menu management unit, permission management unit, and attribute information management unit. The department terminal management unit is used to add new department terminals, modify terminal information, and modify their affiliation. The user management unit is used to create users and add, modify, and delete users' login names, passwords, departments, job responsibilities, and user groups. The role management unit is used to create multiple roles and assign at least one of the roles to the user; The menu management unit is used to create menus and to add, modify, and delete menus. The permission management unit is used to create, modify, and delete permissions, and to assign the permissions to the user; The attribute information management unit is used to add, modify, and delete attribute information.

9. The data management system according to claim 8, characterized in that, The data management system further includes a comprehensive management module, which comprises one or more units selected from the authentication unit and the data retrieval unit. The authentication unit is used to authenticate users using the login name and password, and to store the IP address, login time, and cumulative login count of successfully authenticated users. The data retrieval unit is used to retrieve all initial metadata, the attribute information, the newly added and / or modified attribute information, and the updated metadata.

10. A data management method, characterized in that, Includes the following steps: Obtain multiple initial metadata and the data types corresponding to each of the multiple preset business systems; The database to be processed for each of the preset business systems is constructed by using multiple initial metadata of each preset business system and the data types corresponding to each initial metadata. Each of the databases to be processed is updated to obtain the updated databases of each of the preset business systems; Each target model of the preset business system is constructed based on the updated database. Import standard management information and update the standard management information to obtain updated standard management information; Based on the updated standard management information, all target models are checked and analyzed to obtain a data quality check result report; The process of constructing the database to be processed for each of the preset business systems using multiple initial metadata and the data types corresponding to each initial metadata includes: Obtain multiple initial metadata and the data types corresponding to each of the multiple preset business systems; Calculate the data capacity of multiple initial metadata for each of the preset business systems to obtain the data capacity of each preset business system, and determine whether each preset business system meets the following conditions: the data type corresponding to all initial metadata in the preset business system is any one of multiple preset data types, and the data capacity of the preset business system is greater than or equal to the preset data capacity; if yes, create a first database and store all initial metadata of the preset business systems that meet the conditions in the first database; if no, create a second database and store all initial metadata of the preset business systems that do not meet the conditions in the second database. The first database is a relational database, and the second database is a non-relational database.

Images