Data center construction method, system and storage medium

Abstract

The present invention relates to a method, system and storage medium for building a data middle platform, and the method includes: building a process management layer for standardizing and streamlining the data management and application processes so that all aspects of the data life cycle can be controlled; building an asset management layer for metadata and data management to form an enterprise data asset map to understand the data source, transmission and storage methods; building a quality management layer for sorting out and analyzing data quality issues and establishing data standards. The present invention provides a systematic data management method for the entire life cycle of applications for massive data collection, calculation, storage and management, so that managers of data applications can gain insight into the complex dependencies between data, applications and systems, and then effectively manage data.

Description

Technical Field

[0001] The present invention relates to a data middle platform construction method, system and storage medium based on an asset management system, and relates to the field of information system and big data technology. Background Art

[0002] With the vigorous development of the digital economy, the research and practice of data assets have received more and more attention. Many companies have begun to build their own data middle platforms to better manage and apply their own data assets. If you do not start to build a data middle platform, a large amount of data will exist in the form of isolated islands in decentralized business systems, which is not only not conducive to the unified management of business data, but also unable to transform valuable data into data assets and play its important role in business analysis and guidance. Building an enterprise data middle platform can reduce the cost of IT system construction and interaction between systems, and quickly respond to front-end business needs.

[0003] Although the concept of data middle platform is in full swing and many enterprises have embarked on the road of trying to build a middle platform, a mature theory has not been formed, and the construction method of data middle platform has not been summarized as an effective method. At present, most of the data middle platform builders are limited by the traditional PaaS thinking, and most of them focus on platform construction and neglect standardization, which is lacking in the soft science level. Summary of the invention

[0004] In response to the above problems, the purpose of the present invention is to provide a method, system and storage medium for building a data middle platform that can collect, calculate, store and manage massive data, so that the data can be effectively managed.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for building a data center, comprising:

[0007] Build a process management layer to standardize and streamline the data management and application processes, so that all aspects of the data life cycle can be controlled;

[0008] Build an asset management layer for metadata and data management, and form an enterprise data asset map to understand data sources, transmission and storage methods;

[0009] Build a quality management layer to sort out and analyze data quality issues and establish data standards.

[0010] The method for building a data middle platform further includes building a process management layer including a data scheduling process, a data monitoring process, a data management process and a data alarm process, specifically including: a data scheduling process, which is used to provide a scheduling method triggered by a calendar, a frequency and an event; a data monitoring process, which is used to provide a monitoring method for job flows, jobs, events, plans and logs, each monitoring method using a list and / or a graph; a data management process: used to support unified management of jobs of ETL tools, scripts, stored procedures and executable programs; a data alarm process: used to provide alarms and have an error notification mechanism.

[0011] The data middle platform construction method further includes: asset management layer construction including metadata management, metadata management including metadata management maintenance, metadata query and metadata maintenance;

[0012] Metadata management and maintenance is used to add, delete, update, and query metadata types;

[0013] Metadata query is used to utilize the storage and computing characteristics of big data, such as no fixed columns, horizontal scalability, and real-time concurrency, to uniformly store all metadata instances and metadata instance relationships in a table through a graph database engine.

[0014] Metadata maintenance is used to maintain and manage published metadata.

[0015] The data middle platform construction method further includes building an asset management layer including data management, which includes data foundation management, data model management and data collection management;

[0016] Data infrastructure management is used to manage, query and maintain data;

[0017] Data model management includes data model management, model relationship maintenance and model import / export; data model management unifies the enterprise's data view by designing data models, defines the business department's needs for data information, builds the atomic layer foundation of the data warehouse, and initializes the ownership of business data; model relationship maintenance is used to check and maintain the data model after the data model design is completed, in accordance with the model relationship integrity constraints; model import and export are used to import and export the data model after the data model is established;

[0018] Data collection management includes adapter management, collection management and scheduling management; adapter management, the data adapter is centered on the processing engine, and the built-in services of the processing engine include data collection, data cleaning, data filling, and data format translation data processing functions; collection management, data collection includes log collection and data source data synchronization; scheduling management, the data adapter provides real-time monitoring tools to monitor and manage the system, and the system administrator can detect system failures and abnormal operation conditions in a timely manner and deal with them in a timely manner; at the same time, it can also conduct statistical analysis on the data processing situation and keep abreast of the data processing situation.

[0019] The data center construction method further includes the following steps:

[0020] Conceptual model design: define the system boundaries, determine the subject domain and its content;

[0021] Logical model design: determine the dimensional modeling method and organize data;

[0022] Physical model design: Physicalize the logical model of the data warehouse into the database.

[0023] The data middle platform construction method described above further includes metadata checking in the quality management layer construction, and the metadata checking includes: metadata consistency checking; metadata attribute checking; generating metadata quality reports and metadata standardization checking.

[0024] The data middle platform construction method further includes the construction of the quality management layer, which also includes the construction of data standards. The construction process of data standards includes:

[0025] Standards planning: sort out the overall scope of data standards construction, define the data standards system framework and classification, and formulate an implementation plan for data standards;

[0026] Standard preparation: determine the standard templates for each classification data, prepare the data standards, and form a preliminary draft of the data standards;

[0027] Standard review and release: revise and improve data standards to form formal data standards;

[0028] Enhanced implementation and maintenance of standards: Track the implementation of standards and evaluate their effectiveness, establish corresponding management processes for standard changes, and manage standard versions.

[0029] In a second aspect, the present invention further provides a data middle platform construction system, the system comprising:

[0030] The process management layer building unit is used to build the process management layer, standardize and streamline the data management and application processes, and control all aspects of the data life cycle;

[0031] The asset management layer building unit is used to build the asset management layer for metadata and data management, forming an enterprise data asset map to understand the data source, transmission and storage methods;

[0032] The quality management layer building unit is used to build the quality management layer, sort out and analyze data quality issues, and establish data standards.

[0033] In a third aspect, the present invention further provides an electronic device, comprising at least a processor and a memory, wherein a computer program is stored in the memory, and wherein the processor executes the computer program to implement the data middle platform building method described in the first aspect of the present invention.

[0034] In a fourth aspect, the present invention further provides a computer storage medium having computer-readable instructions stored thereon, and the computer-readable instructions can be executed by a processor to implement the data middle platform building method described in the first aspect of the present invention.

[0035] The present invention adopts the above technical solution, which has the following advantages:

[0036] 1. The present invention provides a systematic data platform construction method for the entire life cycle of applications for massive data collection, calculation, storage and management, which manages a series of matters such as data definition, format, value range, business rules, processing logic, security permissions and processing dependencies between data in each link. Through the data middle platform, data users can clearly understand data and data relationships; in addition, managers of data applications can gain insight into the complex dependencies between data, applications and systems, and effectively manage data;

[0037] 2. With the influx of massive data, enterprises are increasingly in need of efficient data management. The data middle platform built by the present invention can perform efficient data governance, which helps to make data management more standardized, application more efficient, and the decision-making it supports more accurate;

[0038] 3. Metadata management in the data middle platform system built by the present invention is an important component and an important foundation for enterprises to realize data assets and asset services. It is inextricably linked to data security, data quality, data architecture, data model, etc. in the data management environment, and is also a bridge for business and technology communication. In addition, the present invention also incorporates data standard management into the data asset management system, establishes data standards for the subjects and objects of activities, and uses data models to describe the logical relationship between objects;

[0039] In summary, the present invention provides a more systematic method theory, which provides a basis for the construction and management of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:

[0041] Figure 1 This is an architecture diagram of the data middle platform system in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0043] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0044] For ease of description, spatially relative terms may be used herein to describe the relationship of one element or feature relative to another element or feature as shown in the figures, such as "inside", "outside", "inner side", "outer side", "below", "above", etc. Such spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the figures.

[0045] The data middle platform construction method, system and storage medium provided by the embodiments of the present invention include: building a process management layer for standardizing and streamlining the data management and application processes so that all aspects of the data life cycle can be controlled; building an asset management layer for metadata and data management to form an enterprise data asset map to understand the data source, transmission and storage methods; building a quality management layer for sorting out and analyzing data quality issues and establishing data standards. The present invention provides a systematic data platform for the entire life cycle of applications for massive data collection, calculation, storage and management, which allows data users to clearly understand data and data relationships, and allows managers of data applications to gain insight into the complex dependencies between data, applications and systems, and effectively manage data.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment provides a method for building a data middle platform based on a data asset governance system. The data middle platform based on the data asset governance system includes a three-level construction process, which are: process management layer, asset management layer and quality management layer. The construction process of each layer is as follows:

[0048] S1. Build process management layer

[0049] The process management layer is used to standardize and streamline the processes involving data management and application, and to clarify the roles and permissions of various actions so that all aspects of the data life cycle can be effectively managed, including data scheduling process, data monitoring process, data management process and data alarm process. The construction specifications of the above processes can be set according to the actual use of the data middle platform, and are not limited here.

[0050] In some implementations, the process management layer is visually configured and built by the data flow platform, and unified management of all jobs, data processing, and unified scheduling of stored procedures and scripts are achieved through the process management layer. The specific implementation includes:

[0051] Data scheduling process: Provides scheduling methods such as calendar, frequency, and event triggering. Each type can provide multiple scheduling rules according to needs, and these scheduling methods can be used in combination with each other to implement various complex business calls, which can be configured according to specific needs.

[0052] Data monitoring process: Provide monitoring of job flows, jobs, events, plans, logs, etc. Each method can be in both list and graphic form. Preferably, the execution log can view historical and current data status; further, it can also provide manual intervention methods such as immediate execution, interruption, breakpoint resumption, reset, etc., which greatly reduces data operation and maintenance work.

[0053] Data management process: supports unified management of various jobs such as ETL tools, scripts, stored procedures, and executable programs, and can transcend platform limitations to achieve unified calling of various jobs.

[0054] Data alarm process: It provides centralized alarm function, which can check the job failure in real time; it also has an error notification mechanism, which can realize real-time acquisition of job operation status through configuration.

[0055] S2. Building the asset management layer

[0056] The asset management layer is used for in-depth management of metadata and data, forming a "map" of enterprise data assets and understanding each data source, transmission and storage method.

[0057] The asset management layer mainly includes metadata management and data management.

[0058] Metadata management provides users with high-quality, accurate, and easy-to-manage data, which runs through the entire life cycle of data center construction, operation, and maintenance. At the same time, in the entire process of data center construction, data source analysis, ETL process, database structure, data model, organization of business application themes, and front-end display all need to be supported by corresponding metadata. Through metadata management, an accurate view of the information data assets of the entire system is formed. Through a unified view of metadata, the data cleaning cycle is shortened and data quality is improved so that the massive data from various business systems in the data center project can be systematically managed, the relationship between business metadata can be sorted out, and information data standards can be established to improve the interpretation and definition of these data, forming a consistent and unified data definition within the enterprise, and these data sources, operation conditions, changes, etc. can be tracked and analyzed.

[0059] The basic function of data management is to extract valuable data from a large amount of data resources according to user requirements. For example, these data can be obtained through retrieval, sorting, merging, conversion, aggregation, etc. Data management has to solve two main problems: one is to define the required forms of various data, and the other is how the system handles these requirements.

[0060] In some embodiments of the present invention, metadata management includes metadata management maintenance, metadata query and metadata maintenance.

[0061] Metadata management and maintenance are used to add, delete, update, and query metadata types. The system uses the real-world object-oriented model to define metadata types. Each metadata type has several attributes, and there are inheritance relationships, association relationships, and inclusion relationships between metadata types. The metadata type modeling method can define both structured metadata and unstructured metadata, and has strong flexibility and versatility. Preferably, the steps for establishing metadata management and maintenance include: designing basic types, designing object attribute types, designing object types, and designing inheritance relationships of object types. When the system loads a new metadata type, it first parses the enumeration, structure, and tag definitions to create a basic type instance; secondly, it parses the inheritance relationship to create an abstract metadata type instance; then it parses the object type to create a metadata type instance; finally, it parses the association, inclusion, and inheritance relationships between object types, initiates a query request to the metadata query module, creates nodes for each metadata type, and adds edges between metadata type nodes to establish a graph relationship.

[0062] Metadata query utilizes the storage and computing characteristics of big data, such as no fixed columns, horizontal scalability, and high real-time concurrency, to store all metadata instances and metadata instance relationships in a table (for example: HBASE) through a graph database engine, reducing the workload of traditional data table definition. While storing metadata instances, the system extracts information from metadata instances to create metadata indexes for efficient query (for example, SOLR and Elastic Search).

[0063] Preferably, the steps for establishing a metadata query are as follows:

[0064] 1. In the early stage of building a data warehouse system, determine the boundary scope of the system based on preset requirements. The principle of determining the scope of the system is to first ensure the key points, not to seek size, but to seek precision.

[0065] 2. After the boundary scope of the system is determined, the metadata of the existing system is sorted out and added to the semantic layer. Then it is stored in a database, which can be a dedicated metadata knowledge base or a general relational database.

[0066] 3. Determine the scope of metadata management. For example, use metadata to manage the data conversion process in the data warehouse and the data extraction route, so that data warehouse developers and users can understand the entire historical process of the data in the warehouse.

[0067] 4. Determine the tools for metadata management. Use certain tools to complete the corresponding work. Currently, the relevant tools include Microsoft's Repositry, which has a corresponding programming interface, which can be used to complete the function of metamodel storage and retrieval; similar to it is Platinum's OEE; in addition, there is Sybase's Wcc, which can integrate extraction tools and conversion tools through an old standard before MDC - MDIS, so that data extraction and conversion can be displayed in one window, and the semantic layer can be exported in MDIS format to a front-end tool (such as Cognos's Impropttu).

[0068] Metadata maintenance mainly involves the maintenance and management of metadata that has been published. If metadata that has been published and put online needs to be adjusted or optimized, the metadata publishing process must be repeated. Direct modification of metadata is not allowed. For security reasons, all metadata operations must be recorded in the metadata operation log. The system automatically generates a general maintenance interface based on metadata type attributes and their extended attribute rules, the inclusion relationship between metadata types, and the association relationship between metadata types. When a user enters the module to maintain a metadata instance, the system parses the metadata type attributes and their extension rules, creates an attribute attribute configuration interface, and the user's configuration operation complies with the attribute extension rules; the system parses the metadata type inclusion relationship and creates a drill-down interface, where users can drill down to view sub-metadata; the system parses the metadata type association relationship and creates an association interface. The sub-module interacts with the metadata query module to implement the addition, modification, deletion, and query of metadata instances.

[0069] In some embodiments of the present invention, data management includes data foundation management, data model management and data collection management.

[0070] In some implementations, data infrastructure management includes data management maintenance, data query, and data maintenance.

[0071] Data management and maintenance is used to add, delete, update, and query metadata types. Its establishment structure is the same as metadata management and maintenance, so it will not be repeated here.

[0072] Data query, taking advantage of the storage and computing characteristics of big data, such as no fixed columns, horizontal scalability, and high real-time concurrency, all metadata instances and metadata instance relationships are uniformly stored in a table (for example: HBASE) through a graph database engine, reducing the workload of traditional data table definition. While storing metadata instances, the system extracts information from metadata instances to create metadata indexes for efficient query (for example, SOLR and Elastic Search). The specific establishment process is similar to the metadata query process and will not be repeated here.

[0073] Data maintenance automatically generates a general maintenance interface based on metadata type attributes and their extended attribute rules, the inclusion relationship between metadata types, and the association relationship between metadata types. When a user enters the module to maintain a metadata instance, the system parses the metadata type attributes and their extension rules, creates an attribute-characterized configuration interface, and the user's configuration operations comply with the attribute extension rules; the system parses the metadata type inclusion relationship, creates a drill-through interface, and the user can drill through to view the sub-metadata; the system parses the metadata type association relationship, and creates an association interface. The sub-module interacts with the metadata query module to implement the addition, modification, deletion, and query of metadata instances, which is similar to the metadata maintenance establishment process and will not be repeated here.

[0074] In other implementations, data model management includes data model management, model relationship maintenance, and model import / export.

[0075] Data model management: Data model refers to the unified definition, coding and naming of enterprise operations and logical rules using entities, attributes and their relationships; it is a set of languages ​​for communication between business personnel and developers. The role of data model is to unify the data view of the enterprise, define the needs of business departments for data information, is the basis for building the atomic layer of the data warehouse, supports the development plan of the data warehouse, and initializes the ownership of business data; among them, the steps of data model design are:

[0076] Conceptual model design (business model): define the system boundary; determine the main subject areas and their contents;

[0077] Logical model design: dimensional modeling method (fact table, dimension table); organizing data in star and snowflake formats;

[0078] Physical model design: the process of physicalizing the logical model of the data warehouse into the database;

[0079] Model relationship maintenance: After the data model is designed, the data model must be checked and maintained in accordance with the model relationship integrity constraints. Integrity constraints are the constraints and dependencies of data and its relationships in a given data model, which are used to limit the database state and state changes that conform to the data model to ensure the correctness, validity and compatibility of the data, including: entity integrity, referential integrity, and user-defined integrity.

[0080] Model import and export: After the data model is established, the model can be imported and exported.

[0081] In some embodiments of the present invention, data collection management includes adapter management, collection management and scheduling management.

[0082] In some implementations, adapter management: Data adapters are distributed data collection management services that can provide flexible and general data collection management functions that meet the needs of enterprise-level applications, allowing users to easily mine, extract, convert and manage data anytime and anywhere. The data adapter is centered on the processing engine, and the built-in services of the processing engine include: basic data processing functions such as data collection, data cleaning, data filling, and data format translation.

[0083] In other implementations, collection management: data collection is mainly divided into log collection and data source data synchronization.

[0084] Log collection can be divided into browser page log collection and client log collection according to the product type; data source data synchronization can be divided into direct data source synchronization, generated data file synchronization, and database log synchronization according to the synchronization method.

[0085] Among them, browser page collection mainly collects page browsing logs (PV / UV, etc.) and interactive operation logs (operation events). The collection of these logs is generally carried out by implanting standard statistical JS code on the page. The process of implanting code can be written manually during the page function development stage, or it can be dynamically implanted by the server when the corresponding page is requested when the project is running. After the page logs are collected, they need to be cleared and preprocessed on the server side. For example, cleaning false traffic data, identifying attacks, normal completion of data, elimination of invalid data, data formatting, data isolation, etc.

[0086] Client log collection generally develops a dedicated statistical SDK for APP client data collection. The collection of client data has high business characteristics and high customization requirements. Therefore, in addition to some basic data of the application environment, data is collected more from the perspective of "by event", such as click events, login events, business operation events, etc. Basic data can be collected by SDK by default, and other events are defined by the business side, and the SDK interface is called according to the specifications. Merge the data on H5 to Native, and then send it uniformly through the SDK. An important principle of log collection is "standardization" and "normalization". Only when the collection method is standardized and normalized can the collection cost be minimized, the efficiency of log collection can be improved, and the subsequent statistical calculations can be implemented more efficiently.

[0087] Direct data source synchronization refers to directly connecting to the business database and reading the data of the target database through a standardized interface (such as JDBC). This method is relatively easy to implement, but if the data source has a large business volume, it may affect performance.

[0088] Generating data file synchronization means generating data files from the data source system and then synchronizing them to the target database through the file system. This method is suitable for scenarios where data sources are relatively scattered. Verification must be done before and after data file transmission. At the same time, file compression and encryption must be performed appropriately to improve efficiency and ensure security.

[0089] Database log synchronization refers to synchronization based on the log files of the source database. Most databases now support the generation of data log files and the use of data log files to restore data. Therefore, this data log file can be used for incremental synchronization. This method has less impact on system performance and has higher synchronization efficiency.

[0090] In some other implementations, scheduling management: the data adapter provides a real-time monitoring tool to monitor and manage the system, so that the system administrator can immediately detect system failures and abnormal operating conditions and handle them in a timely manner; at the same time, the data processing situation can also be statistically analyzed to keep abreast of the data processing situation.

[0091] S3. Build a quality management layer

[0092] The quality management layer is used to sort out and analyze data quality issues and find out the current status of data quality. By finding out the current status and then selecting appropriate solutions for different quality issues, detailed solutions are formulated, and finally a knowledge base for solving data quality problems is formed for reference by later generations. Data quality governance refers to the standardization of each information collection point of the information system in order to meet the needs of information utilization, including a series of processes such as establishing standardized operating procedures, verifying original information, feedback and correction of erroneous information, etc.

[0093] The quality management layer includes metadata checking, data quality and data standards.

[0094] In some embodiments of the present invention, metadata checking is to comprehensively check the definition, organization, accuracy and relationship between data based on the standardization, integrity and correctness of data to determine whether the data meets the quality requirements. Metadata checking includes:

[0095] 1. Metadata consistency check

[0096] According to common data storage types, metadata consistency checks mainly complete object-level record consistency checks, field-level value, time, and character type data consistency checks, namely, the commonly mentioned table-level count, field-level value sum, time type difference sum, and character type checksum.

[0097] 2. Metadata attribute check

[0098] Attribute item checking method: number of attribute items, attribute item definition

[0099] Attribute value checking method: Check by the characteristics of the attribute value. | Illegal character check, non-emptiness check, frequency method check, fixed length check.

[0100] 3. Metadata Quality Report

[0101] After the metadata is verified for consistency and attributes, a metadata quality report is generated, detailing the quality issues and solutions for the metadata.

[0102] 4. Metadata compliance check

[0103] After resolving metadata quality issues, metadata needs to be checked for compliance, including metadata standards and specifications.

[0104] In some embodiments of the present invention, data quality includes three basic categories: form quality, content quality, and utility quality. Among them, form quality mainly considers whether the data set can well match the business needs in terms of structure and expression, and whether it is easy to understand and obtain. Content quality mainly considers whether the specific content and value of the data set are consistent with the actual business. Utility quality mainly examines whether the data set has a high correlation in terms of business characteristics and time dimensions. The definition of data quality indicators is shown in Table 1:

[0105] Table 1

[0106]

[0107] In some embodiments of the present invention, data quality also includes data verification. Data verification refers to checking the integrity and consistency of data to improve data quality. Data verification is a complete data quality control chain from data collection, preprocessing, comparison, analysis, early warning, notification and problem repair. The verification rules are: correlation check, including: whether the key value association between the two data tables exists; whether the data volume of the two data tables is consistent; whether the table structures of the two data tables are consistent, such as whether the number of fields, field type and width are consistent; whether the contents of the two data tables are consistent, and whether the content of one of the data tables is missing. Configure verification rules, such as field mapping, etc. Configure scheduling rules, such as scheduling frequency, etc.; configure report templates, such as verification results, etc.

[0108] In some embodiments of the present invention, data quality also includes problem management. The analysis must first find the reasons or situations for the quality problems, and formulate data cleaning strategies for these reasons. Common data quality problems include: invalidity, duplication, missing, inconsistency, wrong value, format error, business logic rule problems, data extraction program errors, etc. In addition, inconsistent statistical calibers can also lead to the data you see is not what you want. According to these situations, according to the amount of data and the requirements of the mining system, a program must be written for automatic cleaning by the computer. The problem management process is as follows:

[0109] Define business needs and methods: clarify the focus, timing and goals of data quality management to guide all work during the entire project; Analyze the information environment: collect, organize and analyze the information environment related to data quality, clarify the information life cycle, ensure that relevant data can be evaluated, and design data acquisition and evaluation plans; Evaluate data quality: evaluate data quality for the data quality dimensions applicable to this problem. The evaluation results are used to determine the root causes of data quality problems and areas that need improvement; Determine the causes of data quality problems: determine the root causes of data quality problems, prioritize them, and provide specific suggestions for solving these problems; Develop improvement plans: determine the final specific solutions; Prevent future data errors: Implement solutions to solve the root causes of data quality problems. Real-time control: monitor and verify the improvements made, and maintain the results through standardization, archiving and continuous monitoring of improvements. Communicate actions and results: Archive and communicate the effectiveness of quality management, improvements made and the results of improvements.

[0110] In some embodiments of the present invention, data standards are the basis of data asset management and are the process of accurately defining data assets; the embodiments of the present invention manage the data models, databases, tables, fields, reports (indicator storage fields), fields, and data relationships between fields in the source systems, data platforms, data marts, and data applications in the data middle platform through the construction of metadata and data management. According to the big data standard system formulated by the Big Data Standard Working Group of the National Information Technology Standardization Technical Committee, the data standard system framework consists of seven categories of standards, namely: basic standards, data standards, technical standards, platform and tool standards, management standards, security and privacy standards, and industry application standards. Data standards are consistent agreements on the expression, format, and definition of data, including unified definitions of data business attributes, technical attributes, and management attributes; the purpose of data standards is to make the data used and exchanged within and outside the organization consistent and accurate. The construction and implementation process of data standards is as follows:

[0111] Phase 1: Standard Planning

[0112] Starting from the actual situation and combining industry experience, we collect national standards, current standards, new system requirement standards and industry common standards, sort out the overall scope of data standard construction, define the data standard system framework and classification, and formulate an implementation plan for data standards.

[0113] Phase 2: Standards Preparation

[0114] According to the framework and classification of the data standard system, the data standard templates for each classification are determined first. Then, relevant personnel compile the data standards based on the research results of relevant national standards, industry standards, technical business needs, etc. to form a preliminary draft of the data standards.

[0115] Phase 3: Standard Review and Release

[0116] After the standard is compiled, in order to ensure the integrity and standardization of the data standard, it is necessary to review the data standard, revise and improve the data standard after fully soliciting the opinions of experts and personnel from relevant departments. After the improved data standard is approved by the leadership, it can be released to the entire enterprise to form a formal data standard.

[0117] Phase 4: Standard implementation and maintenance enhancement

[0118] Not all data standards can be fully implemented. In actual work, there may be situations where historical systems cannot be transformed. Therefore, we must first determine the data standard implementation strategy and scope, and formulate a corresponding implementation plan. Then, we must promote the implementation of the data standard implementation plan, track the implementation of the standards, and evaluate the effectiveness.

[0119] Data standards may need to be continuously updated and improved as business development, national standards, industry standards, and regulatory requirements change. During the data standard maintenance phase, it is necessary to establish a corresponding management process for standard changes and manage standard versions.

[0120] In summary, the present invention forms a closed loop of data quality management by building the above-mentioned management layers.

[0121] Example 2

[0122] The above-mentioned embodiment 1 provides a method for building a data middle platform, and correspondingly, this embodiment provides a system for building a data middle platform. The building system provided in this embodiment can implement the method for building a data middle platform in embodiment 1, and the building system can be implemented by software, hardware, or a combination of software and hardware. For example, the building system may include integrated or separate functional modules or functional units to execute the corresponding steps in each method of embodiment 1. Since the building system of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the building system of this embodiment is only schematic.

[0123] This embodiment provides a data middle platform construction system, the system includes:

[0124] The process management layer building unit is used to build the process management layer, standardize and streamline the data management and application processes, and control all aspects of the data life cycle;

[0125] The asset management layer building unit is used to build the asset management layer for metadata and data management, forming an enterprise data asset map to understand the data source, transmission and storage methods;

[0126] The quality management layer building unit is used to build the quality management layer, sort out and analyze data quality issues, and establish data standards.

[0127] Example 3

[0128] This embodiment provides an electronic device corresponding to the data middle platform construction method provided in this embodiment 1. The electronic device can be an electronic device used for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the construction method of embodiment 1.

[0129] The electronic device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through a bus to complete mutual communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The memory stores a computer program that can be run on the processor, and the processor executes the data middle platform construction method provided in this embodiment 1 when running the computer program.

[0130] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0131] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0132] Example 4

[0133] The data middle platform construction method of this embodiment 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the construction method described in this embodiment 1 are loaded.

[0134] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0135] It should be noted that the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. Each box in the flowchart or block diagram may represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not deviate from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention. The above content is only a specific implementation of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be the scope of protection of the claims.

Claims

1. A method for building a data center, characterized in that include: Build a process management layer to standardize and streamline the data management and application processes, so that all aspects of the data life cycle can be controlled; Build an asset management layer for metadata and data management, and form an enterprise data asset map to understand data sources, transmission and storage methods; Build a quality management layer to sort out and analyze data quality issues and establish data standards; including: The asset management layer construction further includes data management, which includes data foundation management, data model management and data collection management; Data infrastructure management is used to manage, query and maintain data; Data model management includes data model management, model relationship maintenance and model import / export; data model management unifies the enterprise's data view by designing data models, defines the business department's needs for data information, builds the atomic layer foundation of the data warehouse, and initializes the ownership of business data; model relationship maintenance is used to check and maintain the data model after the data model design is completed, in accordance with the model relationship integrity constraints; model import and export are used to import and export the data model after the data model is established; Data collection management includes adapter management, collection management and scheduling management; adapter management, the data adapter is centered on the processing engine, and the built-in services of the processing engine include data collection, data cleaning, data filling, and data format translation data processing functions; collection management, data collection includes log collection and data source data synchronization; scheduling management, the data adapter provides real-time monitoring tools to monitor and manage the system, and the system administrator can detect system failures and abnormal operation conditions in a timely manner and deal with them in a timely manner; at the same time, it can also conduct statistical analysis on the data processing situation and keep abreast of the data processing situation.

2. The data middle platform construction method according to claim 1 is characterized in that: The process management layer includes the construction of data scheduling process, data monitoring process, data management process and data alarm process, which specifically include: Data scheduling process, used to provide calendar, frequency, and event-triggered scheduling methods; Data monitoring process, used to provide monitoring methods for job flows, jobs, events, plans, and logs, each of which uses lists and / or graphics; Data management process: used to support unified management of ETL tools, scripts, stored procedures, and executable programs; Data alarm process: used to provide alarms and have an error notification mechanism.

3. The data middle platform construction method according to claim 1 is characterized in that: The construction of the asset management layer includes metadata management, which includes metadata management and maintenance, metadata query and metadata maintenance; Metadata management and maintenance, used to add, delete, update, and query metadata types; Metadata query is used to utilize the storage and computing characteristics of big data, such as no fixed columns, horizontal scalability, and real-time concurrency, to uniformly store all metadata instances and metadata instance relationships in a table through a graph database engine; Metadata maintenance is used to maintain and manage published metadata.

4. The data middle platform construction method according to claim 1 is characterized in that: The data model design steps include: Conceptual model design: define the system boundaries, determine the subject domain and its content; Logical model design: determine the dimensional modeling method and organize data; Physical model design: Physicalize the logical model of the data warehouse into the database.

5. The method for building a data middle platform according to any one of claims 1 to 4, characterized in that: The construction of the quality management layer includes metadata checking, which includes: metadata consistency checking; metadata attribute checking; generating metadata quality reports and metadata standardization checking.

6. The method for building a data middle platform according to any one of claims 1 to 4, characterized in that: The establishment of the quality management layer also includes the establishment of data standards. The process of establishing data standards includes: Standards planning: sort out the overall scope of data standards construction, define the data standards system framework and classification, and formulate an implementation plan for data standards; Standard preparation: determine the standard templates for each classification data, prepare the data standards, and form a preliminary draft of the data standards; Standard review and release: revise and improve data standards to form formal data standards; Enhanced implementation and maintenance of standards: Track the implementation of standards and evaluate their effectiveness, establish corresponding management processes for standard changes, and manage standard versions.

7. A data middle platform construction system, characterized in that: The system includes: The process management layer building unit is used to build the process management layer, standardize and streamline the data management and application processes, and control all aspects of the data life cycle; The asset management layer building unit is used to build the asset management layer for metadata and data management, forming an enterprise data asset map to understand the data source, transmission and storage methods; The quality management layer construction unit is used to build the quality management layer, sort out and analyze data quality issues, and establish data standards, including: The asset management layer construction further includes data management, which includes data foundation management, data model management and data collection management; Data infrastructure management is used to manage, query and maintain data; Data model management includes data model management, model relationship maintenance and model import / export; data model management unifies the enterprise's data view by designing data models, defines the business department's needs for data information, builds the atomic layer foundation of the data warehouse, and initializes the ownership of business data; model relationship maintenance is used to check and maintain the data model after the data model design is completed, in accordance with the model relationship integrity constraints; model import and export are used to import and export the data model after the data model is established; Data collection management includes adapter management, collection management and scheduling management; adapter management, the data adapter is centered on the processing engine, and the built-in services of the processing engine include data collection, data cleaning, data filling, and data format translation data processing functions; collection management, data collection includes log collection and data source data synchronization; scheduling management, the data adapter provides real-time monitoring tools to monitor and manage the system, and the system administrator can detect system failures and abnormal operation conditions in a timely manner and deal with them in a timely manner; at the same time, it can also conduct statistical analysis on the data processing situation and keep abreast of the data processing situation.

8. An electronic device, comprising at least a processor and a memory, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program, it is executed to implement the data middle platform building method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the data middle platform building method described in any one of claims 1 to 6.

Images