Data management system

Abstract

The embodiment of the invention provides a data management system, and the system comprises a data virtualization center which is used for obtaining data through a plurality of multi-source heterogeneous data platforms, and achieving at least one data management function; the data management function comprises a data view function, a data query function and a data index semantic modeling function; a metadata center; the metadata management center is used for realizing at least one metadata management function according to the data acquired by the data virtualization center and the data acquired through the multiple multi-source heterogeneous data platforms; the metadata management function comprises a metadata knowledge graph function and an intelligent strategy management function; the data index center is used for providing at least one data index function for the data virtualization center and the metadata center; the data index functions comprise an index management function, a data dimension function, an intelligent function, a data safety function and an automatic index production function, and unified management, efficient processing and sharing utilization of data are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data management system. Background Art

[0002] In today's complex enterprise data environment, many companies are faced with the problem of decentralized data systems and inefficient data processing. Especially in companies with multi-level organizational structures, such as a group headquarters and its multiple branches, each level or department may maintain its own independent data system to support its specific business operations and decision-making needs. This decentralized data system architecture often requires a large amount of data exchange and data extraction, conversion, and loading processes when conducting global data analysis or cross-level data integration, which not only increases the complexity and time cost of data processing, but may also lead to problems with data consistency and accuracy.

[0003] At the system engineering level, each branch may need to build a big data processing system or data analysis system to meet its own data processing needs, which not only leads to repeated investment in system construction, but also increases the complexity of subsequent maintenance and upgrades. At the same time, each system needs to be equipped with corresponding storage resources, which further aggravates the redundancy and waste of storage resources.

[0004] In terms of human resources, data developers and data analysts need to repeat data warehouse modeling, data development, and data analysis on multiple similar data systems, which not only consumes a lot of human resources, but also reduces work efficiency and data utilization value.

[0005] In addition, due to the similarity of business needs and data needs, there is a lack of effective data sharing and collaboration mechanisms among various data systems, resulting in data islands, hindering the circulation and integration of data and further increasing the difficulty of data processing.

[0006] Therefore, a new data system construction solution is urgently needed to achieve unified management, efficient processing and shared use of data, reduce duplicate investment in system construction, optimize resource allocation, and improve the efficiency of data processing and analysis, so as to better support the business operations and decision-making of enterprises. This new solution needs to provide an effective solution to the current problems of decentralized data systems and inefficient processing, bringing substantial benefits and value enhancement to enterprises. Summary of the invention

[0007] In view of the above problems, embodiments of the present invention are proposed to provide a data management system that overcomes the above problems or at least partially solves the above problems.

[0008] In order to solve the above problems, an embodiment of the present invention discloses a data management system, the system comprising:

[0009] A data virtualization center is used to obtain data through multiple multi-source heterogeneous data platforms and implement at least one data management function; the at least one data management function includes: a data view function, a data query function, and a data indicator semantic modeling function;

[0010] Metadata center; used to implement at least one metadata management function based on the data collected by the data virtualization center and the data obtained through multiple multi-source heterogeneous data platforms; the at least one metadata management function includes: metadata knowledge graph function and intelligent policy management function;

[0011] A data indicator center is used to provide at least one data indicator function for the data virtualization center and the metadata center; the at least one data indicator function includes: an indicator management function, a data dimension function, an intelligent function, a data security function and an automated indicator production function.

[0012] Optionally, the data view function includes: displaying data logical view function, displaying data materialized view function, performance optimization function based on materialized view, and data caching function;

[0013] The data query function includes: query function and query engine management function;

[0014] The data indicator semantic modeling function includes: building a data indicator semantic model function and managing a data indicator semantic model function.

[0015] Optionally, the data query function also includes: a query performance optimization function and a query efficiency optimization function.

[0016] Optionally, the metadata knowledge graph function includes: a metadata knowledge graph function, a metadata knowledge graph function, a data lineage function for displaying data, and a data impact analysis function for displaying data;

[0017] The metadata management function also includes: data source management function and data asset management function;

[0018] The data source management function includes: adding a data source, deleting a data source, modifying a data source, and monitoring a data source;

[0019] The data asset management functions include: data classification function, data storage function, data backup function and data recovery function.

[0020] Optionally, the intelligent policy functions include: data classification function, data grading function, data protection function, and data access control function;

[0021] The metadata management function also includes: data exploration and pre-management function and metadata storage and retrieval function;

[0022] The data exploration and pre-management functions include: data exploration function, data cleaning function, data conversion function, and data integration function;

[0023] The metadata storage and retrieval function includes: a metadata storage function and a metadata retrieval function.

[0024] Optionally, the indicator management function includes: a data indicator creation function, a data indicator update function, a data indicator deletion function and a data indicator query function.

[0025] Optionally, the data dimension functions include: data dimension creation function, data dimension modification function, data dimension deletion function, data dimension hierarchical structure management function and data dimension attribute management function; the data dimension is used to describe and classify the attributes or characteristics of data indicators.

[0026] Optionally, the intelligent functions include: artificial intelligence content generation function, data semantic recognition function, system expectation monitoring function and data traceability analysis function.

[0027] Optionally, the data security function includes: data encryption function, access control function, and authority management function.

[0028] Optionally, the automated indicator production function includes: a data indicator calculation function, a data indicator update function, and a data indicator generation function.

[0029] The embodiment of the present invention includes the following advantages: through the data virtualization center, enterprises can efficiently integrate data from multiple multi-source heterogeneous data platforms, realize logical integration of data, and avoid repeated storage and redundancy of data. This greatly reduces the repeated investment in system construction, optimizes the utilization of storage resources, and provides a unified data access interface, simplifies the process of data query and acquisition, and improves the efficiency of data processing. With the help of the metadata knowledge graph function and intelligent policy management function in the metadata center, enterprises can more comprehensively understand and manage the structure and relationship of data, and improve the quality and consistency of data. The metadata knowledge graph provides rich contextual information for data, making data analysis and mining more in-depth and accurate. Intelligent policy management automatically adjusts and optimizes data management strategies according to the characteristics of data and business needs, and improves the intelligence and automation level of data management. The comprehensive data indicator management function provided by the data indicator center includes indicator management, data dimension management, intelligent function, data security function and automatic indicator production function. Through these functions, enterprises can more conveniently create, manage and analyze data indicators, and improve the accuracy and timeliness of data indicators. In particular, the automated indicator production function greatly reduces manual intervention, improves the efficiency of indicator updates, and enables enterprises to respond to market changes and business needs more quickly. At the same time, the data security function ensures the security and confidentiality of data indicators and protects the data assets of the enterprise. In summary, the embodiments of the present invention solve the problems of decentralized enterprise data systems, low processing efficiency, waste of resources, and high manpower consumption through the synergy of the data virtualization center, metadata center, and data indicator center, and realize the unified management, efficient processing, and shared utilization of data, which brings significant economic benefits and competitive advantages to enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a block diagram of an embodiment of a data management system of the present invention;

[0031] Figure 2 It is a schematic diagram of the architecture of a data management system embodiment of the present invention;

[0032] Figure 3 is a data processing schematic diagram of an embodiment of a data management system to which the present invention is not applied;

[0033] Figure 4 It is a data processing schematic diagram of a data management system embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] One of the core concepts of the embodiments of the present invention is to integrate data from multiple multi-source heterogeneous data platforms through the concept of data latitude and longitude, and integrate a data management platform that can support multi-source heterogeneous data by combining relevant technologies in the field of digital virtualization technology.

[0036] Reference Figure 1 , shows a block diagram of a data management system embodiment of the present invention:

[0037] Data management system 10;

[0038] The data virtualization center 11 is used to obtain data through multiple multi-source heterogeneous data platforms and implement at least one data management function; the at least one data management function includes: data view function, data query function, and data indicator semantic modeling function;

[0039] In the traditional data integration exchange method, data is usually uploaded and issued through the whole network data exchange. For example, in the telecommunications field, the data of domain B (business support domain) and domain O (operation support domain) need to be processed from the provincial company and moved to the group headquarters, while the data of domain M (management domain) needs to be processed from the group headquarters and moved to the provincial companies. This data exchange method not only reduces the speed of data acquisition because data needs to be transmitted back and forth between different levels, but also increases the cost of storage and computing because data needs to be stored and processed in multiple places, resulting in a waste of resources.

[0040] In order to solve this problem, the embodiment of the present invention proposes to use the DataFabric architecture. The core idea of ​​the DataFabric architecture is to connect the metadata of the data source at two levels and update it in real time. This means that the data of both the group headquarters and the provincial company can be processed and handled at their original location without the need for physical relocation.

[0041] The DataFabric architecture is a data management architecture specifically designed to simplify data access and promote data integration and sharing. The DataFabric architecture aims to break down data silos and achieve cross-source and cross-domain data integration and sharing. It can unify the management and access of data from different data sources and in different formats, thereby improving the utilization and value of data. The DataFabric architecture uses data virtualization technology to achieve logical rather than physical integration of data. This means that users can access the required data through the virtualized data layer without moving the data, greatly improving the flexibility and efficiency of data access. Metadata is a key element in the DataFabric architecture.

[0042] Through continuous analysis and utilization of metadata, DataFabric can provide smarter data governance, data discovery, and data access capabilities. The DataFabric architecture supports multiple data types, multiple data sources, and multiple deployment environments (such as local, cloud, hybrid cloud, etc.), and has good flexibility and scalability.

[0043] In the context of DataFabric architecture, the data virtualization center 11 is an important part, which is responsible for obtaining data through multiple multi-source heterogeneous data platforms, which may include relational databases, non-relational databases, data warehouses, big data platforms, etc. The data virtualization center is not just a data collector, it implements at least one data management function, which includes:

[0044] Data view function: allows users to view data from different perspectives, such as displaying data by business line, region, time and other dimensions, making the data more intuitive and easy to understand.

[0045] Data query function: Provides powerful data query capabilities, allowing users to quickly obtain the required data through simple query statements or graphical interfaces without having to worry about underlying details such as where and how the data is stored.

[0046] Data indicator semantic modeling function: supports users to define and manage data indicators according to business needs, including the indicator name, calculation method, dimension, etc., as well as the association and dependency between indicators. This makes data analysis and mining more in-depth and accurate.

[0047] The metadata center 12 is used to implement at least one metadata management function based on the data collected by the data virtualization center and the data obtained through multiple multi-source heterogeneous data platforms; the at least one metadata management function includes: metadata knowledge graph function and intelligent policy management function;

[0048] The metadata center 12 is responsible for implementing metadata management functions based on the data collected by the data virtualization center and the data directly obtained through multiple multi-source heterogeneous data platforms. Metadata, in short, is data about data, which describes key information such as the structure, source, meaning, and relationship of the data.

[0049] In one example, the metadata center 12 may be implemented as follows:

[0050] Using active metadata technology, we build a global metadata engine. This engine can comprehensively and automatically collect metadata from the data virtualization center and multiple multi-source heterogeneous data platforms. These metadata cover all aspects of data structure, source, meaning, relationship, etc.

[0051] After the metadata is collected, a metadata knowledge graph is constructed using the metadata. This graph graphically displays the associations and dependencies between data, forming a comprehensive and intuitive understanding of data assets.

[0052] Based on the collected metadata, the metadata center can formulate and implement intelligent management strategies. These strategies may involve data storage, access control, data security, data quality and other aspects.

[0053] By analyzing the information in the metadata, such as the sensitivity of the data, the frequency of use, etc., the metadata center 12 can automatically adjust the policy to ensure efficient, secure and compliant use of the data.

[0054] The intelligent policy management function also supports manual intervention and re-creation capabilities, allowing administrators to manually adjust and optimize policies based on actual conditions, increasing the flexibility and controllability of metadata management.

[0055] The data indicator center 13 is used to provide at least one data indicator function for the data virtualization center and the metadata center; the at least one data indicator function includes: indicator management function, data dimension function, intelligent function, data security function and automatic indicator production function.

[0056] In the data management system of an enterprise or organization, data indicators are an important basis for measuring business performance, evaluating results, and making decisions. The core mission of the Data Indicator Center 13 is to provide a series of functions and services around these data indicators to ensure the accuracy, consistency, timeliness, and availability of data.

[0057] In one example, the data indicator center 13 uses standardized indicator dimension construction technology to perform standard definition and management of indicators, including atomic indicators (calculated directly from fact tables), derived indicators (derived through dimension slicing or dicing), and derived indicators (derived through native indicators and constant term operations).

[0058] Through standardized management of indicators, the Data Indicator Center 13 achieves governance at the indicator level, forms a standardized paradigm for indicator dimensions, and supports the flexible construction and consumption of indicators.

[0059] Data Indicator Center 13 provides rich dimension management functions, including standard dimensions (ordinary standard dimensions and hierarchical standard dimensions) and degenerate dimensions.

[0060] Standard dimensions record the relationship between dimensions and dimension values ​​through data tables, and use aliases to represent dimension values ​​in fact tables, and use dimension tables to map the meaning of aliases. Hierarchical standard dimensions also support the hierarchical structure of dimensions, such as the year, quarter, and month levels of the time dimension.

[0061] Degenerate dimensions record the dimension values ​​directly in the data fact table, simplifying dimension management.

[0062] The data indicator center 13 can combine the AI ​​big model to build relevant prediction and analysis algorithms to complete functions such as early warning, difference analysis, and problem analysis of indicator data.

[0063] These intelligent functions enable the Data Indicator Center 13 to not only provide basic indicator calculation and management, but also conduct deeper data analysis and prediction to provide support for business decision-making.

[0064] In the process of indicator management and data dimension management, the data indicator center 13 will take corresponding data security measures to ensure the security and privacy of the data.

[0065] Automated indicator production function:

[0066] The Data Indicator Center 13 uses automated indicator production technology to achieve automated indicator production. This includes developing NoETL's automated indicator production capabilities through standardized indicator definitions, and automatically building data calculation SQL and data query SQL.

[0067] The automated indicator production function eliminates the need for data developers and data analysts to perform complex ETL operations, thus improving the efficiency and accuracy of indicator production.

[0068] At the same time, the data indicator center 13 can also automatically select appropriate computing engines to produce and query data indicators based on the characteristics of different data sources and data types, thereby realizing cross-platform and cross-data source data indicator production.

[0069] In one embodiment, the data view function includes: displaying data logical view function, displaying data materialized view function, performance optimization function based on materialized view, and data caching function;

[0070] Display data logical view function: Logical view is a representation of the logical organization of data in a database or data warehouse. It describes the relationship between data and the structure of data, but does not involve the physical storage of data. Display data logical view function allows users to view the logical structure of data in a graphical or tabular way, including tables, fields, relationships (such as primary key-foreign key relationship), etc. This helps users understand the organization of data and provides a basis for data query and analysis.

[0071] Display data materialized view function: Materialized view is a pre-calculated and stored data query result, which is based on one or more tables and can include operations such as aggregation and join. Materialized view is used to improve query performance, because the query results have been pre-calculated and do not need to be recalculated every time a query is made. The display data materialized view function allows users to view the created materialized views, including their definitions, data sources, refresh strategies, etc. Users can choose to use materialized views to speed up queries as needed.

[0072] Performance optimization based on materialized views: This function automatically or manually optimizes query performance by using materialized views. For example, when a query request is received, the system can check whether there is a related materialized view. If so, the result of the materialized view can be used directly to answer the query, thus avoiding complex calculations on the original data.

[0073] The data cache feature stores frequently accessed data or query results in memory to reduce the number of disk accesses, thereby increasing data access speed. This is somewhat similar to materialized views, but the cache is usually temporary and more flexible and can be used for any data or query results.

[0074] The data query function includes: query function and query engine management function;

[0075] Query function: Provides a user-friendly query interface, allowing users to enter query requests in SQL, natural language or other ways, and returns query results. The query function supports various complex query operations, such as filtering, sorting, aggregation, connection, etc.

[0076] Query engine management function: manage the configuration and performance of the query engine, including selecting the appropriate query engine to handle different types of query requests, optimizing query execution plans, monitoring resource consumption and performance during query execution, etc.

[0077] The data indicator semantic modeling function includes: building a data indicator semantic model function and managing a data indicator semantic model function.

[0078] Build data indicator semantic model function: allows users to define and manage the semantic model of data indicators, including the indicator name, definition, calculation method, dimension, measurement unit, etc. The semantic model provides a unified understanding and interpretation of data indicators, helping to ensure the consistency and accuracy of data indicators.

[0079] Data indicator semantic model management function: Provides management functions for the created data indicator semantic models, including viewing, editing, deleting, version control, etc. This helps maintain the accuracy and effectiveness of the data indicator semantic model and ensures the consistent use of data indicators in the system.

[0080] Through detailed functional division and description, the powerful capabilities of the metadata center in data view, data query and data indicator semantic modeling are demonstrated. In particular, by introducing concepts such as logical view, materialized view and blood relationship, the metadata center can better support data organization, query, analysis and governance.

[0081] In one embodiment, the data query function also includes: a query performance optimization function and a query efficiency optimization function.

[0082] In one example, the query performance optimization function can be implemented in the following ways: Query plan generation: The database query optimizer generates multiple possible query execution plans based on the query statement submitted by the user, and selects the optimal execution plan through cost estimation. This process involves multiple factors such as the table connection order and the use of indexes. Index selection: Index is a key technology for accelerating data query. The query performance optimization function intelligently selects and uses indexes to reduce the scope and number of data scans, thereby improving query speed. Query rewriting: In some cases, a more efficient query plan can be generated by rewriting or transforming the query statement. For example, converting a subquery into a join operation, or adjusting the join order to reduce the amount of data exchange. Concurrency control: In a high-concurrency scenario, the query performance optimization function also needs to consider how to reasonably allocate system resources to avoid resource competition between queries and ensure that each query can be responded to in a timely manner. Data partitioning and sharding: For large-scale data sets, data partitioning or sharding technology can be used to disperse data to different physical storage units, thereby reducing the scan range of a single query and improving query performance.

[0083] In another example, the query efficiency optimization function may be implemented in the following manner:

[0084] Caching strategy: Use caching technology to store frequently queried data in memory and reduce direct access to the database. This can significantly shorten query response time and improve user experience.

[0085] Calculation and materialized views: For queries that require complex calculations or aggregations, you can generate materialized views through pre-calculation and store the results in the database. In this way, when users initiate similar queries, the results of the materialized view can be directly returned without recalculation.

[0086] Query result paging and lazy loading: For large-scale data sets, query results can be returned to users in batches through paging technology to avoid performance problems caused by loading too much data at one time. At the same time, lazy loading technology can load data when users need it, further improving query efficiency.

[0087] User feedback and adaptive optimization: Dynamically adjust query optimization strategies based on user query behavior and feedback. For example, for frequently accessed queries, you can prioritize using technologies such as caching or materialized views to accelerate them.

[0088] In one embodiment, the metadata knowledge graph function includes: a metadata knowledge graph function, a metadata knowledge graph function, a data lineage relationship function of displaying data, and an impact analysis function of displaying data;

[0089] Lineage relationships describe the flow and transformation process of data in the system, including information such as the source of the data, how it is processed, and how it is ultimately used. In the data view function, displaying lineage relationships can help users understand the source and destination of data, track the data transformation process, ensure data accuracy and consistency, and meet data governance and compliance requirements.

[0090] Through operator-level lineage analysis technology, the metadata center 12 can track the source and destination of data and determine the precise dependencies between data. This lineage analysis is not limited to the association between data tables, but can also go deep into the operator level in SQL statements to achieve more sophisticated lineage analysis.

[0091] In one example, the impact analysis function of the display data can be implemented in the following manner:

[0092] Identification of impact scope: Based on the relationships in the knowledge graph, identify the entities and scopes affected by data changes. For example, when the structure of a data table changes, it is necessary to find all other data tables or applications that depend on the table. Impact assessment: Further analyze and evaluate the identified impact scope to determine the impact and consequences that the data change may have on other data or systems. Visualization: Display the results of the impact analysis in a graphical way, such as an impact scope map, impact degree heat map, etc. Users can intuitively understand the impact and risks that data changes may bring through these charts.

[0093] The metadata management function also includes: data source management function and data asset management function;

[0094] The data source management function includes: adding a data source, deleting a data source, modifying a data source, and monitoring a data source;

[0095] Data source addition function: allows users to add new data sources such as databases, files, APIs, etc. to the system for subsequent data processing and analysis.

[0096] Data source deletion function: remove data sources that are no longer needed or have been discarded in the system to keep the data source clean and effective.

[0097] Data source modification function: modify the connection information, configuration parameters, etc. of the data source to adapt to changes in the data source or meet new data processing requirements.

[0098] Data source monitoring function: real-time monitoring of data source status such as connection status, data volume, data update frequency, etc., to ensure the stability and availability of the data source.

[0099] The data asset management functions include: data classification function, data storage function, data backup function and data recovery function.

[0100] Data classification function: According to the nature, purpose or business needs of the data, the data can be reasonably classified to facilitate data management and query.

[0101] Data storage function: Provide a secure and efficient storage mechanism to ensure data integrity and accessibility.

[0102] Data backup function: Back up data regularly or on demand to prevent data loss or damage.

[0103] Data recovery function: When data is lost or damaged, it can be quickly restored to ensure business continuity.

[0104] In one embodiment, the intelligent policy functions include: data classification function, data grading function, data protection function, and data access control function;

[0105] Data classification function: classify and manage data based on factors such as data sensitivity and business value.

[0106] Data classification function: Based on classification, data can be further classified to implement different levels of protection and control measures.

[0107] Data protection function: Use encryption, desensitization and other technical means to protect the confidentiality and integrity of data.

[0108] Data access control function: strictly control data access based on user roles, permissions and other conditions to ensure secure data access. The intelligent strategy function aims to improve the efficiency and security of data management through a series of intelligent strategies and methods.

[0109] The metadata management function also includes: data exploration and pre-management function and metadata storage and retrieval function;

[0110] The data exploration and pre-management functions include: data exploration function, data cleaning function, data conversion function, and data integration function;

[0111] Data exploration function: conduct preliminary exploration and analysis of data sources to understand the structure, type, distribution and other characteristics of the data.

[0112] Data cleaning function: remove redundant, erroneous or inconsistent parts in the data to improve the quality of the data.

[0113] Data conversion function: convert data from one format or type to another to meet different data processing requirements.

[0114] Data integration function: Integrate data from different data sources to form a unified data view or data set.

[0115] The metadata storage and retrieval function includes: a metadata storage function and a metadata retrieval function.

[0116] Metadata storage function: Metadata is stored in a structured manner to facilitate subsequent query and management.

[0117] Metadata retrieval function: provides an efficient retrieval mechanism that allows users to quickly find metadata based on keywords, attributes or conditions.

[0118] In one embodiment, the indicator management function includes: a data indicator creation function, a data indicator update function, a data indicator deletion function and a data indicator query function.

[0119] Data indicator creation function: Create new data indicators based on business needs or analysis objectives.

[0120] Data indicator update function: modify or update existing data indicators to adapt to business changes or changes in analysis needs.

[0121] Data indicator deletion function: remove data indicators that are no longer needed or have been discarded in the system.

[0122] Data indicator query function: provides a convenient query mechanism, allowing users to quickly find and obtain the required data indicators. The indicator management function is used to create, update, delete and query data indicators, which is an important basis for data analysis and decision support.

[0123] In one embodiment, the data dimension functions include: data dimension creation function, data dimension modification function, data dimension deletion function, data dimension hierarchical structure management function and data dimension attribute management function; the data dimension is used to describe and classify the attributes or characteristics of data indicators.

[0124] Data dimension creation function: Create new data dimensions based on business needs or analysis objectives.

[0125] Data dimension modification function: modify or update existing data dimensions.

[0126] Data dimension deletion function: remove data dimensions that are no longer needed or have been discarded in the system.

[0127] Hierarchical structure management function of data dimensions: manage the hierarchical relationship between data dimensions to form a multidimensional data model.

[0128] Data dimension attribute management function: manage and maintain the attributes of data dimensions, such as the name, type, value range, etc. The data dimension function is used to describe and classify the attributes or characteristics of data indicators, and is the key to data analysis and multidimensional data integration.

[0129] In one embodiment, the intelligent functions include: artificial intelligence content generation function, data semantic recognition function, system expectation monitoring function and data traceability analysis function.

[0130] AI-generated content function: Use natural language generation technology to automatically generate data reports, analysis summaries and other content.

[0131] Data semantic recognition function: understand the semantics and contextual information of data to improve the accuracy and efficiency of data processing.

[0132] The system's anticipated monitoring function: discover potential problems and risks in advance by predicting and monitoring the system's operating status.

[0133] Data traceability analysis function: Track the source and flow of data to ensure data accuracy and traceability. At the same time, it can also be used to control data quality. Intelligent functions improve the intelligence level of the data management platform by introducing artificial intelligence and machine learning technologies.

[0134] In one embodiment, the data security functions include: data encryption function, access control function, and authority management function.

[0135] In one example, the data security function may include the following functions:

[0136] Data encryption function: Encrypt sensitive data to ensure that the data is not accessed or leaked without authorization during transmission and storage. Symmetric encryption or asymmetric encryption technology can be used, and the appropriate encryption algorithm can be selected according to the sensitivity of the data and the usage scenario;

[0137] Permission management function: Establish a strict permission allocation mechanism to ensure that only authorized personnel can access and operate data indicators. Through role and permission management, different access rights are assigned to different users to control the scope of use of data indicators.

[0138] Identity authentication function: Use strong identity authentication methods such as multi-factor authentication to ensure that the identity of users accessing data indicators is authentic. This can effectively prevent unauthorized access and malicious attacks.

[0139] Logging function: Record all access and operation logs to data indicators, including access time, access user, operation content, etc. This helps track the use of data indicators and detect abnormal behavior in a timely manner.

[0140] Audit analysis function: Regularly analyze and audit logs to identify potential security risks and illegal operations. Through audit analysis, we can continuously optimize permission allocation and access control strategies to improve the security of data indicators.

[0141] Data desensitization function: Desensitize sensitive data to reduce the risk of data leakage. For example, sensitive data can be processed using methods such as replacement, masking, and generalization to reduce the risk of data leakage without changing the meaning of the original data.

[0142] Data anonymization function: remove or replace personal identity information from the data set so that the data cannot be directly associated with a specific individual. This helps protect personal privacy and prevent legal risks caused by data leakage.

[0143] Secure storage function: Select safe and reliable storage devices and storage environments to ensure the security of data indicators during storage. Distributed storage, disaster recovery and other technical means can be used to improve the reliability and availability of data indicators.

[0144] Regular backup function: Back up data indicators regularly to prevent data loss or damage. The backup data should be stored in a safe and reliable place, and recovery tests should be performed regularly to ensure the availability of the backup data;

[0145] Real-time monitoring function: Real-time monitoring of the data indicator center, including monitoring of network traffic, system logs, abnormal behavior, etc. This helps to detect potential security threats and abnormal behavior in a timely manner.

[0146] Early warning mechanism function: Establish an early warning mechanism to issue early warnings in a timely manner when potential security threats or abnormal behaviors are detected. The early warning information should include the type of threat, scope of impact, and handling suggestions, so that relevant personnel can take timely measures to deal with the threat.

[0147] Compliance review function: Regularly conduct compliance reviews on the data indicator center to ensure that all data processing activities comply with the requirements of laws and regulations. The review content should include compliance in data collection, storage, use, transmission, etc.

[0148] In summary, the data security function of the data indicator center 13 can be implemented through a variety of methods, including encryption technology, access control, security auditing, data desensitization and anonymization, secure storage and backup, security monitoring and early warning, and compliance management. These methods can complement each other and work together to ensure the security and privacy of data indicators during generation, storage, transmission, and use. The implementation method of the above data security function can also be other algorithms, which are not limited in the embodiments of the present invention.

[0149] In one embodiment, the automated indicator production function includes: a data indicator calculation function, a data indicator update function, and a data indicator generation function.

[0150] Data indicator calculation function: The system will automatically extract relevant data from the data source and perform calculation operations based on the predefined indicator calculation formula or logic. The calculation results will be stored and used for subsequent report generation, analysis or decision support.

[0151] Data indicator update function: You can configure scheduled tasks for the system to automatically check changes in data sources and recalculate indicator values ​​based on new data. In addition, users can also manually trigger update operations to meet specific business needs.

[0152] Data indicator generation function: Users can define new indicators through the graphical interface or programming interface, and specify their calculation logic and data source. The system will automatically generate new data indicators based on this information and add them to the indicator library for subsequent use. The automated indicator production function provides users with an efficient and flexible data indicator management platform by integrating the calculation, update and generation functions of data indicators. This function not only improves the production efficiency and quality of data indicators, but also reduces the cost and risk of manual participation, providing a strong guarantee for the company's data analysis and decision support.

[0153] Through the data virtualization center, enterprises can efficiently integrate data from multiple multi-source heterogeneous data platforms, realize logical integration of data, and avoid repeated storage and redundancy of data. This greatly reduces the repeated investment in system construction, optimizes the utilization of storage resources, and provides a unified data access interface, simplifies the process of data query and acquisition, and improves the efficiency of data processing. : With the metadata knowledge graph function and intelligent policy management function in the metadata center, enterprises can more comprehensively understand and manage the structure and relationship of data, and improve the quality and consistency of data. The metadata knowledge graph provides rich contextual information for data, making data analysis and mining more in-depth and accurate. Intelligent policy management automatically adjusts and optimizes data management strategies according to the characteristics of data and business needs, and improves the intelligence and automation level of data management. The comprehensive data indicator management functions provided by the data indicator center include indicator management, data dimension management, intelligent functions, data security functions, and automated indicator production functions. Through these functions, enterprises can more conveniently create, manage and analyze data indicators, and improve the accuracy and timeliness of data indicators. In particular, the automated indicator production function greatly reduces manual intervention, improves the efficiency of indicator updates, and enables enterprises to respond to market changes and business needs more quickly. At the same time, the data security function ensures the security and confidentiality of data indicators and protects the data assets of the enterprise. In summary, the embodiments of the present invention solve the problems of scattered enterprise data systems, low processing efficiency, waste of resources and high manpower consumption through the synergy of the data virtualization center, metadata center and data indicator center, and realize the unified management, efficient processing and shared utilization of data, which brings significant economic benefits and competitive advantages to the enterprise.

[0154] Reference Figure 2 , shows a schematic diagram of the architecture of a data management system embodiment of the present invention:

[0155] The business scenarios of the embodiments of the present invention may be business analysis, precise management, precision marketing, lean network operation, refined services, location services, leadership windows, wireless construction and maintenance optimization, customer perception, customer retention, inventory management, etc.

[0156] The data source may come from the data sources of the B, M, and O domains of each provincial company; as shown in the figure, the data source of the B, M, and O domains of provincial company 1 to the data source of the B, M, and O domains of provincial company 2 to ... to the data source of the B, M, and O domains of provincial company 31;

[0157] Correspondingly, data lakes have evolved from data lakes of various companies in traditional systems to regional centralized data lakes to facilitate subsequent processing;

[0158] In an embodiment of the present invention, the data management system includes a data indicator center, a data virtualization center, and a metadata center;

[0159] The data indicator center includes a unified open indicator service API, an indicator management module, a data security module, a dimension management module, an intelligent module and an automated indicator production module;

[0160] The indicator management module can realize the indicator management function, which can operate indicators in various aspects including indicator name, indicator status, indicator type, subject domain, subject, indicator description, responsible person, update record, etc., as well as the indicator caliber management function, which can operate indicators in aspects including data source, database, fact table, field, aggregation method, condition, measurement unit, associated dimension, dimension value, multi-dimensional management, calculation method, etc.;

[0161] The dimension management module can implement dimension management functions, and can operate indicators including dimension name, dimension description, dimension type, dimension value, etc., as well as dimension caliber management functions, and can operate indicators including standard dimension, degenerate dimension, data source, dimension table, dimension primary key, dimension foreign key, dimension value, dimension level, sorting, etc.;

[0162] The data security module can realize functions including authority management, data desensitization, data encryption, and data watermarking.

[0163] The intelligent module can realize functions including AIGC (Artificial Intelligence Generated Content), semantic recognition, early warning monitoring, and traceability analysis.

[0164] The data virtualization center includes a multi-source heterogeneous data logic integration module, a logical view integration module, a query engine management module, a query engine acceleration module, an indicator semantic modeling module and a materialized view module.

[0165] The metadata center includes an active metadata exploration module, a metadata knowledge graph function, a data asset metadata management function, an intelligent strategy management function and a data source management function.

[0166] Reference Figure 3 and Figure 4 , Figure 3 and Figure 4 A data processing schematic diagram of an embodiment of a data management system not applying the present invention and a data processing schematic diagram of an embodiment of a data management system applying the present invention are respectively shown;

[0167] Figure 3The top layer is the group headquarters-level data center and data application system, which is connected to the group headquarters-level data warehouse or data lake. The middle part is the whole network hub data exchange, connecting the data center and data application systems of 31 provinces. Each provincial data center and data application system has a provincial data warehouse or data lake under it.

[0168] Data flows bidirectionally between the B, M, O domain data sources and the provincial data warehouse or data lake. On the one hand, data can flow from the B, M, O domain data sources into the provincial data warehouse or data lake; on the other hand, data in the provincial data warehouse or data lake can also flow back to the B, M, O domain data sources.

[0169] The data in the provincial data warehouse or data lake is uploaded to the group headquarters data warehouse or data lake through the data middle platform and data application system, and then the data is exchanged through the whole network hub. Or the data is sent down from the group headquarters data warehouse or data lake. The group headquarters data middle platform and data application system process and apply the data in the group headquarters data warehouse or data lake.

[0170] Although there is a two-way flow of data between provincial data warehouses or data lakes and B, M, and O domain data sources, there is a lack of effective data sharing and collaboration mechanisms between provincial data systems.

[0171] This architecture easily forms data silos at the provincial level, causing difficulties in data circulation and integration between provinces, and increasing the difficulty of cross-provincial data processing. For example, cross-provincial data analysis may face problems such as inconsistent data formats and inconvenient data acquisition.

[0172] and Figure 4 , which is a data processing diagram of a data management system embodiment provided by the present invention, is similar to Figure 3 different.

[0173] The top layer is a unified data analysis indicator platform. The middle part connects the data lakes of 31 provinces through data virtualization technology, and the provincial data lakes are regional centralized data lakes.

[0174] Data flows bidirectionally between the B, M, and O domain data sources and the centralized data lakes in each region. Data can enter the centralized data lakes in each region from the B, M, and O domain data sources, and the data in the data lakes will also be fed back to the B, M, and O domain data sources.

[0175] Through data virtualization technology, the data lakes of each province are connected to a unified data analysis indicator platform. Using the DataFabric architecture, the metadata of the data source is connected at two levels and updated in real time. The data is processed and processed locally according to the specifications to build a unified data directory and data arrangement. The original data and processed data are retained in the lake by the provincial companies, or regional data lake integration is carried out.

[0176] Through data virtualization and unified data management methods, we can solve problems such as decentralized enterprise data systems, inefficient processing, waste of resources and high manpower consumption, and achieve unified management, efficient processing and shared use of data, bringing significant economic benefits and competitive advantages to enterprises.

[0177] It should be noted that, for the sake of simplicity, the system embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0178] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0179] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0180] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0181] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0183] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0184] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0185] The data management system provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the system of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data management system, characterized in that: The system comprises: A data virtualization center is used to obtain data through multiple multi-source heterogeneous data platforms and implement at least one data management function; the at least one data management function includes: a data view function, a data query function, and a data indicator semantic modeling function; Metadata center; used to implement at least one metadata management function based on the data collected by the data virtualization center and the data obtained through multiple multi-source heterogeneous data platforms; the at least one metadata management function includes: metadata knowledge graph function and intelligent policy management function; A data indicator center is used to provide at least one data indicator function for the data virtualization center and the metadata center; the at least one data indicator function includes: an indicator management function, a data dimension function, an intelligent function, a data security function and an automated indicator production function.

2. The system according to claim 1, characterized in that The data view functions include: displaying data logical view function, displaying data materialized view function, performance optimization function based on materialized view, and data caching function; The data query function includes: query function and query engine management function; The data indicator semantic modeling function includes: building a data indicator semantic model function and managing a data indicator semantic model function.

3. The system according to claim 2, characterized in that The data query function also includes: query performance optimization function and query efficiency optimization function.

4. The system according to claim 3, characterized in that The metadata knowledge graph function includes: a metadata knowledge graph function, a metadata knowledge graph function, a data lineage relationship function, and a data impact analysis function; The metadata management function also includes: data source management function and data asset management function; The data source management function includes: adding a data source, deleting a data source, modifying a data source, and monitoring a data source; The data asset management functions include: data classification function, data storage function, data backup function and data recovery function.

5. The system according to claim 1, characterized in that The intelligent strategy functions include: data classification function, data grading function, data protection function, and data access control function; The metadata management function also includes: data exploration and pre-management function and metadata storage and retrieval function; The data exploration and pre-management functions include: data exploration function, data cleaning function, data conversion function, and data integration function; The metadata storage and retrieval function includes: a metadata storage function and a metadata retrieval function.

6. The system according to claim 1, characterized in that The index management function includes: a data index creation function, a data index update function, a data index deletion function and a data index query function.

7. The system according to claim 6, characterized in that The data dimension functions include: data dimension creation function, data dimension modification function, data dimension deletion function, data dimension hierarchical structure management function and data dimension attribute management function; the data dimension is used to describe and classify the attributes or characteristics of data indicators.

8. The system according to claim 7, characterized in that The intelligent functions include: artificial intelligence content generation function, data semantic recognition function, system expectation monitoring function and data traceability analysis function.

9. The system according to claim 8, characterized in that The data security functions include: data encryption function, access control function, and authority management function.

10. The system according to claim 9, characterized in that The automated indicator production function includes: data indicator calculation function, data indicator update function, and data indicator generation function.

Images