Data service system

Abstract

The invention relates to the technical field of data service, and discloses a data service system, which comprises a data access module used for accessing various data sources; the data processing module is used for performing standard design, metadata management, data development and data security processing on various data sources; the data management module is used for performing basic management, API (Application Program Interface) management, data modeling, data treatment, data table management, index management, label management, safety management and data service on the various data sources processed by the data processing module; the data query module is used for querying various data sources managed by the data management module; and the data analysis module is used for analyzing various data sources queried by the data query module. Finally, a unified data management platform is formed through the data access module, the data processing module, the data management module, the data query module and the data analysis module, and the life cycles of various data assets can be managed and controlled, so that the purpose of optimizing the uniformity, the safety and the normalization of the data is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data services, and in particular to a data service system. Background Art

[0002] Data services are designed to help users better collect, process, analyze and utilize data through a series of data-related services to meet their business needs and support decision making.

[0003] In the related technologies, due to the relatively immature construction of business data warehouses and service structures, data is deposited in the data warehouse after being extracted, converted, loaded and produced. There is a lack of data consumption service capabilities for the business side. When facing different business needs, customized development is used to realize data consumption, resulting in high data acquisition costs, inconsistent data, non-standard business services, and insecure data. Summary of the invention

[0004] In view of this, the present invention provides a data service system to solve the problems of high data acquisition cost, inconsistent data, non-standard business services, and insecure data.

[0005] In a first aspect, a data service system is provided, the system comprising:

[0006] Data access module, used to access various data sources;

[0007] Data processing module, used for standard design, metadata management, data development, and data security processing of various data sources;

[0008] The data management module is used to perform basic management, API management, data modeling, data governance, data table management, indicator management, label management, security management and data services on various data sources processed by the data processing module;

[0009] Data query module, used to query various data sources managed by the data management module;

[0010] The data analysis module is used to analyze various data sources after being queried by the data query module.

[0011] The disclosed embodiment forms a unified data management platform through a data access module, a data processing module, a data management module, a data query module and a data analysis module, which can realize the management and control of the life cycle of various data assets to achieve the purpose of optimizing the uniformity, security and standardization of data.

[0012] In some optional implementations, the data processing module includes: a specification design unit, a metadata management unit, a data development unit, and a first security management unit;

[0013] The specification design unit is used to carry out specification design for various data sources in accordance with the data warehouse specification requirements and data standard requirements. The metadata management unit is used to collect metadata and manage data lineage for various data sources. The data development unit is used to develop and process data integration, data cleaning, data conversion, data loading, and data synchronization for various data sources. The first security management unit is used for security management and control of classification and grading of various data sources, sensitive data monitoring, and permission management.

[0014] The disclosed embodiment performs standard design, metadata management, data development and security management on various data sources through a data processing module, which is conducive to ensuring the standardization and security of data.

[0015] In some optional implementations, the data management module includes: a grassroots management submodule, a middle-level management submodule and a high-level management submodule; the grassroots management submodule includes: a metadata synchronization unit, an approval center unit and an authority center unit.

[0016] The disclosed embodiment performs basic management, middle management, and high-level management on various data sources through a data management module, thereby achieving unified data management.

[0017] In some optional implementations, the metadata synchronization unit is used to synchronize metadata for the various data sources managed by the metadata management unit; the approval center unit is used to perform data approval on the various data sources managed by the middle-level management submodule; and the authority center unit is used to interact with the first security management unit to process the various data sources.

[0018] The disclosed embodiments ultimately help ensure data consistency, security, and stability through specific metadata synchronization, data approval, and data authority management.

[0019] In some optional implementations, the middle-level management submodule includes: an API management unit, a data modeling unit, a data table management unit, an indicator management unit, a label management unit, a governance workbench unit, and a second security management unit.

[0020] The disclosed embodiment performs API management, data modeling, data table management, indicator management, label management, data governance, and data security management through a middle-level management submodule, which is beneficial to ensuring the consistency, accuracy, and security of data in the production process.

[0021] In some optional implementations, the API management unit is used to perform API construction, API services, SQL adaptation, query acceleration, call dashboard, interface monitoring, data source management, and resource management for various data sources;

[0022] Data modeling unit, used for logical modeling, visual modeling, model management, and automated modeling of various data sources;

[0023] The data table management unit is used to perform data retrieval, data table lineage management, table building tool management, and business isolation management on multiple data sources;

[0024] The indicator management unit is used to manage indicator dictionaries, indicator lineage, indicator semantics registration, and indicator production for various data sources;

[0025] The label management unit is used to manage label registration, label lineage and label production for various data sources;

[0026] The governance workbench unit is used to perform business data health governance, resource usage analysis governance, governance reporting, and workbench governance on multiple data sources;

[0027] The second security management unit is used to classify and grade various data sources, scan sensitive data, manage security policies, and send them to the approval center unit.

[0028] The disclosed embodiments facilitate full management and control of the entire life cycle of data assets through specific API management, data modeling, data tables, indicator management, label management, governance workbench management, and security management, thereby achieving unified data management and consistency of data indicators.

[0029] In some optional implementations, the high-level management submodule includes: an asset metadata service unit, an indicator service unit, an offline service unit, a real-time service unit, and a front-end specification adaptation unit.

[0030] The asset metadata service unit is used to provide services for various data sources according to table metadata, indicator metadata or tag metadata, and to provide lineage services for various data sources;

[0031] The indicator service unit is used to perform indicator query, indicator retrieval, materialization acceleration, pre-broadening, pre-aggregation, and indicator set query on various data sources;

[0032] Offline service unit, used to support offline services for multiple data sources;

[0033] Real-time service unit, used to support real-time services for multiple data sources;

[0034] The front-end specification adaptation unit is used to support front-end adaptation services for multiple data sources.

[0035] The disclosed embodiment supports asset metadata services, indicator services, offline services, real-time services, and front-end specification adaptation through a high-level management submodule, thereby providing unified data service capabilities.

[0036] In some optional implementations, the data query module includes:

[0037] SQL adaptation unit, SQL optimization unit, cross-table association unit, cache acceleration unit, query acceleration unit, multi-table query routing unit and multi-engine decision unit.

[0038] The disclosed embodiment provides a unified multi-source heterogeneous data query engine capability from the perspective of data consumption capability through a data query module, solves cross-engine federated query scenarios, and reduces query development costs for data consumption.

[0039] In some optional implementations, the data analysis module includes: an SQL query unit, an indicator data acquisition unit, a report dashboard unit, a multidimensional analysis unit, an attribution analysis unit, and a point-of-care analysis unit.

[0040] The disclosed embodiments provide unified data service capabilities from the perspective of data application, support data analysis scenarios and business development scenarios, and ensure the uniformity and stability of data service exports.

[0041] In some optional implementations, the data access module includes: MySQL database, Hive database, CK database, Holo database, Hbase database.

[0042] The data access module of the embodiment of the present disclosure supports various types of databases to meet the different business needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a structural block diagram of a data service system according to an embodiment of the present invention;

[0045] Figure 2 It is a schematic diagram of the implementation process of the data service system in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] The present disclosure provides a data service system. Figure 1 As shown, it includes: a data access module 10, a data processing module 11, a data management module 12, a data query module 13 and a data analysis module 14.

[0051] exist Figure 1 In the data access module 10, it is used to access various data sources.

[0052] Specifically, the multiple data sources include: data from external related departments, data from the internal business departments and transcription departments of the Emergency Management Bureau, public data on the social Internet, perception data, and data from other platforms. For example, the multiple data sources include data source A, data source B, data source C, and data source D. Therefore, the data access module can integrate multiple data sources from multiple different platforms. Figure 2 FIG. 1 is a schematic diagram of the implementation process of the data service system in the embodiment of the present disclosure. Figure 2 The business access module accesses a variety of data sources including: user portrait data, business analysis data, real-time statistical data, automatically pulled data, offline exported data, and compliance certification data.

[0053] The data access methods of the data access module mentioned above include database access methods, file access methods, interface call methods and data exchange methods. The data access module may also include a data exploration submodule, a data reading submodule and a data reconciliation submodule. The data exploration submodule is used to perform business exploration, access method exploration, field exploration, data set exploration, problem data exploration and data push on the source data. The data reading submodule is used to detect whether the data extracted from the source system or the data read from the specified location is consistent with the data definition. If there is inconsistency, stop access and re-explore and define the data; if there is consistency, perform further access, decrypt and decompress the data, generate a record ID that acts on the entire life cycle of the data, and convert the data into a character set format that meets the data processing requirements.

[0054] In some optional embodiments, Figure 1 In the example, the data access module 10 includes: MySQL database, Hive database, CK database, Holo database, and Hbase database.

[0055] Specifically, the data access module supports multiple different types of databases, and stores corresponding types of data through each type of database. Therefore, the data access module supports multiple types of databases to meet different business needs of users.

[0056] In some optional embodiments, Figure 1 In the data processing module 11, it is used to perform standard design, metadata management, data development, and data security processing on various data sources.

[0057] Due to the lack of standardization of data quality in the data production process, the data quality is uneven, there is a lack of control measures for data specifications, and management specifications are difficult to implement.

[0058] Therefore, in the embodiments of the present disclosure, a data processing module is used to perform specification design, metadata management, data development, and data security processing on various data sources.

[0059] In some optional embodiments, Figure 1 In the embodiment, the data processing module 11 includes: a specification design unit 110 , a metadata management unit 111 , a data development unit 112 , and a first security management unit 113 .

[0060] The specification design unit is used to carry out specification design for various data sources in accordance with the data warehouse specification requirements and data standard requirements; the metadata management unit is used to collect metadata and manage data lineage for various data sources; the data development unit is used to develop and process data integration, data cleaning, data conversion, data loading, and data synchronization for various data sources; the first security management unit is used to carry out security management and control of classification and grading of various data sources, sensitive data monitoring, and permission management.

[0061] Through the standardized design unit, multiple data sources are standardized and designed to ensure the integrity and standardization of the data. Through the metadata management unit, metadata management of multiple data sources is conducive to understanding the relationship between multiple data sources. Through the data security unit, multiple data sources are managed securely, which is conducive to ensuring the security of multiple data sources during production, consumption and data access control, and avoiding multiple data sources from leaking and violating regulations, which may bring serious legal and reputation risks to enterprises.

[0062] In some optional embodiments, Figure 1 In the data management module 12 , it includes: a grassroots management submodule 120 , a middle-level management submodule 121 and a high-level management submodule 122 ; the grassroots management submodule 120 includes: a metadata synchronization unit 1200 , an approval center unit 1201 and an authority center unit 1202 .

[0063] Specifically, the metadata synchronization unit is used to synchronize metadata for various data sources managed by the metadata management unit; the approval center unit is used to approve data for various data sources managed by the middle-level management submodule; and the authority center unit is used to interact with the first security management unit to process various data sources.

[0064] Specifically, the grassroots management submodule is equivalent to the basic layer of the data management module. The metadata synchronization unit synchronizes the metadata of the various data sources managed by the metadata management unit in the above-mentioned data processing module, thereby ensuring that the data descriptions between different components remain consistent and avoiding data conflicts and misunderstandings. The approval center unit is mainly used to approve the various data sources managed by the middle-level management submodule, thereby ensuring data quality, data compliance and consistency. The authority center unit is used to interact with the first security management unit to process various data sources, further enhance data security, and provide data access rights to authorized users.

[0065] In some optional embodiments, Figure 1The middle-level management submodule 121 includes: an API management unit 1210, a data modeling unit 1211, a data table management unit 1212, an indicator management unit 1213, a label management unit 1214, a governance workbench unit 1215, and a second security management unit 1216.

[0066] In some optional implementations, the API management unit is used to perform API construction, API services, SQL adaptation, query acceleration, call dashboards, interface monitoring, data source management, and resource management for multiple data sources.

[0067] Specifically, API, the full name of which is Application Programming Interface, uses an API management unit to perform API construction, API services, SQL adaptation, query acceleration, call dashboards, interface monitoring, data source management, and resource management on a variety of data sources, which is conducive to ensuring high availability and high stability of data services.

[0068] In some optional implementations, the data modeling unit is used to perform logical modeling, visual modeling, model management, and automated modeling on multiple data sources.

[0069] The data modeling unit in the disclosed embodiment supports various modeling methods and manages the models, which is beneficial to ensuring the consistency and accuracy of data during the production process.

[0070] In some optional implementations, the data table management unit is used to perform data retrieval, data table lineage management, table building tool management, and business isolation management on multiple data sources.

[0071] The data table management unit in the disclosed embodiment supports data retrieval, data table lineage management, table building tool management, and business isolation management, which is conducive to unified data management.

[0072] In some optional implementations, the indicator management unit is used to perform indicator dictionary management, indicator lineage management, indicator semantic registration, and indicator production management on multiple data sources.

[0073] Due to the lack of general data integration capabilities in the data development and production process, data integration relies on user code and display specification constraints, and data indicator inconsistency problems such as "same name but different meaning" and "same meaning but different name" frequently occur. Therefore, the embodiment of the present disclosure manages and controls the entire life cycle of data assets through an indicator management unit to solve the problems of inconsistent data management and inconsistent indicators.

[0074] In some optional implementations, the label management unit is used to perform label registration management, label lineage management, and label production management on multiple data sources.

[0075] The disclosed embodiment performs label registration management, label lineage management, and label production management on a variety of data sources through a label management unit, which is beneficial to ensure the accurate, unified, and safe use of labels and effectively improve the efficiency and quality of data management.

[0076] In some optional implementations, the governance workbench unit is used to perform business data health governance, resource usage analysis governance, governance reporting, and workbench governance on multiple data sources.

[0077] The disclosed embodiment uses a governance workbench unit to uniformly perform business data health governance, resource usage analysis governance, governance reporting and workbench governance on multiple data sources, thereby ensuring data quality.

[0078] In some optional implementations, the second security management unit is used to classify and grade various data sources, scan sensitive data, manage security policies, and send them to the approval center unit.

[0079] Therefore, the data service system in the embodiment of the present disclosure performs data management through the middle-level management submodule, can provide management capabilities for the entire life cycle of data, and ensure data consistency, data accuracy, and data security during the data production process.

[0080] In some optional implementations, the high-level management submodule includes: an asset metadata service unit, an indicator service unit, an offline service unit, a real-time service unit, and a front-end specification adaptation unit. The asset metadata service unit is used to provide services for various data sources according to table metadata, indicator metadata, or tag metadata, and to provide lineage services for various data sources; the indicator service unit is used to perform indicator query, indicator retrieval, materialized acceleration, pre-broadening, pre-aggregation, and indicator set query for various data sources; the offline service unit is used to support offline services for various data sources; the real-time service unit is used to support real-time services for various data sources; and the front-end specification adaptation unit is used to support front-end adaptation services for various data sources.

[0081] The disclosed embodiment provides a unified data service capability through a high-level management submodule to ensure high availability and high stability of data services.

[0082] In some optional implementations, the data query module includes: an SQL adaptation unit, an SQL optimization unit, a cross-table association unit, a cache acceleration unit, a query acceleration unit, a multi-table query routing unit and a multi-engine decision unit.

[0083] The SQL in the disclosed embodiments stands for Structured Query Language, which supports SQL adaptation, SQL optimization, cross-table association, cache acceleration, query acceleration, multi-table query routing and multi-engine decision-making through a data query module, thereby providing a unified multi-source data engine query capability and reducing the query development cost of data consumption.

[0084] In some optional implementations, the data service system in the embodiment of the present disclosure, the data analysis module includes: an SQL query unit, an indicator data acquisition unit, a report dashboard unit, a multidimensional analysis unit, an attribution analysis unit and a point of interest analysis unit.

[0085] The data analysis module in the disclosed embodiment supports SQL queries, indicator data acquisition, report dashboards, multidimensional analysis, attribution analysis, and point analysis, thereby providing unified data service capabilities, supporting data analysis scenarios and business development scenarios, ensuring the uniformity and stability of data service exports, and improving the efficiency of indicator usage in data analysis services.

[0086] Therefore, the data service system provided by the embodiment of the present disclosure includes: a data access module for accessing multiple data sources; a data processing module for performing standardized design, metadata management, data development, and data security processing on multiple data sources; a data management module for performing basic management, API management, data modeling, data governance, data table management, indicator management, label management, security management, and data services on multiple data sources processed by the data processing module; a data query module for querying multiple data sources managed by the data management module; and a data analysis module for analyzing multiple data sources queried by the data query module. Finally, the embodiment of the present disclosure forms a unified data management platform through the data access module, the data processing module, the data management module, the data query module, and the data analysis module, which can realize the control of the life cycle of multiple data assets to achieve the purpose of optimizing the uniformity, security, and standardization of data. That is, the data service system in the embodiment of the present disclosure can realize unified data service management and multi-heterogeneous data API service, achieve data standardization and unification, data sharing, and service capability sharing, solve the traditional interaction mode of data handling and inconsistent indicators, and enhance data security. The disclosed embodiment provides the ability to publish data from different sources and in different forms into standard services through unified data services, establishes self-service data sharing services, provides a global service view and a complete data sharing link, and has the ability to monitor the entire life cycle of data services. In addition, the data service system in the disclosed embodiment is developed using the cloud-native microservice concept, and has the characteristics of high concurrency, high availability, and scalability, making the data sharing process more efficient, more stable, and more secure.

[0087] The present disclosure embodiment Figure 2 During the implementation process, the data production end sends indicator definition data, data production data, model acceleration data, quality monitoring data, and API management data to the data management platform. The data consumer end uses data, searches for models, indicators, interfaces, and monitors services through the data management module of the data service system. The data management module is connected to the business access module.

[0088] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data service system, characterized in that: The system comprises: Data access module, used to access various data sources; Data processing module, used for standard design, metadata management, data development, and data security processing of various data sources; A data management module, which is used to perform basic management, API management, data modeling, data governance, data table management, indicator management, label management, security management and data services on various data sources processed by the data processing module; A data query module, used to query the multiple data sources managed by the data management module; The data analysis module is used to analyze the multiple data sources after being queried by the data query module.

2. The data service system according to claim 1, characterized in that: The data processing module includes: a specification design unit, a metadata management unit, a data development unit, and a first security management unit; The specification design unit is used to perform specification design on the various data sources in accordance with the data warehouse specification requirements and data standard requirements; the metadata management unit is used to collect metadata and manage data lineage for the various data sources; the data development unit is used to perform development processing for data integration, data cleaning, data conversion, data loading, and data synchronization for the various data sources; the first security management unit is used to perform security management and control for classification and grading of the various data sources, sensitive data monitoring, and permission management.

3. The data service system according to claim 2, characterized in that: The data management module includes: a grassroots management submodule, a middle-level management submodule and a high-level management submodule; the grassroots management submodule includes: a metadata synchronization unit, an approval center unit and an authority center unit.

4. The data service system according to claim 3, characterized in that: The metadata synchronization unit is used to synchronize metadata for the various data sources managed by the metadata management unit; the approval center unit is used to perform data approval for the various data sources managed by the middle-level management submodule; and the authority center unit is used to interact with the first security management unit to process the various data sources.

5. The data service system according to claim 3, characterized in that: The middle-level management submodule includes: an API management unit, a data modeling unit, a data table management unit, an indicator management unit, a label management unit, a governance workbench unit, and a second security management unit.

6. The data service system according to claim 5, characterized in that: The API management unit is used to perform API construction, API services, SQL adaptation, query acceleration, call dashboard, interface monitoring, data source management and resource management for various data sources; The data modeling unit is used to perform logical modeling, visual modeling, model management, and automated modeling on various data sources; The data table management unit is used to perform data retrieval, data table lineage management, table building tool management, and business isolation management on the multiple data sources; The indicator management unit is used to manage indicator dictionaries, indicator lineage, indicator semantics registration, and indicator production for various data sources; The label management unit is used to manage label registration, label lineage and label production for various data sources; The governance workbench unit is used to perform business data health governance, resource usage analysis governance, governance reporting and workbench governance on various data sources; The second security management unit is used to classify and grade the various data sources, scan sensitive data, manage security policies, and send them to the approval center unit.

7. The data service system according to claim 3, characterized in that: The high-level management submodules include: asset metadata service unit, indicator service unit, offline service unit, real-time service unit, and front-end specification adaptation unit. The asset metadata service unit is used to provide services for the various data sources according to table metadata, indicator metadata, or tag metadata, and to provide lineage services for the various data sources; The indicator service unit is used to perform indicator query, indicator retrieval, materialization acceleration, pre-broadening, pre-aggregation, and indicator set query on multiple data sources; The offline service unit is used to support offline services for the multiple data sources; The real-time service unit is used to support real-time services for the multiple data sources; The front-end specification adaptation unit is used to support front-end adaptation services for the multiple data sources.

8. The data service system according to claim 1, characterized in that: The data query module includes: SQL adaptation unit, SQL optimization unit, cross-table association unit, cache acceleration unit, query acceleration unit, multi-table query routing unit and multi-engine decision unit.

9. The data service system according to claim 1, characterized in that: The data analysis module includes: an SQL query unit, an indicator data acquisition unit, a report dashboard unit, a multidimensional analysis unit, an attribution analysis unit and a buried point analysis unit.

10. The data service system according to claim 1, characterized in that: The data access module includes: MySQL database, Hive database, CK database, Holo database, and Hbase database.

Images