Enterprise data sharing platform

Abstract

The invention discloses an enterprise data sharing platform comprising a data source module used for providing a data source; the data acquisition module is used for performing data acquisition of the multi-source heterogeneous system based on the data source module by using an established data interface and an ETL mode; the data standard module is used for setting a standard set, a user-defined standard template and a standard attribute, and the standard set refers to the user-defined standard template and defines a data standard under the standard set; the data quality module is used for monitoring task execution conditions according to set task scheduling, feeding back upstream data collected for failed tasks, and setting a rule set and a verification rule to realize data quality verification; and the data calculation module is used for performing data extraction, data cleaning and data calculation on various database data qualified in data quality verification to obtain standard data meeting the requirements of a business party, and providing data service to the outside through an interface. Therefore, the method can be widely applied to enterprise informatization construction.

Description

Technical Field

[0001] The present invention relates to an enterprise data sharing platform for solving existing enterprise data isolated islands, and relates to the field of data sharing. Background Art

[0002] In the construction of enterprise informatization, each business party and each branch and subsidiary establishes its own informatization system and adopts various technology stacks, which makes data inconvenient to circulate and share, and there are inconsistencies in data standards and data quality. The enterprise has no unified data export caliber, cannot gather unified data assets, and cannot maximize the value of data elements.

[0003] Existing technologies often connect interfaces between systems based on actual business needs to achieve inter-system communication, but there are situations where repeated connection and development occur, and data is not circulated or shared. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, in view of the above problems, the present invention aims to provide an enterprise data sharing platform that can achieve unified management, unified standards and shared interaction of data.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] An enterprise data sharing platform, characterized by comprising:

[0007] Data source module, used to provide data source;

[0008] A data acquisition module is used to collect data from multi-source heterogeneous systems based on the data source module using the established data interface and ETL method;

[0009] The data standard module is used to set standard sets, custom standard templates, and standard attributes. The standard set references the custom standard template and defines the data standards under the standard set.

[0010] The data quality module is used to monitor task execution according to the set task scheduling and provide feedback on upstream data collected by failed tasks. At the same time, it sets rule sets and verification rules to implement data quality verification;

[0011] The data calculation module is used to extract, clean and calculate data from various databases that have passed data quality verification, obtain standard data that meets the requirements of the business party, and provide data sharing services to the outside world through interfaces.

[0012] In some possible implementations, the data source comes from various business systems or third-party databases.

[0013] In some possible implementations, the data acquisition module builds a data interface by using an API interface or by using a synchronization tool sqoop to build a data interface.

[0014] In some possible implementations, the data standard is a specification that defines the data under a field. When the accessed data is raw data, a data standard is associated with a field of the raw data, that is, there is a standard definition for the field; the data field stored after calculation is also associated with a data standard, that is, the data standard can be associated with any data field, that is, the association relationship between a data field and a data standard is bound.

[0015] In some possible implementations, the verification rules include checking for null values, value ranges, or repeatability. After specifying a library, table, or field, corresponding SQL statements are used to monitor null values, value ranges, and repeatability, and the SQL statements are run through quality tasks.

[0016] In some possible implementations, the implementation process of the data quality module is as follows: binding data standards to the collected data, binding quality rules to specific data fields of the collected library tables, and performing quality monitoring tasks; or binding data standards to the data to execute corresponding quality rules and perform quality monitoring tasks; data that does not pass the quality test will be corrected by the source business party or cleaned to correct data.

[0017] In some possible implementations, the data collection module further includes defining data metadata, where the metadata information of the data includes data classification, data name, and data ownership.

[0018] In some possible implementations, the data calculation module uses Dinky, which implements data extraction, cleaning, and calculation through FlinkSQL.

[0019] The present invention adopts the above technical solution, which has the following characteristics:

[0020] 1. The enterprise data sharing platform of the present invention performs unified and standardized steps such as cleaning and quality verification on the data of each system, solving the existing enterprise data silos, so that data can be retained in a unified and standardized manner on the data sharing platform, and can provide unified data services to the outside world. The data calculation module can be opened to users to realize business data processing independently.

[0021] 2. The enterprise data sharing platform of the present invention is driven by business needs, so data access and processing are more business-oriented, can generate high actual value, occupy less resources, and have high data quality.

[0022] 3. While retaining the original system construction, the present invention designs an enterprise data sharing platform to achieve unified data management, unified standards and shared interaction, providing users with a place to obtain data and data services.

[0023] 4. The data acquisition module of the present invention defines metadata, and the data lineage can trace the data source and data process, and unify the data version to provide reliable data.

[0024] 5. The present invention establishes an enterprise data sharing platform based on the data islands of multiple independent business information systems to solve problems such as data collection, data standards, data quality, data calculation and data sharing of various systems.

[0025] In summary, the present invention can be widely applied to enterprise informatization construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0027] Figure 1 Schematic diagram of the principles of the enterprise data sharing platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0029] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates otherwise, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0030] For ease of description, spatially relative terms may be used herein to describe the relationship of one element or feature relative to another element or feature as shown in the figures, such as "inside," "outside," "inner side," "outer side," "lower," "upper," etc. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures.

[0031] Since the existing technology often performs interface docking between various systems according to actual business needs to achieve inter-system communication, there are situations where repeated docking, development, and data are not circulated or shared. The enterprise data sharing platform provided by the present invention includes: a data source module for providing a data source; a data acquisition module for using the built data interface and ETL method to perform data acquisition of multi-source heterogeneous systems based on the data source module; a data standard module for setting a standard set, a custom standard template and a standard attribute, the standard set references the custom standard template, and defines the data standard under the standard set; a data quality module for monitoring the execution of tasks according to the set task scheduling, and providing feedback on the upstream data collected by failed tasks, and at the same time, setting a rule set and verification rules to achieve data quality verification; a data calculation module for extracting, cleaning and calculating data from various database data that have passed the data quality verification, obtaining standard data that meets the requirements of the business party, and providing data services to the outside through the interface. Therefore, the present invention can solve the existing enterprise data islands, so that data can be uniformly and standardizedly retained on the data sharing platform, and can provide data services to the outside in a unified manner.

[0032] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0033] like Figure 1 As shown, the enterprise data sharing platform provided in this embodiment includes:

[0034] The data source module is used to provide a data source, where the data source can come from various business systems or third-party databases.

[0035] The data acquisition module is used to collect data from multi-source heterogeneous systems using the established data interface and ETL methods.

[0036] The data standards module is used to set standard sets, customize standard templates, and set standard attributes. Standard sets reference customized standard templates and define the data standards within these standard sets. The data standards module can be linked with the data quality module. By defining standards and associating them with the data quality module's verification rules, data quality verification for these standards can be implemented and managed. The data standards module can adopt a microservices architecture.

[0037] The data quality module is used to monitor task execution according to the set task schedule, provide feedback on upstream data corresponding to failed tasks, and set rule sets and verification rules to implement data quality verification. The data quality module can adopt a microservice architecture.

[0038] The data calculation module is used to extract, clean and calculate data from various databases that have passed data quality verification, obtain data indicators and data that meet the business party's standard requirements, and provide data sharing services to the outside world through interfaces.

[0039] In a preferred embodiment of the present invention, the data acquisition module supports the following methods for data acquisition: 1) collecting data in the form of an interface; 2) directly collecting data in the form of a library table of a given data source; 3) collecting data from multiple types of data sources in the form of ETL. These methods are parallel methods.

[0040] Furthermore, the data interface is constructed by using an API interface or a synchronization tool such as sqoop. This is taken as an example and is not limited thereto.

[0041] In a preferred embodiment of the present invention, a multi-source heterogeneous system refers to different business systems with different data sources, that is, different databases used, such as MySQL, Oracle or Mongo, etc., which can be connected to multiple types of data sources to collect data from multiple types of sources.

[0042] Furthermore, using the established data interface and ETL method to collect data from multi-source heterogeneous systems means being able to directly connect to its database tables for offline data synchronization, thus realizing multi-source heterogeneous data collection. Among them, the data interface is a restful data transmission mode. The business system transmits data through the interface according to the interface specification. After the system receives the data, it parses the json data according to the pattern. This is not restricted by the database type. All types of databases transmit data through the interface as json strings, which can be parsed. Synchronization tools such as sqoop also support various types of database tables (such as mysql, oracle, etc.). ETL tools can also collect data. For example, dinky can also collect data from specified fields of database tables during the ETL process. Data collection can be done by ETL based on collection, that is, collecting specified fields of database tables, or collecting field data from the original database table to new fields in the new database table.

[0043] Furthermore, data collection can also realize timed data collection. The scheduling platform can schedule dinky component files or shell scripts to realize timed data collection.

[0044] In a preferred embodiment of the present invention, the data standard refers to the specification for defining the data under a field, for example, the type is a string type, the length is 128, and it cannot be empty. These are the basic attribute definitions for this field.

[0045] When the data being accessed is raw data, a data standard is associated with a field of the raw data, meaning that a standard definition exists for the field.

[0046] The data fields stored after calculation can also be associated with a certain data standard, that is, the data standard can be associated with any data field, that is, the association relationship between a data field and a data standard is bound.

[0047] Furthermore, the standard attributes, standard sets, and standard templates provided in this embodiment are shown in Tables 1 to 3, which are examples and are not limited thereto.

[0048] Table 1 Examples of standard attributes

[0049]

[0050] Table 2 Example of standard set

[0051]

[0052] Table 3 Standard template examples

[0053] Standard template name Standard template properties Communication number standard template 1 Standard name, type, length, description, and standard content requirements

[0054] In a preferred embodiment of the present invention, the data quality module sets a rule set and verification rules, wherein the verification rules include null value, value range, repeatability check, etc. After specifying the library, table, and field, the corresponding SQL is used to perform null value, value range, and repeatability monitoring, and the above SQL is run through the quality task.

[0055] Furthermore, the data quality module also includes binding data standards for the collected data, binding quality rules to specific data fields of the collected library tables, and performing quality monitoring tasks; or executing corresponding quality rules on the standard-bound data to perform quality monitoring tasks; data that does not pass the quality test will be corrected by the source business party or cleaned to correct data.

[0056] In a preferred embodiment of the present invention, the data acquisition module also includes defining data metadata. The metadata information of the data includes data classification, data name, and data ownership, etc. The purpose is to define some attributes of the collected data. For example, if a batch of data is classified as top secret, the data will be securely managed in accordance with top secret requirements in subsequent data use.

[0057] In a preferred embodiment of the present invention, the data calculation module uses Dinky to implement data extraction, cleaning, and calculation through FlinkSQL.

[0058] Furthermore, Dinky's FlinkSQL development is done through the registration center - adding a data source. After selecting the data table to be processed, SQL (Flink DDL) can be generated, which is used to connect to the data source table in FlinkSQL for subsequent SQL data processing. According to the needs, multi-table data merging, aggregation, calculation and other operations are calculated through FlinkSQL. First, configure the Connector in the Flink component and connect in FlinkSQL mode (Flink DDL). After configuration, write to the target database or table in insert into select mode. For complex business logic, you can use FlinkJar mode to write the logic program in the development language, compile it into a jar file, and execute it on the Flink cluster. Dinky supports Dolphin scheduling. After completing the data processing task (including FlinkSQL or FlinkJar), ​​the task is launched and pushed to the Dolphin scheduling end. After configuring the timing parameters, the timing scheduling link is completed.

[0059] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In the description of this specification, the reference terms "a preferred embodiment", "further", "specifically", "in the present embodiment", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An enterprise data sharing platform, characterized by: include: Data source module, used to provide data source; A data acquisition module is used to collect data from multi-source heterogeneous systems based on the data source module using the established data interface and ETL method; The data standard module is used to set standard sets, custom standard templates, and standard attributes. The standard set references the custom standard template and defines the data standards under the standard set. The data quality module is used to monitor task execution according to the set task scheduling and provide feedback on upstream data collected by failed tasks. At the same time, it sets rule sets and verification rules to implement data quality verification; The data calculation module is used to extract, clean and calculate data from various databases that have passed data quality verification, obtain standard data that meets the requirements of the business party, and provide data sharing services to the outside world through interfaces.

2. The enterprise data sharing platform according to claim 1, characterized in that: The data sources come from various business systems or third-party databases.

3. The enterprise data sharing platform according to claim 1, characterized in that: The data acquisition module builds a data interface by using an API interface or a synchronization tool sqoop.

4. The enterprise data sharing platform according to claim 1, characterized in that: The data standard is a specification that defines the data under a field. When the accessed data is raw data, a data standard is associated with a field of the raw data, that is, there is a standard definition for the field; a data field stored after calculation is also associated with a data standard, that is, the data standard can be associated with any data field, that is, the association relationship between a data field and a data standard is bound.

5. The enterprise data sharing platform according to claim 1, characterized in that: The verification rules include null value, value range or repeatability check. After specifying the library, table and field, the corresponding SQL is used to monitor null value, value range and repeatability, and the SQL is run through the quality task.

6. The enterprise data sharing platform according to claim 5, characterized in that: The implementation process of the data quality module is as follows: for the collected data, the data standards are bound, the quality rules are bound to the specific data fields of the collected database table, and the quality monitoring task is performed; or the data standards are bound to the data and the corresponding quality rules are executed to perform the quality monitoring task; Data that does not pass the quality test will be corrected or cleaned by the source business party to become correct data.

7. The enterprise data sharing platform according to claim 1, characterized in that: The data collection module also includes defining data metadata, and the metadata information of the data includes data classification, data name and data ownership.

8. The enterprise data sharing platform according to claim 1, characterized in that: The data calculation module uses Dinky, which implements data extraction, cleaning and calculation through FlinkSQL.

Images