Data asset management system

Abstract

The invention relates to the technical field of data asset processing, in particular to a data asset management system, which comprises a front acquisition database for acquiring to-be-stored data from an external data source; the security gateway is used for verifying the input data to be stored and then generating a data input record; the data lake is used for storing the to-be-stored data; the access client sends a data access request to the data lake through the security gateway; and the data lake returns request data to the access client. Aiming at the problems that a data asset management system in the prior art lacks a management and control process for data access and access requests, and ineffective sources and illegal access are difficult to avoid, a security gateway is arranged to verify input and output data, corresponding record information is generated, and then the data are stored in a data lake or returned to a client, so that the data asset management system is convenient to use. The problems are avoided, meanwhile, a front acquisition database is arranged on the acquisition side to improve the acquisition efficiency, and low data acquisition efficiency and interruption caused by a security gateway are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data asset processing, and in particular to a data asset management system. Background Art

[0002] Data assets refer to data resources legally owned or controlled by an enterprise, which are recorded electronically or otherwise, such as text, images, voice, video, web pages, databases, sensor signals and other structured or unstructured data. Data assets can bring direct or indirect economic and social benefits to enterprises. Types of data assets include but are not limited to customer information, transaction records, market research data, internal operational data, etc. Through effective management and analysis, these data can be transformed into competitive advantages and business value for enterprises.

[0003] In the prior art, there are storage systems designed for data assets. For example, Chinese patent CN202211438126.8 discloses a data asset management system, including: a data map module, which is used to collect and store various metadata and the popularity information and freshness information corresponding to the metadata, add a popularity identifier to the metadata according to the popularity information, call the target SQL statement to analyze the metadata to obtain the blood relationship of the metadata and store it, and receive the user's query operation on the metadata through a unified interface; a data standard module, which is used to perform data standard verification on the metadata stored in the data map module and output the standard verification result; and a data quality module, which is used to perform data quality inspection on the metadata stored in the data map module and output the quality inspection result. Compared with the prior art, it avoids the problem of unified monitoring of all data, inability to distinguish data quality, and low monitoring effectiveness caused by equal monitoring or utilization of data of different qualities.

[0004] However, during the actual implementation process, the inventors found that this type of technical solution usually focuses on the storage and management of the collected data, and does not control the inflow and outflow of data. As a result, problems such as the inability to verify the source of data and illegal access may easily occur during use. Summary of the invention

[0005] In view of the above problems existing in the prior art, a data asset management system is now provided.

[0006] The specific technical solutions are as follows:

[0007] A data asset management system, comprising:

[0008] A front-end acquisition database, the front-end acquisition database is connected to a plurality of external data sources, and the front-end acquisition database obtains data to be stored from the external data sources;

[0009] A security gateway, the security gateway is connected to the front-end acquisition database, and the security gateway verifies the input data to be stored and generates a data input record;

[0010] A data lake, the data lake being connected to the security gateway, and the data lake storing the data to be stored after the security gateway generates the data input record;

[0011] An access client, the access client is connected to the security gateway, and the access client sends a data access request to the data lake via the security gateway;

[0012] The data lake returns the requested data to the access client via the security gateway according to the data access request.

[0013] On the other hand, the pre-collection database includes:

[0014] An interface docking module, the interface docking module is connected to the external data source according to the corresponding data interface;

[0015] A data acquisition module, the data acquisition module is connected to the interface docking module, and the data acquisition module obtains raw data from the external data source according to the data interface;

[0016] A data preprocessing module, wherein the data preprocessing module is connected to the data acquisition module, and the data preprocessing module preprocesses the original data to obtain the data to be stored.

[0017] On the other hand, the data acquisition module comprises:

[0018] A data query module, wherein the data query module initiates a query instruction to the external data source through the data interface to obtain the original data;

[0019] A data subscription module, wherein the data subscription module initiates a subscription request to the external data source through the data interface, and the external data source returns the original data to the data acquisition module through a message queue.

[0020] On the other hand, the security gateway comprises:

[0021] A connection verification module, which verifies the front-end acquisition database or the access client requesting access to generate verification information;

[0022] A record generation module, the record generation module is connected to the connection verification module, and when the verification information is passed, the record generation module generates a data input record corresponding to the front-end acquisition database or generates a data usage record corresponding to the access client.

[0023] On the other hand, the security gateway also includes:

[0024] A billing module, the billing module is connected to the record generation module, and the billing module generates a settlement sheet according to the data usage record and the data input record.

[0025] On the other hand, the data lake includes:

[0026] A data storage module, the data storage module receives the data to be stored and stores it;

[0027] A data analysis module, wherein the data analysis module is connected to the data storage module, and the data analysis module processes the data to be stored according to the analysis algorithm configured by the access client.

[0028] On the other hand, the data storage module includes a plurality of partition storage units, and each of the partition storage units stores the data to be stored respectively.

[0029] On the other hand, the data lake also includes:

[0030] A data quality assessment module, wherein the data quality assessment module is connected to the data storage module, and the data quality assessment module assesses the data quality of the data to be stored and generates a quality assessment result.

[0031] The above technical solution has the following advantages or beneficial effects:

[0032] In view of the fact that the data asset management system in the prior art lacks the control process for data access and access requests, it is difficult to avoid the problems of invalid sources and illegal access. In this embodiment, a security gateway is set to verify the input and output data and generate corresponding record information. The data is then stored in the data lake or returned to the client, avoiding the above problems. At the same time, a front-end collection database is also set for the collection side to improve the collection efficiency and avoid the security gateway causing low data collection efficiency and interruption. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The embodiments of the present invention will be described more fully with reference to the attached drawings, which are provided for illustration and description only and are not intended to limit the scope of the present invention.

[0034] Figure 1 It is an overall schematic diagram of an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of a pre-collection database in an embodiment of the present invention;

[0036] Figure 3This is a schematic diagram of a data acquisition module in an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of a security gateway in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of a billing module in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram of a data lake in an embodiment of the present invention;

[0040] Figure 7 Schematic diagram of a data quality assessment module in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0043] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0044] The present invention comprises:

[0045] A data asset management system, such as Figure 1 As shown, including:

[0046] A front-end acquisition database 1, the front-end acquisition database 1 is connected to multiple external data sources 1A, and the front-end acquisition database 1 obtains data to be stored from the external data sources;

[0047] The security gateway 2 is connected to the front-end acquisition database 1, and the security gateway 2 verifies the input data to be stored and generates a data input record;

[0048] Data Lake 3, Data Lake 3 is connected to Security Gateway 2, and Data Lake 3 stores the data to be stored after Security Gateway 2 generates data input records;

[0049] Access client 4, access client 4 connects to security gateway 2, and access client 4 sends a data access request to the data lake via security gateway 2;

[0050] The data lake 3 returns the requested data to the access client 4 via the security gateway 2 according to the data access request.

[0051] Specifically, the data asset management system in the prior art lacks a control process for data access and access requests, making it difficult to avoid problems of invalid sources and illegal access. In this embodiment, a security gateway is provided to verify input and output data and generate corresponding record information, and then the data is stored in the data lake or returned to the client, thereby avoiding the above problems. At the same time, a front-end collection database is also provided on the collection side to improve collection efficiency and avoid inefficient data collection and interruptions caused by the security gateway.

[0052] In one embodiment, Figure 2 As shown, the pre-collection database 1 includes:

[0053] The interface docking module 11 is connected to an external data source according to a corresponding data interface;

[0054] The data acquisition module 12 is connected to the interface docking module 11, and the data acquisition module 12 obtains raw data from an external data source according to the data interface;

[0055] The data preprocessing module 13 is connected to the data acquisition module 12 , and the data preprocessing module 13 preprocesses the original data to obtain data to be stored.

[0056] Specifically, in order to achieve better data collection effects, in this embodiment, for multiple external data sources 1A, multiple corresponding data interfaces are pre-configured in the interface docking module 11. Each data interface is designed according to the communication protocol adopted by the external data source 1A, and the relevant API and message format are agreed upon.

[0057] Then, the data acquisition module 12 obtains the original data from the external data source 1A according to the data interface. Since the original data contains corresponding verification information, message format, etc., after passing the verification, the data preprocessing module 13 preprocesses the original data, mainly including identifying the corresponding data fields, format conversion of the data fields, normalization, removal of redundant values, and filling of default values ​​with zeros, and finally obtains the corresponding data to be stored.

[0058] At the same time, the data preprocessing process also includes extracting the verification information in the original data and sending it to the security gateway 2.

[0059] In one embodiment, Figure 3 As shown, the data acquisition module 12 includes:

[0060] A data query module 121, which initiates a query instruction to an external data source through a data interface to obtain raw data;

[0061] The data subscription module 122 initiates a subscription request to an external data source through a data interface, and the external data source returns the original data to the data acquisition module through a message queue.

[0062] Specifically, in order to achieve better data collection effect, in this embodiment, a plurality of collection methods are respectively set in the data collection module 12 .

[0063] The data query module 121 is for the case where the external data source 1A is the business database of the production system. It creates a query tool and triggers it according to the corresponding timing, so as to obtain the data in the business database as the original data.

[0064] In addition, the data subscription module 122 is generated for an external data source with a message queue for active push. It initiates a subscription to the message queue used for data publishing in the external data source. When the data in the message queue is updated, it will be pushed to the data subscription module 122 to obtain the data.

[0065] Through the above settings, the data collection module 12 can achieve better collection effects for various data sources.

[0066] In one embodiment, Figure 4 As shown, the security gateway 2 includes:

[0067] A connection verification module 21, which verifies the front-end acquisition database or access client that requests access to generate verification information;

[0068] The record generation module 22 is connected to the connection verification module 21. When the verification information is passed, the record generation module 22 generates a data input record corresponding to the front-end acquisition database or generates a data use record corresponding to the access client.

[0069] Specifically, in order to achieve a better access control function, in this embodiment, the connection verification module 21 verifies the front-end acquisition database or access client requesting access to generate verification information.

[0070] Specifically, when the data source generates corresponding data, it will go through two verification processes, including the verification process between the front-end collection database and the data source, and the verification process of the front-end collection database 1 by the security gateway 2. On the front-end collection database side, the data source information entered during the configuration process will first be verified to determine whether it is data generated by a pre-configured data source, and only when it is determined to be a pre-configured data source will it be cached.

[0071] Then, the security gateway 2 verifies the information sent by the front-end acquisition database 1 and the external data source, and stores them when they match. At the same time, the record generation module 22 generates data input records corresponding to the front-end acquisition database or generates data usage records corresponding to the access client.

[0072] Similarly, when the access client 4 accesses, the security gateway 2 verifies the key of the access client 4, and forwards the data request only after the verification passes, and generates a data usage record.

[0073] In one embodiment, Figure 5 As shown, the security gateway 2 also includes:

[0074] The billing module 23 is connected to the record generating module 22, and the billing module 23 generates a settlement sheet according to the data usage record and the data input record.

[0075] Specifically, in order to achieve a better asset management effect, in this embodiment, a billing module 23 is also configured in the security gateway 2, and the record generation module 22 is connected through the billing module 23. When data is stored or used, the billing module 23 generates a settlement sheet according to the data use record and the data input record, and the settlement sheet is used to push it to the financial system or access the client to settle the data usage fee.

[0076] In one embodiment, Figure 6 As shown, the data lake 3 includes:

[0077] A data storage module 31, the data storage module 31 receives data to be stored and stores it;

[0078] The data analysis module 32 is connected to the data storage module 31 , and processes the data to be stored according to the analysis algorithm configured by the access client 4 .

[0079] Specifically, in order to achieve better data utilization effects, in this embodiment, a data storage module 31 is provided in the data lake 3 for effectively storing the data to be stored in the warehouse.

[0080] At the same time, a data analysis module 32 is also configured. The data analysis module 32 is configured with a corresponding analysis algorithm according to the requirements of the access client 4. When the data to be stored is generated, the data analysis module 32 processes the data according to the configured algorithm, and then stores it as new data to be stored, so that the access client 4 can retrieve it.

[0081] In one embodiment, the data storage module 31 includes a plurality of partition storage units, each of which stores the data to be stored respectively.

[0082] Specifically, in order to achieve better access efficiency, in this embodiment, a plurality of partition storage units are provided in the data storage module 31. By providing the partition storage units to store data in parallel, better access efficiency can be achieved.

[0083] In one embodiment, Figure 7 As shown, the data lake 3 also includes:

[0084] The data quality assessment module 33 is connected to the data storage module 31 . The data quality assessment module 33 assesses the data quality of the data to be stored and generates a quality assessment result.

[0085] Specifically, in order to achieve better data quality, in this embodiment, a data quality assessment module 33 is also provided to connect to the data storage module 31. The data quality assessment module 33 assesses the data quality of the data to be stored and generates a quality assessment result. When the data quality is too low, the data processing algorithm is reconfigured.

[0086] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A data asset management system, characterized in that: include: A front-end acquisition database, the front-end acquisition database is connected to a plurality of external data sources, and the front-end acquisition database obtains data to be stored from the external data sources; A security gateway, the security gateway is connected to the front-end acquisition database, and the security gateway verifies the input data to be stored and generates a data input record; A data lake, the data lake being connected to the security gateway, and the data lake storing the data to be stored after the security gateway generates the data input record; An access client, the access client is connected to the security gateway, and the access client sends a data access request to the data lake via the security gateway; The data lake returns the requested data to the access client via the security gateway according to the data access request.

2. The data asset management system according to claim 1, characterized in that: The pre-collection database includes: An interface docking module, the interface docking module is connected to the external data source according to the corresponding data interface; A data acquisition module, the data acquisition module is connected to the interface docking module, and the data acquisition module obtains raw data from the external data source according to the data interface; A data preprocessing module, wherein the data preprocessing module is connected to the data acquisition module, and the data preprocessing module preprocesses the original data to obtain the data to be stored.

3. The data asset management system according to claim 2, characterized in that: The data acquisition module comprises: A data query module, wherein the data query module initiates a query instruction to the external data source through the data interface to obtain the original data; A data subscription module, wherein the data subscription module initiates a subscription request to the external data source through the data interface, and the external data source returns the original data to the data acquisition module through a message queue.

4. The data asset management system according to claim 1, characterized in that: The security gateway comprises: A connection verification module, which verifies the front-end acquisition database or the access client requesting access to generate verification information; A record generation module, the record generation module is connected to the connection verification module, and when the verification information is passed, the record generation module generates a data input record corresponding to the front-end acquisition database or generates a data usage record corresponding to the access client.

5. The data asset management system according to claim 4, characterized in that: The security gateway also includes: A billing module, the billing module is connected to the record generation module, and the billing module generates a settlement sheet according to the data usage record and the data input record.

6. The data asset management system according to claim 1, characterized in that: The data lake includes: A data storage module, the data storage module receives the data to be stored and stores it; A data analysis module, wherein the data analysis module is connected to the data storage module, and the data analysis module processes the data to be stored according to the analysis algorithm configured by the access client.

7. The data asset management system according to claim 6, characterized in that: The data storage module includes a plurality of partition storage units, and each of the partition storage units stores the data to be stored respectively.

8. The data asset management system according to claim 6, characterized in that: The data lake also includes: A data quality assessment module, wherein the data quality assessment module is connected to the data storage module, and the data quality assessment module assesses the data quality of the data to be stored and generates a quality assessment result.

Images