Comprehensive data intelligent processing system

Abstract

The invention discloses a comprehensive data intelligent processing system, which comprises a data storage and management module, a data analysis module, a resource calculation module and a data blood relationship analysis module, the data storage and management module is used for collecting data and storing and managing the collected data; the data analysis module is used for analyzing and processing the stored data; the resource calculation module is used for calculating an optimal solution of data resource calling; and the data consanguinity analysis module is used for analyzing circulation and change of data. The invention aims to realize unified storage and management of data by integrating the advantages of a data lake and a data warehouse; meanwhile, a storage and calculation separation technology and a deep learning algorithm are utilized, the data processing flow is optimized, and the processing efficiency and accuracy are improved; besides, through an enhanced blood relationship analysis function, accurate tracking of data circulation and change is realized, and deeper data insight and decision support is provided for enterprises.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a comprehensive data intelligent processing system. Background Art

[0002] In the existing technology, in the field of data processing and analysis, although there are many data processing frameworks and algorithms, these technologies often face the problems of data silos, low processing efficiency, and difficulty in effectively supporting the unified processing of complex heterogeneous data. For example, although traditional data warehouse technology can store structured data, it is unable to handle semi-structured and unstructured data; and although the data lake solution provides a wider range of data storage capabilities, it still needs to improve data management and analysis efficiency. In addition, the bloodline analysis function in the existing technology is usually not flexible and intelligent enough, and it is impossible to efficiently track data flow and changes. These problems limit the full mining and utilization of data value. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a comprehensive data intelligent processing system, including a data storage and management module, a data analysis module, and also includes: a resource calculation module, and a data lineage analysis module; the data storage and management module is used to collect data, and store and manage the collected data; the data analysis module is used to analyze and process the stored data; the resource calculation module is used to calculate the optimal solution for data resource calls; the data lineage analysis module is used to analyze the flow and changes of data.

[0004] Furthermore, the data flow and change in the data lineage analysis module mainly include the following steps: A1. Create a data lineage analysis task; A2. Sort and associate the data and store the sorted and associated data; A3. Visualize the analyzed data lineage relationship.

[0005] Furthermore, in the step A2, the storage form of storing the sorted and associated data is specifically a graph database, wherein data tables or data fields are used as graph database nodes, and the relationships between data tables or the relationships between data fields are used as graph database relationships.

[0006] Furthermore, the visualized data lineage relationship in step A3 specifically includes: upstream and downstream hierarchical information, forward tracing back to source information, backward tracing back to data information, and relationship map.

[0007] Furthermore, the data collected in the data storage and management module is mainly collected through the API interface, and the API interface specifically includes: data middle platform metadata API interface, data middle platform business data API interface.

[0008] Furthermore, the management of the collected data in the data storage and management module specifically includes: project management, cluster management, and engine management; wherein the engines managed in the engine management specifically include: Hadoop cluster, Oracle, and GP.

[0009] Furthermore, the overall system adopts a computing and storage separation approach, separating data storage from resource computing and storing data on multiple nodes.

[0010] The present invention provides a comprehensive data intelligent processing system, which has the following beneficial effects: The present invention aims to realize unified storage and management of data by integrating the advantages of data lake and data warehouse; at the same time, it uses storage and computing separation technology and deep learning algorithms to optimize data processing procedures and improve processing efficiency and accuracy; in addition, through enhanced lineage analysis functions, it can achieve accurate tracking of data flow and changes, providing enterprises with deeper data insights and decision-making support. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0012] Figure 1 The system flow chart provided by the present invention. DETAILED DESCRIPTION

[0013] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0014] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.

[0015] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0016] like Figure 1As shown, the present invention provides a comprehensive data intelligent processing system, including a data storage and management module, a data analysis module, and also includes: a resource calculation module, and a data lineage analysis module; the data storage and management module is used to collect data, and store and manage the collected data; the data analysis module is used to analyze and process the stored data; the resource calculation module is used to calculate the optimal solution for data resource calls; the data lineage analysis module is used to analyze the flow and changes of data.

[0017] The data flow and change in the data lineage analysis module mainly include the following steps: A1. Create a data lineage analysis task: Through metadata collection, you can create a task to collect metadata through two methods: DB directly and API interface, so as to conduct effective lineage analysis later; A2. Sort and associate the data and store the sorted and associated data; A3. Visualize the analyzed data lineage relationship. Data lineage analysis is one of the important applications of metadata management, involving the relationship between tables, views, stored procedures, ETL, fields, etc., to form a data asset map for visual display.

[0018] In step A2, the storage form of the sorted and associated data is specifically a graph database, in which the data table or data field is used as a graph database node, and the relationship between data tables or the relationship between data fields is used as a graph database relationship. Automatic task: systematically sort and associate metadata, and store the sorted information; graph database: the table is used as a graph database node, and the relationship between tables is used as a graph database relationship; the field is used as a graph database node, and the relationship between fields is used as a graph database relationship.

[0019] The visualized data lineage relationship in step A3 specifically includes: upstream and downstream hierarchical information, forward tracing to source information, backward tracing data information, and relationship maps. Search the target data table / task to quickly expand the relationship map; display the full link of the data table field lineage relationship; search the target data table / task to quickly expand the field relationship map; view related tasks and view task details; search the data table target field to view the field lineage relationship; auxiliary display and analysis functions such as maximum, minimum, positioning, and full screen of the map relationship.

[0020] The data collection in the data storage and management module is mainly carried out through the API interface, and the API interface specifically includes: data middle platform metadata API interface, data middle platform business data API interface; the management of the collected data in the data storage and management module specifically includes: project management, cluster management, and engine management; among them, the engines managed in the engine management specifically include: Hadoop cluster, Oracle, and GP.

[0021] The overall system adopts a computing and storage separation method, which separates data storage from resource computing and stores data on multiple nodes. This separation allows data storage and computing to be performed independently, thus avoiding the problem of data storage and computing being mutually constrained in traditional architectures.

[0022] By separating data storage and computing processing, data processing tasks can be completed more quickly and the real-time and response speed of data processing can be improved. The storage and computing separation solution can deploy data storage and computing processing on different devices respectively, reducing the company's hardware and maintenance costs. By optimizing the resource allocation of data storage and computing processing, resource utilization can be further improved and data processing costs can be reduced. The storage and computing separation solution stores data on multiple nodes, which increases data security and avoids the risk of data loss or tampering, thereby ensuring data integrity and reliability.

[0023] The present invention aims to realize unified storage and management of data by integrating the advantages of data lake and data warehouse; at the same time, it uses storage and computing separation technology and deep learning algorithms to optimize data processing procedures and improve processing efficiency and accuracy; in addition, through enhanced lineage analysis functions, it can achieve accurate tracking of data flow and changes, providing enterprises with deeper data insights and decision-making support.

[0024] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A comprehensive data intelligent processing system, including a data storage and management module and a data analysis module, characterized in that: Also includes: Resource calculation module, data lineage analysis module; The data storage and management module is used to collect data, and store and manage the collected data; The data analysis module is used to analyze and process the stored data; The resource calculation module is used to calculate the optimal solution for data resource calls; The data lineage analysis module is used to analyze the flow and changes of data.

2. The comprehensive data intelligent processing system according to claim 1 is characterized in that: The data transfer and change in the data lineage analysis module mainly includes the following steps: A1. Create a data lineage analysis task; A2. Sort and associate the data and store the sorted and associated data; A3. Visualize the blood relationship of the analyzed data.

3. The comprehensive data intelligent processing system according to claim 2 is characterized in that: In the step A2, the storage form of storing the sorted and associated data is specifically a graph database, in which data tables or data fields are used as graph database nodes, and the relationships between data tables or the relationships between data fields are used as graph database relationships.

4. The comprehensive data intelligent processing system according to claim 2, characterized in that: The visualized data lineage relationship in step A3 specifically includes: upstream and downstream hierarchical information, forward tracing back to source information, backward tracing back to data information, and relationship map.

5. The comprehensive data intelligent processing system according to claim 1 is characterized in that: The data storage and management module collects data mainly through the API interface, and the API interface specifically includes: data middle platform metadata API interface and data middle platform business data API interface.

6. The comprehensive data intelligent processing system according to claim 5, characterized in that: The management of the collected data in the data storage and management module specifically includes: project management, cluster management, and engine management; wherein the engines managed in the engine management specifically include: Hadoop cluster, Oracle, and GP.

7. The comprehensive data intelligent processing system according to claim 1, characterized in that: The overall system adopts a computing and storage separation approach, separating data storage from resource computing and storing data on multiple nodes.

Images