Data management platform and data management method

Abstract

The invention discloses a data management platform and a data management method.The data management platform comprises a data access module, a data processing module, a data storage module and a capacity management module; the data access module is used for performing metadata acquisition on heterogeneous data sources; wherein the metadata comprises unstructured data, structured data and semi-structured data; the data processing module is used for processing the metadata through a distributed computing framework and a data lake; the data storage module is used for storing the processed metadata; and the capacity management module is used for monitoring the data resource use condition of the current cluster in the data management platform in real time through an integrated monitoring system, and adjusting the data storage capacity according to the data resource use condition of the current cluster. The heterogeneous data from various sources can be effectively managed, and the security and flexibility of the data can be effectively ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data management platform and method. Background Art

[0002] In order to meet the needs of diverse business scenarios, it is often necessary to unify, integrate, share, associate, and effectively manage and monitor the massive amounts of data information between different systems. In traditional data processing architectures, data warehouses are usually used to store structured data for quick query and analysis, while data lakes are used to store large amounts of raw, unstructured or semi-structured data. Data Lakehouse is an emerging data management and analysis architecture that combines the advantages of data warehouses and data lakes to provide a unified, flexible, and high-performance data management platform that can achieve metadata synchronization and unified management between different data sources.

[0003] The existing patent (CN117971988A) proposes a fire data management and control system based on a lake-warehouse integrated architecture. This architecture integrates the data lake and data warehouse through a centralized fire data resource pool and isolates them in a partitioned manner. However, this solution does not have a solution for data backup and when the data management platform cluster encounters special situations, such as processing the server cluster, resulting in the inability to effectively guarantee data security and flexibility. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data management platform and method, which can effectively ensure the security and flexibility of data.

[0005] To solve the above technical problems, an embodiment of the present invention provides a data management platform, including a data access module, a data processing module, a data storage module and a capacity management module;

[0006] The data access module is used to collect metadata from heterogeneous data sources; wherein the metadata includes unstructured data, structured data and semi-structured data;

[0007] The data processing module is used to process the metadata through a distributed computing framework and a data lake;

[0008] The data storage module is used to store the processed metadata;

[0009] The capacity management module is used to monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system, and adjust the data storage capacity according to the data resource usage of the current cluster.

[0010] As an improvement of the above solution, the capacity management module is specifically used for:

[0011] Monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system;

[0012] Determine whether the data storage capacity needs to be expanded based on the current cluster data resource usage;

[0013] If yes, then perform capacity expansion and add new storage nodes;

[0014] Redistribute data storage so that new storage nodes can join the cluster and take on the load.

[0015] As an improvement to the above solution, the data storage is redistributed so that the new storage node can be integrated into the cluster and bear the load, including:

[0016] The improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that new storage nodes can be integrated into the cluster and bear the load.

[0017] As an improvement to the above solution, the improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that the new storage node can be integrated into the cluster and bear the load, including:

[0018] Generate hash values ​​using non-cryptographic hash functions;

[0019] Ensure that the generated hash value falls within a specific range through a modulo algorithm, so that the data stored in the data management platform is evenly distributed on each node of the current cluster;

[0020] Through the virtual bucket algorithm, the data stored in the data management platform is distributed to different storage buckets of the current cluster, so that new storage nodes can be integrated into the cluster and bear the load.

[0021] As an improvement of the above solution, the data storage module is specifically used for:

[0022] A distributed data object storage system is used to store processed unstructured data, and a lake warehouse is used to store processed structured data and semi-structured data.

[0023] As an improvement of the above solution, when storing the processed structured data, a digital watermark is added to the structured data that needs to be backed up, so as to identify the structured data and back up the structured data with the digital watermark.

[0024] As an improvement of the above solution, the data storage module is further used for:

[0025] The stored data is optimized through the optimization strategy.

[0026] As an improvement of the above solution, the optimization strategy includes at least one of data caching, intelligent partitioning and parallel processing.

[0027] As an improvement of the above solution, in the data processing module, the processing of the metadata includes clarifying, converting, aggregating and analyzing the metadata.

[0028] To solve the above technical problems, an embodiment of the present invention further provides a data management method, which is applied to the data management platform as described in any of the above embodiments, comprising:

[0029] Get target data;

[0030] Performing a preprocessing operation on the target data to obtain first data;

[0031] Writing the first data into a corresponding distributed data object storage system or lake warehouse;

[0032] The first data is queried and analyzed using a data warehouse tool.

[0033] Compared with the prior art, the embodiment of the present invention provides a data management platform and a data management method, wherein the data management platform includes a data access module, a data processing module, a data storage module and a capacity management module; the data access module is used to collect metadata from heterogeneous data sources; wherein the metadata includes unstructured data, structured data and semi-structured data; the data processing module is used to process the metadata through a distributed computing framework and a data lake; the data storage module is used to store the processed metadata; the capacity management module is used to monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system, and adjust the data storage capacity according to the data resource usage of the current cluster. A data management platform provided by an embodiment of the present invention can be located between various systems interacting with it, and the data generated by other systems can be accessed to the data management platform for storage and management. When the cluster part of the data management platform encounters problems or needs to expand the server, the data management platform can flexibly perform expansion and contraction processing, maintain the consistency and stability of the data, and ensure the continuity and efficiency of the service. Therefore, the embodiment of the present invention provides a data management platform with good scalability and manageability, which can effectively manage heterogeneous data from various sources and effectively ensure the security and flexibility of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the drawings used in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 is a structural block diagram of a data management platform provided by an embodiment of the present invention;

[0036] Figure 2 It is a flow chart of data access in a data management platform provided by an embodiment of the present invention;

[0037] Figure 3 is a flow chart of data processing in a data management platform provided by an embodiment of the present invention;

[0038] Figure 4 is a flow chart of data storage in a data management platform provided by an embodiment of the present invention;

[0039] Figure 5 is a flow chart of data storage capacity management in a data management platform provided by an embodiment of the present invention;

[0040] Figure 6 It is a flow chart of a data management method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 , Figure 1 : is a structural block diagram of a data management platform provided by an embodiment of the present invention, wherein the data management platform comprises:

[0043] The data access module is used to collect metadata from heterogeneous data sources; wherein the metadata includes unstructured data, structured data and semi-structured data;

[0044] The data processing module is used to process the metadata through a distributed computing framework and a data lake;

[0045] The data storage module is used to store the processed metadata;

[0046] The capacity management module is used to monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system, and adjust the data storage capacity according to the data resource usage of the current cluster.

[0047] It is worth noting that a data management platform provided by an embodiment of the present invention can be located between various systems that interact with it, and the data generated by other systems can be connected to the data management platform for storage and management. When the cluster part of the data management platform encounters problems or needs to expand the server, the data management platform can flexibly perform expansion and contraction processing to maintain the consistency and stability of the data and ensure the continuity and efficiency of the service. Therefore, an embodiment of the present invention provides a data management platform with good scalability and manageability, which can effectively manage heterogeneous data from various sources and provide users with data analysis, data application and data decision support capabilities.

[0048] It is worth noting that the data access module can be implemented based on the big data platform framework and API (interface) technology, and through standardized interface protocols, the business capabilities provided by the scheduling system can be called to complete metadata collection.

[0049] In an optional implementation, in the data processing module, the processing of the metadata includes clarifying, converting, aggregating and analyzing the metadata.

[0050] It should be noted that the data processing module can be implemented based on open source computing frameworks (such as Apache Spark) and data lake technology to process the imported data, including data cleaning, conversion, aggregation, and more complex analysis. Among them, an efficient data storage layer (such as Apache Hudi) can be applied to manage the data lake, which can support incremental data processing and time series data query.

[0051] In an optional implementation, the data storage module is specifically used to:

[0052] A distributed data object storage system is used to store processed unstructured data, and a lake warehouse is used to store processed structured data and semi-structured data.

[0053] It should be noted that the data storage module can be implemented based on the big data platform framework, data warehouse and data lake technology, thereby supporting multiple data formats and storage requirements.

[0054] Exemplarily, the Ceph object storage system can be used to store unstructured data, and the CRUSH (Controlled Replication Under Scalable Hashing) algorithm can be used to evenly distribute data objects throughout the cluster. This algorithm allows fast data location and load balancing, making data storage and retrieval more efficient. Ceph supports multiple storage interfaces, including block storage, file storage, and object storage, which enables it to flexibly handle unstructured data. For structured and semi-structured data, lake warehouses can be used for storage, such as HDFS (distributed file system) and MySQL database.

[0055] In an optional implementation, when storing the processed structured data, a digital watermark is added to the structured data that needs to be backed up, so as to identify the structured data and back up the structured data with the digital watermark.

[0056] It is worth noting that when structured data is connected to the database, the data management platform will automatically perform data backup processing, use data watermarks to embed the data that needs to be backed up in the database to mark it, and then identify the marked data and back it up. It is mainly achieved by modifying or expanding one or more columns of the data and generating verification data in another database to achieve the function of adding watermarks and identifying. Compared with traditional data backup, the digital watermark method of the embodiment of the present invention can significantly reduce the time for selecting backup data, thereby improving the overall efficiency of data processing and ensuring the persistence and reliability of data.

[0057] In an optional implementation, the data storage module is further used for:

[0058] The stored data is optimized through the optimization strategy.

[0059] Preferably, the optimization strategy includes at least one of data caching, intelligent partitioning and parallel processing.

[0060] It is worth noting that, in addition to storing processed data, the data storage unit is also responsible for improving storage efficiency, reducing storage costs, and accelerating data access speed through optimization strategies. Exemplarily, the data caching strategy caches frequently accessed data into memory or faster storage media, and reads directly from the cache the next time it is accessed. The cache framework (such as Redis, Memcached) or the database's own cache mechanism can be used; the intelligent partitioning strategy partitions the data according to data characteristics (such as time, region, user, etc.), so that data with the same characteristics are stored in the same partition. Partition design can be performed in the data modeling stage, or the partitioning function of the database can be used; parallel processing divides the data into multiple subsets, which are processed on different nodes respectively, and the results are finally summarized. Distributed computing frameworks (such as Hadoop, Spark) can be used for parallel data processing. By applying reasonable storage optimization strategies, data storage efficiency and data access speed can be significantly improved, and storage costs can be reduced. In specific implementation, appropriate strategies can be selected for storage optimization according to specific business needs and data characteristics.

[0061] In one implementation, the capacity management module is specifically configured to:

[0062] Monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system;

[0063] Determine whether the data storage capacity needs to be expanded based on the current cluster data resource usage;

[0064] If yes, then perform capacity expansion and add new storage nodes;

[0065] Redistribute data storage so that new storage nodes can join the cluster and take on the load.

[0066] Specifically, the redistribution of data storage so that the new storage node can be integrated into the cluster and bear the load includes:

[0067] The improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that new storage nodes can be integrated into the cluster and bear the load.

[0068] Furthermore, the improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that the new storage node can be integrated into the cluster and bear the load, including:

[0069] Generate hash values ​​using non-cryptographic hash functions;

[0070] Ensure that the generated hash value falls within a specific range through a modulo algorithm, so that the data stored in the data management platform is evenly distributed on each node of the current cluster;

[0071] Through the virtual bucket algorithm, the data stored in the data management platform is distributed to different storage buckets of the current cluster, so that new storage nodes can be integrated into the cluster and bear the load.

[0072] It should be noted that the amount of data and business needs in the data management platform cluster can be monitored in real time through an integrated monitoring system. Exemplarily, the Ceph Manager node is one of the core components of the Ceph cluster, which can be responsible for coordinating, maintaining and managing the status and metadata of the entire cluster, and providing some management and monitoring functions, and can provide real-time data on key performance indicators such as storage usage, computing load, and request latency. When a significant increase in the amount of data or a change in business needs is monitored, the capacity management module can trigger a resource adjustment process, such as involving an increase in computing resources (such as CPU, memory) or an expansion of storage resources (such as adding storage nodes).

[0073] Among them, in order to ensure that the newly added storage nodes can smoothly integrate into the cluster and bear the load, the data is redistributed through the Hashmod (hash) algorithm and the virtual bucket algorithm. Exemplarily, the hash value generated by the MurmurHash algorithm is then optimized by the Hashmod algorithm to ensure that the data is evenly distributed on each node of the cluster. This optimization helps to reduce hash conflicts and improve data access efficiency. Through the virtual bucket algorithm, the data is distributed to different "buckets" in the Ceph cluster. These buckets can be optimized according to the characteristics of the data (such as access frequency, data type) to support the expansion of the cluster and adapt to the growing storage needs. After adding or removing nodes, the capacity management module can use specific commands (Ceph's balancer command) to rebalance resources and redistribute data. It is worth noting that this step ensures that the data remains evenly distributed in the new cluster configuration and business continuity is not affected.

[0074] In addition, capacity management can be integrated with Ceph Manager and open source monitoring solutions (such as Prometheus) so that monitoring data can be automatically converted into a recognizable format, further automating the scaling decision process. This integration also supports real-time feedback and adjustment of cluster status. For example, when Prometheus detects a sudden increase in data writes within a certain period of time, the capacity management module can analyze the current cluster resource usage and decide whether to add new storage nodes. If it is decided to expand capacity, it will trigger the cluster management tool to perform the expansion operation and redistribute data through the Hashmod algorithm and virtual bucket algorithm to ensure that the new node can smoothly integrate into the cluster and bear the load.

[0075] Therefore, the capacity management module can monitor the performance and resource usage of the data management platform in real time based on the characteristics of the data management platform itself and open source monitoring components, and automatically adjust resources according to preset rules, such as adding computing nodes or storage capacity; it can automatically adjust computing and storage resources according to the load and performance of the data management platform to ensure the high performance and high availability of the data management platform under different loads.

[0076] In order to enable those skilled in the art to more clearly understand a data management platform provided by an embodiment of the present invention, Figures 2 to 5 An example explanation of its workflow is given.

[0077] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of data access in a data management platform provided by an embodiment of the present invention. Figure 2 As shown, the data management platform first obtains metadata information from different data sources, and then requests the server where the metadata is located to distribute the data to the data management platform, and the data management platform further subdivides the data. Exemplarily, the data may include unstructured data, semi-structured data, and structured data.

[0078] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of data processing in a data management platform provided by an embodiment of the present invention. Figure 3 As shown, the data management platform selects the data to be processed from the acquired data, and then uses Apache Spark (an open source distributed processing system for big data processing) to process the data, including cleaning and converting the data, and authorizing the data that requires permission management. Finally, the processed data is written to the corresponding library through the storage interface, which can be provided to other systems or business departments using Hive as an analysis tool to analyze or query the processed data.

[0079] It should be noted that the data writing here is equivalent to a pre-storage step. In this step, after all types of data are processed, they are first stored in the data lake through a common interface, and then according to the initial path of each data, according to the set channel program, the function of the program is to query metadata and create channels to distribute them to the specified storage paths. For example, if the initial path source is a folder named " / usr / movie" on a server, its path will be stored in a separate column in the metadata, then the subsequent updated data in the folder will be synchronized to the data platform's object storage system Ceph, and other types of data distribution are distributed in the same way.

[0080] See also Figure 4 , Figure 4 FIG. 1 is a flow chart of data storage in a data management platform provided by an embodiment of the present invention. Figure 4 As shown in the figure, different storage methods are selected according to different data types. Among them, HDFS (Hadoop Distributed File System, a distributed file system) can be used as a storage solution for large-scale data sets, providing high reliability and high throughput; the Ceph object storage system is used to provide high scalability and flexible data access for unstructured data; for structured and semi-structured data, lake warehouses can be used for storage, such as MySQL database storage. In addition, column storage optimization can be provided by HBase (a column-based storage non-relational database management system running on HDFS), which is suitable for scenarios that require high-speed query and interactive analysis.

[0081] Specifically, an index may be created for a table stored in HBase; when data in a MySQL table is written, a watermark is added to the data to be backed up, the watermarked data is identified, and the backed up is performed.

[0082] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of data storage capacity management in a data management platform provided by an embodiment of the present invention. Figure 5 As shown, the monitoring component is used to monitor the current cluster resources of the data management platform. When the amount of data on the data management platform exceeds 70% of the cluster capacity, capacity expansion is performed, such as automatically adding new data storage nodes (such as Ceph osd4, HBase node4) to the cluster; when a server failure occurs in the cluster, capacity reduction is performed, and some data storage nodes or data processing nodes (such as HDFS node3) are automatically removed.

[0083] See also Figure 6 , Figure 6: is a flow chart of a data management method provided by an embodiment of the present invention. As shown in the figure, the data management method comprises steps S1 to S4:

[0084] S1. Obtain target data;

[0085] S2, performing a preprocessing operation on the target data to obtain first data;

[0086] S3. Write the first data into a corresponding distributed data object storage system or lake warehouse;

[0087] S4. Use a data warehouse tool to query and analyze the first data.

[0088] For example, the data generated by various systems are connected to the data management platform for storage and management. This data management method improves data management efficiency and analysis capabilities by standardizing data storage, processing and analysis processes, and can help enterprises better utilize data resources and make more informed decisions.

[0089] It should be noted that a data management method provided in an embodiment of the present invention is applied to the data management platform of any of the above embodiments.

[0090] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data management platform, characterized in that: It includes data access module, data processing module, data storage module and capacity management module; The data access module is used to collect metadata from heterogeneous data sources; wherein the metadata includes unstructured data, structured data and semi-structured data; The data processing module is used to process the metadata through a distributed computing framework and a data lake; The data storage module is used to store the processed metadata; The capacity management module is used to monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system, and adjust the data storage capacity according to the data resource usage of the current cluster.

2. The data management platform according to claim 1, characterized in that: The capacity management module is specifically used for: Monitor the data resource usage of the current cluster in the data management platform in real time through an integrated monitoring system; Determine whether the data storage capacity needs to be expanded based on the current cluster data resource usage; If yes, then perform capacity expansion and add new storage nodes; Redistribute data storage so that new storage nodes can join the cluster and take on the load.

3. The data management platform according to claim 2, characterized in that: The redistribution of data storage so that the new storage node can be integrated into the cluster and bear the load includes: The improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that new storage nodes can be integrated into the cluster and bear the load.

4. The data management platform according to claim 3, characterized in that: The improved hash algorithm is used in combination with the virtual bucket algorithm to redistribute the data storage of the data management platform so that the new storage node can be integrated into the cluster and bear the load, including: Generate hash values ​​using non-cryptographic hash functions; Ensure that the generated hash value falls within a specific range through a modulo algorithm, so that the data stored in the data management platform is evenly distributed on each node of the current cluster; Through the virtual bucket algorithm, the data stored in the data management platform is distributed to different storage buckets of the current cluster, so that new storage nodes can be integrated into the cluster and bear the load.

5. The data management platform according to claim 1, characterized in that: The data storage module is specifically used for: A distributed data object storage system is used to store processed unstructured data, and a lake warehouse is used to store processed structured data and semi-structured data.

6. The data management platform according to claim 5, characterized in that: When storing the processed structured data, a digital watermark is added to the structured data that needs to be backed up, so as to identify the structured data and back up the structured data with the digital watermark added.

7. The data management platform according to claim 6, characterized in that: The data storage module is further used for: The stored data is optimized through the optimization strategy.

8. The data management platform according to claim 7, characterized in that: The optimization strategy includes at least one of data caching, intelligent partitioning and parallel processing.

9. The data management platform according to claim 1, characterized in that: In the data processing module, the metadata is processed, including clarifying, converting, aggregating and analyzing the metadata.

10. A data management method, characterized in that: The data management platform as claimed in any one of claims 1 to 9 comprises: Get target data; Performing a preprocessing operation on the target data to obtain first data; Writing the first data into a corresponding distributed data object storage system or lake warehouse; The first data is queried and analyzed using a data warehouse tool.

Images