A method for data lifecycle governance in big data warehouse

Abstract

The present invention discloses a method for data lifecycle management of a big data warehouse, and belongs to the field of data processing technology. In the present invention, by real-time parsing, filtering, sharding, and parallel parsing of the metadata of a distributed file system, the parsing efficiency is improved, and the entire process of data security offline is completed through the construction of a set of automated processes, the impact on the business is minimized, and rollback operations are supported. For clusters of large-scale massive data, the access information of files can be quickly audited at the minute level, and hot and cold data and zombie data can be discovered in time. Through the data lifecycle management system, enterprises can be assisted to fully control the status of data assets, improve data quality, and reduce the cost of invalid data storage, providing effective decision-making information for enterprise data governance.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method for managing data lifecycle in a large data warehouse. Background Art

[0002] Data warehouse, whose English name is Data Warehouse, can be abbreviated as DW. It is a strategic collection of all types of data support for decision-making processes at all levels of the enterprise.

[0003] As a traditional data technology, data warehouses are still full of vitality in the big data era. However, in the big data era, the storage media, computing engines and application scenarios of data warehouses have undergone great changes. Traditional data warehouses are built on relational databases and are generally used to store GB-level data, such as Oracle, SQLServer, Mysql and other databases. Now big data data warehouses can use PB-level distributed storage systems, such as Hadoop, Clickhouse, etc., which means that more machine resources are needed for data storage. In order to reduce the storage cost of data, it is of great significance to identify invalid data and perform data governance throughout the entire life cycle.

[0004] Chinese patent document CN113886378A discloses a big data governance system, including: a data cleaning management module for managing data; the data cleaning management module includes: a data cleaning rule definition unit for defining rules for cleaning data; a data cleaning query unit for querying data; a data cleaning rule display unit for displaying data cleaning rules; a data modification and deletion unit for modifying or deleting data. By defining data cleaning rules, data can be managed and detected to avoid deleting or cleaning key and necessary data. However, in actual use, it cannot regulate the security of the data layer framework after data modification during data cleaning, and cannot well identify and locate the data to be processed, thereby affecting the data processing effect. Summary of the invention

[0005] The purpose of the present invention is to propose a method for data lifecycle management of a big data warehouse in order to solve the problem that the security of the data layer framework after data modification cannot be regulated during data cleaning, and the data to be processed cannot be well identified and located.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for data lifecycle management of a big data warehouse includes the following steps:

[0008] S101. Establish a data lifecycle management WEB platform to automatically detect and scan the input data through the data lifecycle management WEB platform;

[0009] S102, identifying the data lifecycle by scanning the data lifecycle management WEB platform, and performing zombie data screening and deduplication on the scanned data;

[0010] S103, performing self-service data compression service according to the user's request for entering data, and identifying the data warehouse partition table directory of the compressed data;

[0011] S104: Perform data security offline processing based on the scanned invalid data.

[0012] As a further description of the above technical solution:

[0013] It also includes methods for identifying and deduplicating zombie data, including:

[0014] S201, establish a distributed file subsystem for collecting and parsing metadata to obtain information and establish a metadata mirror file;

[0015] S202, copy the latest file system metadata image file, and slice the image file;

[0016] S203, loading the sliced ​​metadata image file data into the parsing engine, and transmitting the parsing result to the message pipeline asynchronously;

[0017] S204, filtering the message pipeline data to remove useless metadata;

[0018] S205, logical processing is performed in parallel according to the message partitions, and the information of the access time of the library table is updated to identify the zombie data and the hot and cold data.

[0019] As a further description of the above technical solution:

[0020] The metadata mirror file is a permanent checkpoint of the distributed file system metadata, and contains serialized information of all directories and file inodes of the file system.

[0021] As a further description of the above technical solution:

[0022] The self-service data compression service includes the following steps:

[0023] S301. Different plug-ins need to be configured for metadata collection of different tables. For example, to collect partition tables, you need to configure partition fields, and to collect tables, you need to configure the path of actual data storage. Define different plug-ins and then collect metadata;

[0024] S302, loading compression plug-ins for different types of data;

[0025] S303, loading data warehouse meta information, and formulating compression directory merging rules according to the partition situation;

[0026] S304, pre-selection and intelligent selection;

[0027] S305, starting the engine to select different compression components according to the compression meta information to perform data compression;

[0028] S306: Remove historical data and switch the data source to compressed data.

[0029] As a further description of the above technical solution:

[0030] The data security offline service includes the following steps:

[0031] S401. Add the target database table to the offline queue according to the business rules;

[0032] S402, modify the table metadata and rename the target table;

[0033] S403. After a certain period, if there is no abnormal feedback from the business, the table is deleted and the physical data is retained;

[0034] S404: After confirming that the business is not affected, the physical data is deleted and the table metadata is safely offline.

[0035] As a further description of the above technical solution:

[0036] The data loading compression plug-in described in S302 is GZIP, Snappy, BZip2, Parquet format + Snappy.

[0037] As a further description of the above technical solution:

[0038] The target renaming in S402 includes adding "_delete" to the table suffix. If the downstream business depends on the table, a dependency error will be generated, and the business determines whether a rollback is required to be safely offline.

[0039] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0040] In the present invention, the distributed file system metadata is parsed in real time, filtered, sharded, and analyzed in parallel to improve the parsing efficiency. By building a set of automated processes, the entire process of data security offline is completed, the impact on the business is minimized, and rollback operations are supported. For large-scale clusters of massive data, file access information can be quickly audited at the minute level, and hot and cold data and zombie data can be discovered in a timely manner. The data lifecycle governance system can assist enterprises in fully controlling the status of data assets, improving data quality, and reducing the cost of invalid data storage, providing effective decision-making information for enterprise data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A system block diagram of a method for managing data lifecycle in a big data warehouse proposed by the present invention;

[0042] Figure 2 A method flow chart of a method for data lifecycle management of a big data warehouse proposed by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1-2 The present invention provides a technical solution: a method for data lifecycle management of a big data warehouse, comprising the following steps:

[0045] S101. Establish a data lifecycle management WEB platform to automatically detect and scan the input data through the data lifecycle management WEB platform;

[0046] S102, identifying the data lifecycle by scanning the data lifecycle management WEB platform, and performing zombie data screening and deduplication on the scanned data;

[0047] Among them, zombie data identification service: This module is mainly responsible for detecting the last access time, access account and access method of the file. The difficulty lies in scanning a PB-level file system with more than 10 billion files. It is necessary to solve the efficiency and performance problems of the scanning engine, including the identification and deduplication methods of zombie data, including:

[0048] Build a distributed file subsystem to collect and parse metadata to obtain information and create a metadata mirror file;

[0049] Copy the latest file system metadata image file and slice the image file;

[0050] Load the sliced ​​metadata image file data to the parsing engine, and transmit the parsing results asynchronously to the message pipeline. The metadata image file is a permanent checkpoint of the distributed file system metadata, which contains the serialized information of all directories and file inodes of the file system. Inode is translated into Chinese as index node, which is used to store basic information of files and directories, including time, file name, user and group, etc.

[0051] Filter the message pipeline data to remove useless metadata, and use it to set up a permanent communication channel between applications, through which data can be read and written just like your own application accesses a flat file;

[0052] Use parallel logic processing according to message partitions, update the information of library table access time to identify zombie data and cold and hot data;

[0053] S103, performing self-service data compression service according to the user's request for entering data, and identifying the data warehouse partition table directory of the compressed data;

[0054] The self-service data compression service includes the following steps:

[0055] Different plug-ins need to be configured for metadata collection of different tables. For example, to collect metadata from partitioned tables, you need to configure partition fields, and to configure the actual data storage path for external tables. Define different plug-ins and then collect metadata.

[0056] Among them, the self-service data compression service: This module mainly includes setting compression cycles and compression methods for tables or files, automatically compressing data, and deleting historical data. The main difficulty is that it is necessary to detect the directory structure of the data warehouse partition table and solve the problem that merging folders will cause inconsistency between the data warehouse metadata and the actual physical data path;

[0057] For different types of data loading compression plug-ins, the data loading compression plug-ins are GZIP, Snappy, BZip2, and Parquet format + Snappy;

[0058] Load the metadata of the data warehouse and formulate compression directory merging rules based on the partition situation;

[0059] Pre-selection and smart selection;

[0060] The startup engine selects different compression components according to the compression meta-information to perform data compression;

[0061] Remove historical data and switch the data source to compressed data;

[0062] S104, performing data security offline processing according to the scanned invalid data;

[0063] Invalid data needs to be deleted. Conventional direct physical deletion will cause problems such as difficulty in recovery and unsmooth business migration of data-dependent parties. The data security offline service will use the method of smoothly completing the physical deletion of the table within a period of time.

[0064] The data security offline service includes the following steps:

[0065] According to business rules, add the target database table to the offline queue.

[0066] Modify the table metadata and rename the target table. Renaming the target table includes adding "_delete" to the table suffix. If the downstream business depends on this table, a dependency error will be generated. The business determines whether a rollback is required to safely go offline.

[0067] After a certain period, if there is no abnormal business feedback, the table will be deleted and the physical data will be retained.

[0068] After confirming that the business is not affected, the physical data is deleted and the table metadata is safely offline.

[0069] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for data lifecycle management in a big data warehouse. It is characterized in that The following steps are involved: S101. Establish a data lifecycle management WEB platform to automatically detect and scan the input data through the data lifecycle management WEB platform; S102, identifying the data lifecycle by scanning the data lifecycle management WEB platform, and performing zombie data screening and deduplication on the scanned data; S103, performing self-service data compression service according to the user's request for entering data, and identifying the data warehouse partition table directory for compressed data; S104, performing data security offline processing according to the scanned invalid data; The method for identifying and deduplicating zombie data specifically includes: S201, collect and analyze the distributed file system metadata to obtain information, and create a metadata mirror file; S202, copy the latest file system metadata image file, and slice the image file; S203, loading the sliced ​​metadata image file data into the parsing engine, and transmitting the parsing result to the message pipeline asynchronously; S204, filtering the message pipeline data to remove useless metadata; S205, performing logical processing in parallel according to the message partitioning method, updating the information of the library table access time to identify zombie data and cold and hot data; The self-service data compression service includes the following steps: S301, collect data warehouse partition tables, configure partition fields and external tables, configure actual data storage paths, define different plug-ins, and then collect metadata; S302, loading corresponding compression plug-ins for different types of data; S303, loading data warehouse meta information, and formulating compression directory merging rules according to the partition situation; S304, pre-selection and intelligent selection; S305, starting the engine to select different compression plug-ins according to the compression meta information to perform data compression; S306, remove historical data and switch the data source to compressed data; The data security offline processing includes the following steps: S401. Add the target database table to the offline queue according to the business rules; S402, modify the target library table metadata and rename the target library table; S403. After a certain period, if there is no abnormal feedback from the business, the target database table is deleted and the physical data is retained; S404: After confirming that the business is not affected, the physical data is deleted and the metadata of the target database table is safely offline.

2. A method for managing the data lifecycle of a big data warehouse according to claim 1, It is characterized in that The metadata mirror file is a permanent checkpoint of the distributed file system metadata, and contains serialized information of all directories and file index nodes inodes of the file system.

3. A method for managing the data lifecycle of a big data warehouse according to claim 1, It is characterized in that In S302, the compression plug-in is GZIP, Snappy, BZip2, and Parquet format + Snappy.

4. A method for managing the data lifecycle of a big data warehouse according to claim 1, It is characterized in that In S402, the target library table is renamed including adding "_delete" to the table suffix. If the downstream business depends on the table, a dependency error is generated, and the business determines whether a rollback is required to be safely offline.

Images