Data processing method and device in data warehouse, electronic equipment and storage medium

Abstract

The invention relates to a data processing method and device in a data warehouse, electronic equipment and a storage medium. The data processing method in the data warehouse comprises the steps of obtaining a data warehouse Hive table in the data warehouse, wherein HDFS metadata is stored in the Hive table; generating a first data table based on the HDFS metadata, obtaining life cycle metadata corresponding to the HDFS metadata, and performing clustering storage on the HDFS metadata according to a database field, an access field and a partition field in the first data table; based on the access field, the partition field and the life cycle metadata, HDFS metadata in a database corresponding to the database field in the first data table is cleaned or reserved. Through the method and the device, the storage space can be effectively managed, and the storage cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of big data processing and analysis, and in particular to a data processing method, device, electronic device and storage medium in a data warehouse. Background Art

[0002] Apache Hive is a popular data warehouse system built on Hadoop. It is mainly used to process and query large-scale data in the Hadoop Distributed File System (HDFS). Over time, the amount of data in Hive tables will continue to increase, which may lead to tight storage space, degraded query performance, and increased management complexity. As the amount of data continues to grow and accumulate, the data will also show huge differences in access popularity. For example, a platform will continuously write the latest data, but usually the access frequency of the most recently written data is much higher than that of data from a long time ago. If the same storage strategy is used regardless of the hot and cold data, it is a waste of cluster resources. Therefore, data storage needs to be cleaned up regularly. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, device, electronic device and storage medium in a data warehouse.

[0004] According to a first aspect of an embodiment of the present disclosure, a data processing method in a data warehouse is provided, including obtaining a data warehouse Hive table in the data warehouse, wherein the Hive table stores HDFS metadata; generating a first data table based on the HDFS metadata, and obtaining lifecycle metadata corresponding to the HDFS metadata, wherein the HDFS metadata is clustered and stored in the first data table according to a database field, an access field, and a partition field, wherein the database field indicates the database to which the HDFS metadata belongs, the access field indicates whether the HDFS metadata is accessed at a specified time, and the partition field indicates the partition to which the HDFS metadata belongs; based on the access field, the partition field, and the lifecycle metadata, the HDFS metadata in the database corresponding to the database field in the first data table is cleaned up or retained.

[0005] In one embodiment, the HDFS metadata in the database corresponding to the database field in the first data table is cleaned up or retained based on the access field, the partition field and the lifecycle metadata, including: based on the current scheduling time and the lifecycle metadata, obtaining a retention strategy for the HDFS metadata in the database corresponding to the database field in the first data table, the retention strategy including an HDFS metadata cleanup or retention time interval; if the retention strategy is cleanup, then the HDFS metadata in the database corresponding to the database field in the first data table is cleaned up; if the retention strategy is a retention time interval, then a partition list is determined based on the access field and the partition field, and the partition list is determined based on the first data table; in response to the partition list being within the retention time interval, retaining the HDFS metadata in the database of the database field in the partition list; in response to the partition list not being within the retention time interval, cleaning the HDFS metadata in the database of the database field in the partition list.

[0006] In one embodiment, the method also includes: in response to satisfying a condition, updating the retention policy of the HDFS metadata in the database corresponding to the database field in the first data table to cleanup; the condition includes at least one of the following: detecting that the first data table has a first identifier, and the first identifier indicates that the first data table is confirmed not to be used; detecting that the first data table is not associated with other data tables, and the access field of the first data table has not been accessed within a first preset time.

[0007] In one embodiment, the method further includes: in response to retaining the HDFS metadata in the database corresponding to the database field in the first data table, parsing the access field of the HDFS metadata to obtain the access heat of the HDFS metadata; based on the access heat, storing the HDFS metadata to a storage medium matching the access heat; wherein different storage media are used to store data with different access heats, storage media with higher access heats are used to store data with high access heats, and storage media with lower access heats are used to store data with low access heats.

[0008] In one embodiment, parsing the access field of the HDFS metadata to obtain the access popularity of the HDFS metadata includes: obtaining the current scheduling time; based on the current scheduling time and the access field of the HDFS metadata, obtaining the value of the access field of the HDFS metadata within a second preset time; based on the value of the access field of the HDFS metadata, determining the access popularity of the HDFS metadata.

[0009] In one embodiment, generating a first data table based on the HDFS metadata includes: converting the HDFS metadata into a readable metadata table, wherein the readable metadata table records details of the HDFS metadata, wherein the details include storage path, database name, table name, access time, whether the file is accessed at a specified time, the partition to which the file belongs, and the time partition in which the file is generated; clustering the HDFS metadata details recorded in the readable metadata table according to database fields, access fields, and partition fields to obtain the first data table.

[0010] In one embodiment, the HDFS metadata details recorded in the readable metadata table are clustered according to database fields, access fields and partition fields to obtain a first data table, including: in response to clustering according to the database fields, table fields, partition fields, access fields and time partition fields, a first data table aggregated according to partition granularity is obtained; in response to clustering according to the database fields, table fields, access fields and time partition fields, a first data table aggregated according to table granularity is obtained.

[0011] According to a second aspect of an embodiment of the present disclosure, a data processing device in a data warehouse is provided, comprising an acquisition unit, configured to acquire a data warehouse Hive table in the data warehouse, wherein the Hive table stores HDFS metadata; a generation unit, configured to generate a first data table based on the HDFS metadata, and to acquire lifecycle metadata corresponding to the HDFS metadata, wherein the HDFS metadata is clustered and stored in the first data table according to a database field, an access field, and a partition field, wherein the database field indicates a database to which the HDFS metadata belongs, the access field indicates whether the HDFS metadata is accessed at a specified time, and the partition field indicates a partition to which the HDFS metadata belongs; and a processing unit, configured to clean up or retain the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field, and the lifecycle metadata.

[0012] In one embodiment, the processing unit cleans or retains the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field and the lifecycle metadata in the following manner: based on the current scheduling time and the lifecycle metadata, obtain a retention strategy for the HDFS metadata in the database corresponding to the database field in the first data table, the retention strategy including an HDFS metadata cleanup or retention time interval; if the retention strategy is cleanup, clean up the HDFS metadata in the database corresponding to the database field in the first data table; if the retention strategy is a retention time interval, determine a partition list based on the access field and the partition field, and the partition list is determined based on the first data table; in response to the partition list being within the retention time interval, retain the HDFS metadata in the database of the database field in the partition list; in response to the partition list not being within the retention time interval, clean up the HDFS metadata in the database of the database field in the partition list.

[0013] In one embodiment, the processing unit is also used to: in response to satisfying a condition, update the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table to cleanup; the condition includes at least one of the following: detecting that the first data table has a first identifier, and the first identifier indicates that the first data table is confirmed not to be used; detecting that the first data table is not associated with other data tables, and the access field of the first data table has not been accessed within a first preset time.

[0014] In one embodiment, the processing unit is also used to: in response to retaining the HDFS metadata in the database corresponding to the database field in the first data table, parse the access field of the HDFS metadata to obtain the access heat of the HDFS metadata; based on the access heat, store the HDFS metadata to a storage medium that matches the access heat; wherein different storage media are used to store data with different access heats, storage media with higher access heats are used to store data with high access heats, and storage media with lower access heats are used to store data with low access heats.

[0015] In one embodiment, the acquisition unit parses the access field of the HDFS metadata to obtain the access popularity of the HDFS metadata in the following manner: obtain the current scheduling time; based on the current scheduling time and the access field of the HDFS metadata, obtain the value of the access field of the HDFS metadata within a second preset time; based on the value of the access field of the HDFS metadata, determine the access popularity of the HDFS metadata.

[0016] In one embodiment, the generation unit generates a first data table based on the HDFS metadata in the following manner: converting the HDFS metadata into a readable metadata table, wherein the readable metadata table records details of the HDFS metadata, wherein the details include a storage path, a database name, a table name, an access time, whether the file is accessed at a specified time, a partition to which the file belongs, and a time partition for generating the file; clustering the HDFS metadata details recorded in the readable metadata table according to database fields, access fields, and partition fields to obtain a first data table.

[0017] In one embodiment, the generation unit clusters the HDFS metadata details recorded in the readable metadata table according to database fields, access fields, and partition fields to obtain a first data table: in response to clustering according to the database fields, table fields, partition fields, access fields, and time partition fields, a first data table aggregated according to partition granularity is obtained; in response to clustering according to the database fields, table fields, access fields, and time partition fields, a first data table aggregated according to table granularity is obtained.

[0018] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the data processing method in the data warehouse described in the first aspect or any one of the embodiments of the first aspect.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data processing method in the data warehouse described in the first aspect or any one of the embodiments of the first aspect.

[0020] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: based on the database described in the HDFS metadata, whether the HDFS metadata is accessed at a specified time, the partition to which the HDFS metadata belongs generates a first data table, which is convenient for quickly locating relevant data. And the life cycle metadata corresponding to the metadata is obtained, so that the metadata is cleaned or retained based on the HDFS life cycle metadata, partition field and access field, which can effectively manage storage space and reduce storage costs.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] Figure 1 The present invention is a flowchart of a method for processing data in a data warehouse according to an exemplary embodiment.

[0024] Figure 2 The figure is a flowchart of a method for generating a first data table according to an exemplary embodiment.

[0025] Figure 3 The present invention is a flowchart showing a method for determining a storage medium according to an exemplary embodiment.

[0026] Figure 4 The figure is a flow chart showing a method for determining access popularity according to an exemplary embodiment.

[0027] Figure 5 The present invention is a flowchart of a method for processing data in a data warehouse according to an exemplary embodiment.

[0028] Figure 6 The figure is a schematic diagram showing a method for processing hot and cold data according to an exemplary embodiment.

[0029] Figure 7 The invention is a block diagram of a data processing device in a data warehouse according to an exemplary embodiment.

[0030] Figure 8 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0032] In the Apache Hive data warehouse system that stores data, the amount of data in the Hive table will continue to increase over time, resulting in tight storage space, degraded query performance, and increased management complexity.

[0033] In the related art, a retention time is usually preset for the related data, and then the data is cleaned up regularly based on the retention time. For example, in the modeling of the big data field, the data processing link of the data warehouse follows the concept of layering, including the operational data layer (Operational Data Store, ODS), the detailed data layer (Data Warehouse Detail, DWD), the summary data layer (Data Warehouse Summary, DWS) and the application data layer (Application Data Store, ADS). The life cycle of the data is configured according to the application scenario of the data warehouse layer. The ODS data warehouse layer permanently retains all historical data, the DWD data warehouse layer and the DWS data warehouse layer retain 5 years of data, and the DWA data warehouse layer retains 1 to 3 years of data. By controlling the data life cycle, the cost of data storage is reduced. However, in the related art, the data in the data warehouse is set with a fixed life cycle to cause redundancy of the data in the data warehouse, resulting in problems such as tight storage space.

[0034] Therefore, the present disclosure proposes a data processing method in a data warehouse, which customizes life cycles for different data and regularly cleans up data based on the life cycle to dynamically retain or not retain data, thereby reducing storage costs.

[0035] Figure 1 FIG. 1 is a flow chart showing a method for processing data in a data warehouse according to an exemplary embodiment. Figure 1 As shown, the method includes steps S11 to S13.

[0036] In step S11, a data warehouse Hive table in the data warehouse is obtained, and the HDFS metadata is stored in the Hive table.

[0037] In the embodiment of the present disclosure, in the field of big data processing and analysis, Apache Hive is a data warehouse system for storing large-scale data in HDFS. Therefore, a Hive table storing HDFS metadata in the data warehouse is obtained to facilitate data processing of the HDFS metadata in the Hive table.

[0038] In step S12, a first data table is generated based on the HDFS metadata, and lifecycle metadata corresponding to the HDFS metadata is obtained. The HDFS metadata is clustered and stored in the first data table according to database fields, access fields, and partition fields.

[0039] In the disclosed embodiment, a first data table is generated based on HDFS metadata, and the HDFS metadata is clustered and stored in the first data table according to database fields, access fields, and partition fields. Among them, the database field represents the database to which the HDFS metadata belongs. The access field indicates whether the HDFS metadata is accessed at a specified time, for example, whether the HDFS metadata is accessed on the same day. The partition field indicates the partition to which the HDFS metadata belongs. The partition field is usually related to the date on which the HDFS metadata is stored. If a certain HDFS metadata is stored in a partition today, the date corresponding to the partition field is today's date.

[0040] In the disclosed embodiment, HDFS metadata corresponds to lifecycle metadata. Lifecycle metadata can be understood as the retention time of HDFS metadata in the data warehouse. The lifecycle of HDFS metadata is controlled through lifecycle metadata, which is convenient for reducing data storage costs.

[0041] In the embodiment of the present disclosure, the life cycle metadata of HDFS metadata can be controlled based on the layered concept of data warehouse, which can include different layers as shown in Table 1 below, and the retention time of each layer of data is different.

[0042] ODS Permanent retention DWD 5 years DWS 5 years DWA 1-3 years data

[0043] Table 1

[0044] In the embodiment of the present disclosure, for the life cycle metadata of HDFS metadata, the retention time of HDFS metadata can be customized when the Hive table is created to control the life cycle of HDFS metadata. For example, the retention time of HDFS metadata set in Table 2.

[0045] Recent N months + end of each month Nearly N days Recent year + end of each month Not reserved Permanent retention

[0046] Table 2

[0047] In the disclosed embodiment, the HDFS metadata may be retained based on the data layer where the HDFS metadata is located, or the retention time of the HDFS metadata may be customized, for example, setting the current HDFS metadata to be retained for nearly N days.

[0048] In step S13, based on the access field, the partition field and the lifecycle metadata, the HDFS metadata in the database corresponding to the database field in the first data table is cleaned or retained.

[0049] In the disclosed embodiment, since the access field can indicate whether the HDFS data is accessed at a specified time, and the partition field can indicate the partition to which the HDFS metadata belongs and the characteristics of the lifecycle metadata that controls the lifecycle of the HDFS metadata, the HDFS metadata in the data of the corresponding database field in the first data table can be cleaned up or retained.

[0050] In the embodiment of the present disclosure, the lifecycle metadata corresponding to the HDFS metadata in the data of the corresponding database field in the first data table can be persisted to the data in the format shown in Table 3.

[0051]

[0052] Table 3

[0053] In the disclosed embodiment, after HDFS data exists in the Hive table, the life cycle metadata corresponding to the HDFS data will be automatically generated based on the HDFS custom or preset life cycle and persisted in the database. Based on the life cycle metadata of the above-mentioned custom HDFS metadata, or the life cycle metadata of the preset HDFS metadata, the life cycle of the HDFS metadata is controlled. Therefore, based on the life cycle metadata, access fields, and partition fields, the first data table can be dynamically retained or cleaned up, thereby realizing dynamic management of data storage and reducing the cost of data storage.

[0054] The following embodiment of the present disclosure describes how to generate a first data table based on HDFS metadata. Figure 2 FIG. 1 is a flow chart showing a method for generating a first data table according to an exemplary embodiment. Figure 2 As shown, the method includes steps S21 to S22.

[0055] In step S21, the HDFS metadata is converted into a readable metadata table.

[0056] In the disclosed embodiment, since the HDFS metadata is serialized and its content cannot be viewed directly in the file system, it is necessary to convert the HDFS metadata into a readable format for easy viewing and analysis. The metadata in HDFS can be obtained by the hdfsdfsadmin-fetchImage command, and the HDFS metadata binary can be converted into a readable CSV format using the hdfs oiv-i command. The converted readable metadata table is shown in Table 4 (Table 4 is only an example and does not limit the specific metadata), which may include the following metadata:

[0057]

[0058] Table 4

[0059] In the disclosed embodiment, based on the above data records the details of HDFS metadata, an HDFS metadata detail table is created. The readable metadata table file is imported into the data partition (data, dt) partition directory of the day of the HDFS metadata detail table. Among them, the dt partition value is the data time for generating HDFS metadata. Each dt partition saves the latest HDFS metadata of the dt day, and the dt time string is in yyyymmdd format, such as 20240721.

[0060] In the disclosed embodiment, the HDFS metadata details table may include storage path, database name, table name, access time, whether the file has been accessed at the specified time, the partition to which the file belongs, and the time partition of the generated file. Among them, whether the file has been accessed at the specified time is processed by accesstime and dt. The specific logic is to determine whether the accesstime is greater than or equal to dt. If it is greater, it means that the file has been accessed today, and whether the file has been accessed at the specified time is set to 1. Otherwise, it is 0.

[0061] Exemplarily, the HDFS metadata detail table includes the details shown in Table 5:

[0062]

[0063] Table 5

[0064] For example, based on the data in Table 5, an example of HDFS metadata details is shown in Table 6:

[0065] path / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20240718 table_name hdfs_meta_analysis db_name mfw_archive.db accesstime 2024-07-20 09:40 is_today_access 1 partition_name dt=20240718 dt 20240720

[0066] Table 6

[0067] In step S22, the HDFS metadata details recorded in the readable metadata table are clustered according to the database field, the access field, and the partition field to obtain a first data table.

[0068] In the disclosed embodiment, the HDFS metadata details recorded in the readable metadata table are clustered based on the database field, the access field, and the partition field to obtain a first data table. It can be understood that the HDFS metadata details are clustered based on different data to obtain a first data table containing different data.

[0069] In the disclosed embodiment, by converting the HDFS metadata into a readable metadata table, it is convenient to generate a first data table so as to clean up or retain the HDFS metadata in the first data table.

[0070] In the embodiment of the present disclosure, HDFS metadata details are clustered based on different data to obtain a first data table containing different data, which may specifically include the following two types.

[0071] In the disclosed embodiment, in response to grouping the records in the dt partition specified in the HDFS metadata detail table by the table_name, db_name, and partition_name fields, the grouped data is aggregated to obtain a first data table aggregated based on the partition field. The is_today_access field value in the database corresponding to the database field in the first data table is judged to determine whether the metadata corresponding to the database field in the first data table has been accessed within the specified time, for example, whether it has been accessed on the same day. If the access field value is 0, it indicates that the data of the partition_name partition of the Hive table named table_name has been accessed on the dt day of the HDFS metadata detail table.

[0072] Exemplarily, the first data table aggregated based on the partition field is described through Table 7.

[0073]

[0074] Table 7

[0075] For example, as shown in Table 7, the table name hdfs_meta_analysis and the path / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20240718 are used as an example. The data is grouped by the table_name, db_name, and partition_name fields, and then after aggregation, sum(is_partition_today_access) is equal to 1, so the partition dt=20240718 of the table hdfs_meta_analysis is accessed on 20240720. That is, since hdfs_meta_analysis is stored on 20240718, and then when partitioning is performed on 20240720, is_partition_today_access is judged to be 1, it can be considered that the partition dt=20240718 of the table hdfs_meta_analysis is accessed on 20240720.

[0076] In the embodiment of the present disclosure, in response to the HDFS metadata detail table specifying that the records in the dt partition are grouped by the table_name and db_name fields, the grouped data is aggregated to obtain a first data table aggregated based on the table granularity. Based on the access field value of the first data table, if the access field value is greater than 0, it indicates that the data of the Hive table named table_name is accessed on the dt day of the HDFS metadata detail table.

[0077] Exemplarily, Table 8 is used to illustrate the first data table that is aggregated based on table granularity.

[0078]

[0079]

[0080] Table 8

[0081] For example, in Table 8, HDFS metadata is grouped by the table_name and db_name fields, and then after aggregation, sum(is_today_access) is equal to 1, and the table hdfs_meta_analysis is accessed on 20240720.

[0082] In the disclosed embodiment, the first data table obtained after aggregation according to the partition granularity or the first data table obtained after aggregation according to the table granularity can be used to determine whether the HDFS metadata has been accessed on the day from the perspective of the partition field, and can also be used to determine whether the HDFS metadata has been accessed on the day from the perspective of the table, thereby determining whether the HDFS metadata in the first data table should be cleaned or retained, or determining whether the HDFS metadata is cold data, and thus placing it in the target storage medium.

[0083] In the disclosed embodiment, after the first data table is acquired based on the HDFS metadata, the HDFS metadata in the database corresponding to the database field in the first data table is cleaned or retained based on the access field, the partition field and the lifecycle metadata.

[0084] In the disclosed embodiment, the current scheduling time is obtained, and based on the current scheduling time and the life cycle metadata of the first data table, a retention strategy of the HDFS metadata in the database corresponding to the data field in the first data table is obtained. The retention strategy includes a time interval for cleaning or retaining the HDFS metadata, and the HDFS metadata is cleaned or retained through the time interval for cleaning or retaining the HDFS metadata.

[0085] In the embodiment of the present disclosure, if the retention policy is cleanup, such as "do not retain" in Table 2, the HDFS metadata in the database of the corresponding database field in the first data table is cleaned up.

[0086] In the disclosed embodiment, if the retention policy is a retention time interval, such as a retention policy other than "Retain" in Table 2, a partition list is determined based on the access field and the partition field, and further based on the retention time interval, the access field, and the partition list, the HDFS metadata in the data of the corresponding database field in the first data table is determined to be cleaned up or retained. The partition list is determined based on the first data table. Specifically, in response to the partition list being within the retention time interval, the HDFS metadata in the database of the database field in the partition list is retained. In response to the partition list not being within the retention time interval, the HDFS metadata in the database of the database field in the partition list is cleaned up.

[0087] For example, the retention policy of the table hdfs_meta_analysis is to retain for nearly 1 year, and the current scheduling time is 20240720. Then the hdfs_meta_analysis table data retention range is calculated to be the time interval of 20230720 to 20240720. Then the actual existing partition list of the current hdfs_meta_analysis table is queried from the Hive table. If there are two partition lists of 20230620 and 20230820. The partition list of dt=20230620 is not within the retention time interval, and the data storage path rule of the data warehouse table is spliced ​​to obtain the actual path / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20230620. The path is deleted and the metadata of the partition dt=20230620 of the table hdfs_meta_analysis is deleted from the hive table, completing the partition metadata and metadata cleanup. However, the partition list dt=20230820 is within the retention time interval, so the HDFS metadata in the database of the database field in the partition list is retained.

[0088] In the disclosed embodiment, the HDFS metadata is retained or cleaned up based on the current scheduling time and lifecycle metadata. Since the lifecycle metadata is customizable, the configuration process of the retention strategy is more flexible, thereby realizing dynamic processing of metadata and reducing storage costs.

[0089] In an embodiment of the present disclosure, in response to satisfying a condition, the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table is updated to cleanup. The condition may include at least one of the following: detecting that the first data table has a first identifier. The first identifier indicates that the first data table is confirmed to be inapplicable. It is detected that the first data table is not associated with other data, and the access field of the first data table has not been accessed within a first preset time.

[0090] In the disclosed embodiment, in response to the condition that the first data table is detected to have a first identifier, since in the data processing of the data warehouse, a table is usually processed from one or more tables, these processing logics will be put into the processing task and will be responsible for execution. If, due to demand adjustment, it is analyzed that the first data table is no longer needed, then the first data table is marked as no longer in use. If the presence of the above-mentioned identifier is detected, the task of the first data table can be taken offline, that is, the first data table is cleaned up. Therefore, the lifecycle metadata in the retention policy of the first data table is updated to not be retained.

[0091] In the embodiment of the present disclosure, in view of the condition that the first data table is detected to be unrelated to other data tables and the access field of the first data table has not been accessed within the first preset time, if the data processing of other tables does not depend on the table, it can be considered that the first data table has no downstream, that is, the first data table is unrelated to other data tables. And based on the first data table obtained at the table granularity, it is determined whether the access field of each partition in the dt partition has not been accessed within the first preset time, that is, whether it is 0. If the access field of each partition in the dt partition of the first data table obtained at the table granularity is 0 within the first preset time, it means that the storage access heat is 0 within the first preset time, and the lifecycle metadata in the retention strategy of the first data table can be updated to cleanup.

[0092] For example, if the current time is 20240720, in the aggregate table of the HDFS metadata table granularity, dt is between 20240420 and 20240720, and the table is mfw_dwd.db.hdfs_meta_analysis, if sum(is_table_today_access) is 0, it means that the storage access popularity of the table in the past three months is 0. And the table hdfs_meta_analysis has no downstream, then the lifecycle metadata update in the retention policy of the table is not retained and the table is cleaned up.

[0093] In the disclosed embodiment, if the conditions are met, the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table is updated to clean up. Since whether the first data table is associated with other data tables and whether it is accessed changes dynamically every day, it is possible to dynamically manage the HDFS metadata in the data table, realize automatic cleaning of metadata, improve the efficiency of data processing, and reduce the overall storage cost in an automated way.

[0094] In the disclosed embodiment, for the retained data table, it is necessary to determine the access popularity of the HDFS metadata in the data table after the data table is aggregated based on the partition granularity, and then manage the metadata with lower access popularity. Figure 3 FIG. 1 is a flowchart of a method for determining a storage medium according to an exemplary embodiment. Figure 3 As shown, the method includes steps S31 to S32.

[0095] In step S31, in response to retaining the HDFS metadata in the database corresponding to the database field in the first data table, the access field of the HDFS metadata is parsed to obtain the access popularity of the HDFS metadata.

[0096] In the disclosed embodiment, the HDFS metadata in the first data table is parsed to obtain the access field of the HDFS metadata, and based on the access field of the HDFS metadata, the access popularity of the HDFS metadata is obtained.

[0097] In step S32, based on the access popularity, the HDFS metadata is stored in a storage medium that matches the access popularity.

[0098] In the disclosed embodiments, different storage media are used to store data with different access heats. The storage media can be divided into two types, ordinary storage media (DISK) and high storage density, low speed and cheap storage media (ARCHIVE). DISK, for example, a hard disk drive (HDD) or a solid state drive (SSD), or an elastic block storage (EBS) network disk, etc. ARCHIVE, for example, tape storage, high-density hard disk, Amazon Simple Storage Service, etc. Data with different access heats can be stratified according to the access frequency of the file to obtain cold data (COLD) and hot data (HOT). HOT can be understood as frequently accessed data, which is stored on the DISK storage medium. COLD can be understood as rarely accessed data, which is stored on the ARCHIVE storage medium.

[0099] In the disclosed embodiment, hot data is stored on the DISK storage medium by default. Subsequently, the access popularity of the metadata of the first data table is analyzed to identify cold data, and the cold data is cooled, that is, the data is migrated to the ARCHIVE storage medium.

[0100] In the disclosed embodiment, after cold data is identified, the cold data is migrated to the ARCHIVE storage medium, which can reduce the storage cost of the DISK storage medium and realize automatic identification of cold data and automatic migration of cold data in the first data table aggregated based on partition granularity.

[0101] In the disclosed embodiment, cold data may be identified in the following manner. Figure 4 FIG. 1 is a flow chart showing a method for determining access popularity according to an exemplary embodiment. Figure 4 As shown, the method includes steps S41 to S43.

[0102] In step S41, the current scheduling time is obtained.

[0103] In step S42, based on the current scheduling time and the access field of the HDFS metadata, the value of the access field of the HDFS metadata within the second preset time is obtained.

[0104] In the embodiment of the present disclosure, the definition of cold data may be that the value of the access field is less than a preset value within the second preset time, for example, the number of access days of the table within the past year is less than 5.

[0105] In step S43, the access popularity of the HDFS metadata is determined based on the value of the access field of the HDFS metadata.

[0106] In the disclosed embodiment, the conversion of hot and cold data is triggered by a timer every day. If the current scheduling time is 20240720, determine whether the partition dt=20230820 of the Hive table hdfs_meta_analysis is cold data. In the aggregate table of the HDFS metadata partition granularity, the condition dt is limited to the range of 20230720 and 20240720, the table name is hdfs_meta_analysis, and the partition name is dt=20230820. Calculate whether sum(is_table_today_access) is less than 5. If it is less than 5, it means that the partition data of dt=20230820 of the table hdfs_meta_analysis is cold data. And migrate the identified cold data to the ARCHIVE storage medium.

[0107] In the disclosed embodiment, the first data table aggregated based on the partition granularity determines the access popularity of the HDFS metadata, thereby reducing the storage cost of the storage medium.

[0108] The present disclosure embodiment combines Figure 5 Describe the data processing methods in the data warehouse. Figure 5 FIG. 1 is a flow chart showing a method for processing data in a data warehouse according to an exemplary embodiment. Figure 5 As shown, Figure 5It includes steps 1 to 8. In step 1, NameNode exports HDFS metadata and converts it to CSV format. Use the hdfsdfsadmin-fetchImage command to obtain the fsimage file in HDFS. fsimage is a metadata file automatically generated and maintained by HadoopNameNode, which stores the latest storage information of each file in HDFS. This paper uniformly refers to the fsimage file as HDFS metadata. Hadoop provides the OfflineImageViewer tool. The hdfsoiv-i command can be used to convert HDFS metadata binary into a readable CSV format. HDFS metadata includes file path, number of copies, file modification time, file access time, file block size, number of file blocks, file size, directory permissions, directory creator, and directory creator group. In step 2, the HDFS metadata detail table is processed. Create a Hive detail table (collectively referred to as the HDFS metadata detail table) to store HDFS metadata to store HDFS metadata detail data and analyze data through the Hive engine. Based on the readable HDFS metadata converted in step 1, the Hive library, table, and partition corresponding to the current file path are parsed from the file path value through regular expressions. The HDFS metadata file converted in step 1 is imported into the dt partition directory of the HDFS metadata detail table. The dt partition value is the data time for generating HDFS metadata. Each dt partition saves the latest HDFS metadata of the day dt. The dt time string is in yyyymmdd format, such as 20240721. Among them, is_today_access is processed by accesstime and dt fields. The specific logic is to judge whether accesstime is greater than or equal to dt time. If so, it means that the file has been accessed today, and is_today_access is set to 1, otherwise it is 0. The HDFS metadata detail table includes the path, the name of the table to which the file belongs, the name of the database to which the file belongs, the time when the file was accessed, whether the file was accessed at that time, the name of the partition to which the file belongs, and the data partition of the day.

[0109] In the disclosed embodiment, in step three, the aggregate table of HDFS metadata partition granularity is processed, and the aggregate table of HDFS metadata table granularity is processed. When processing the aggregate table of HDFS metadata partition granularity, the records in the dt partition specified in the HDFS metadata detail table are grouped according to the table_name, db_name, and partition_name fields respectively, and the grouped data is_today_access field value is aggregated. If sum(is_today_access) is greater than 0, it indicates that the data of the partition_name partition of the Hive table named table_name has been accessed on the dt day of the HDFS metadata detail table. When processing the aggregate table of HDFS metadata table granularity, the records in the dt partition specified in the HDFS metadata detail table are grouped according to the table_name and db_name fields respectively, and the grouped data is_today_access field value is aggregated. If sum(is_today_access) is greater than 0, it indicates that the data of the Hive table named table_name has been accessed on the dt day of the HDFS metadata detail table. In step four, the Hive table is created and the retention policy of the table is specified. When creating a table, according to the default specification of the output layer, for example, mfw_dwd.db.hdfs_meta_analysis belongs to the DWD data warehouse layer. According to the default rule, the data retention policy of the mfw_dwd.db.hdfs_meta_analysis table is permanent retention, and the retention time can be customized. It is understandable that the retention policy of the table can be specified by default according to the data warehouse layer and can also be manually customized.

[0110] In the disclosed embodiment, in step 5, lifecycle management metadata. After the table is created, the table data lifecycle management metadata is automatically generated, and the specified retention policy of the table in step 4 is mapped to the Hive table and persisted in the database. The table data lifecycle management metadata information includes the database name of the Hive table, the Hive table name, the data retention policy, the fields designed for only n days / months, and the creation time. In step 6, it involves model processing offline, the task has no downstream, and the table data storage access heat is 0. The model processing offline can be understood as in the data processing of the data warehouse, a table is usually processed by one or more tables, and these processing logics will be put into the processing task and responsible for execution. If the table is no longer used for analysis due to demand adjustment, the task of the table can be offline, so historical data needs to be cleaned. Therefore, the retention policy in the lifecycle management metadata of the table will be dynamically updated to not retain. The task has no downstream and the table data storage access heat is 0. It can be understood that first, the data processing of other tables does not depend on the table, so it is considered that the table has no downstream. Secondly, in step 3, the aggregate table at the HDFS metadata table granularity is processed. If is_table_today_access is 0 for each partition in the dt partition of a table in the HDFS metadata table granularity in the past three months, it means that the storage access heat of the table in the past three months is 0. Example: If the current time is 20240720, dt is between 20240420 and 20240720 in the aggregate table at the HDFS metadata table granularity, and the table is mfw_dwd.db.hdfs_meta_analysis, if sum(is_table_today_access) is 0, it means that the storage access heat of the table hdfs_meta_analysis in the past three months is 0. If the table satisfies both no downstream and access heat of 0, the retention policy in the lifecycle management metadata of the table is updated to no retention.

[0111] In the disclosed embodiment, for step seven, data is automatically cleaned up. The lifecycle management of data, that is, the automatic cleanup of data, is automatically triggered by a timer every day. The maximum retention time range of the table data is calculated from the lifecycle management metadata of the table in the database according to the retention policy of each table in the metadata and the current time. For example, if the retention policy of the table hdfs_meta_analysis is to retain for nearly 1 year, and the current scheduling time is 20240720, then the hdfs_meta_analysis table data retention range is calculated to be the time interval of 20230720 to 20240720. Then, the actual existing partition list of the current hdfs_meta_analysis table is queried from the Hive engine. If the partition list is not within the retention range, the partition is deleted. For example, if the partition dt=20230620 is not within the reserved range, the data storage path rule of the data warehouse table will be spliced ​​into the actual path / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20230620. Delete this path and delete the metadata of the partition dt=20230620 of the table hdfs_meta_analysis from the hive engine to complete the cleanup of the partition metadata and data. Or if the retention policy of the table is not to retain, the table will be deleted while directly cleaning up the historical data. In step eight, based on the partition access heat analysis of the table, cold data is identified and the identified cold data is placed in cold storage. Figure 6 As shown, Figure 6It is a schematic diagram of a method for processing hot and cold data according to an exemplary embodiment. It can be understood that all data in the table are hot data on the DISK storage medium by default. By analyzing the data heat and access heat of the table, the cold data is identified, and the identified cold data is placed in a cold state, and the data is migrated to the ARCHIVE storage medium. Among them, cold data can be understood as data whose table partition access days within a preset time are less than a preset value. For example, the preset time can be one year, and the preset value can be 5. The present disclosure does not limit the specific data. And the identification of cold data can be performed in the following manner: the hot and cold data conversion is automatically triggered by a timer at a fixed time every day. After the data cleaning is completed in step seven, all partitions of all tables are scanned for cold data based on the definition of cold data, and the access to each table and partition is analyzed by querying the aggregation table of the HDFS metadata partition granularity to determine whether the partition access meets the definition of cold data. If the current date is 20240720, determine whether the partition dt=20230820 of the Hive table hdfs_meta_analysis is cold data. First, in the aggregate table of the HDFS metadata partition granularity, the conditions dt is limited to the range of 20230720 and 20240720, the table name is hdfs_meta_analysis, and the partition name is dt=20230820. Then calculate whether sum(is_table_today_access) is less than 5. If it is less than 5, it means that the partition data of dt=20230820 in the table hdfs_meta_analysis is cold data. After scanning out the cold data, the data needs to be cooled. For example, cool the partition dt=20230820 of hdfs_meta_analysis in the previous step. The steps for cooling are as follows: According to the data storage path rules of the data warehouse table, the actual directory is spliced, for example: hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20240720b. Set the directory storage to the ARCHIVE policy hdfsstoragepolicies-setStoragePolicy-path / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20240720-policyARCHIVE. Move the file to the ARCHIVE storage medium hdfsmover / hive / warehouse / mfw_archive.db / hdfs_meta_analysis / dt=20240720. Finally, the cooling operation of the 20230720 partition is completed.

[0112] In the disclosed embodiment, the access data information of Hive tables and partitions is obtained based on HDFS metadata, and the access heat analysis model of table and partition granularity is designed, which provides an important decision basis for data cleaning and cold and hot data identification. Based on the characteristics of data warehouse stratification, a unified retention strategy is designed for each table of each layer of each data warehouse, and a variety of customized strategies are designed based on actual scenarios, such as the strategy of the last N months + the end of each month, the last year + the end of each month, etc., so that users can customize the retention strategy of the specified table, and the configuration process of the retention strategy is more flexible. Secondly, by designing the data model data of the life cycle management and realizing the automatic execution through timer scheduling and systematization, the linkage of data cleaning is also realized for the offline model task and the task has no downstream and the access heat of the table data storage is 0, which connects the entire data life cycle management process in series, improves the overall data processing efficiency, and reduces the overall storage cost through automation. The access heat analysis model analyzes the access heat of all partitions in the Hive table of the data warehouse, identifies the cold data, and combines the cold and hot data stratification and the heterogeneous storage design of the cold and hot data to further reduce the overall storage cost.

[0113] Based on the same concept, the embodiment of the present disclosure also provides a data processing device 100 for a data warehouse.

[0114] It is understandable that the data processing device 100 of the data warehouse provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0115] Figure 7 FIG. 1 is a block diagram of a data processing device 100 in a data warehouse according to an exemplary embodiment. Figure 7 The device includes an acquisition unit 101, a generation unit 102 and a processing unit 103.

[0116] The acquisition unit 101 is used to acquire a data warehouse Hive table in the data warehouse, where the HDFS metadata is stored.

[0117] The generation unit 102 is used to generate a first data table based on the HDFS metadata and obtain lifecycle metadata corresponding to the HDFS metadata. The first data table clusters and stores the HDFS metadata according to database fields, access fields, and partition fields, wherein the database field indicates the database to which the HDFS metadata belongs, the access field indicates whether the HDFS metadata is accessed at a specified time, and the partition field indicates the partition to which the HDFS metadata belongs.

[0118] The processing unit 103 is used to clean up or retain the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field and the lifecycle metadata.

[0119] In one embodiment, the processing unit cleans or retains the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field and the lifecycle metadata in the following manner: based on the current scheduling time and the lifecycle metadata, obtain the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table, and the retention strategy includes the HDFS metadata cleanup or retention time interval; if the retention strategy is cleanup, then the HDFS metadata in the database corresponding to the database field in the first data table is cleaned; if the retention strategy is the retention time interval, then based on the access field and the partition field, determine the partition list, and the partition list is determined based on the first data table; in response to the partition list being within the retention time interval, retain the HDFS metadata in the database of the database field in the partition list; in response to the partition list not being within the retention time interval, clean up the HDFS metadata in the database of the database field in the partition list.

[0120] In one embodiment, the processing unit is also used to: in response to satisfying a condition, update the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table to cleanup; the condition includes at least one of the following: detecting that the first data table has a first identifier, and the first identifier indicates that the first data table is confirmed not to be used; detecting that the first data table is not associated with other data tables, and the access field of the first data table has not been accessed within a first preset time.

[0121] In one embodiment, the processing unit is also used to: in response to retaining the HDFS metadata in the database corresponding to the database field in the first data table, parse the access field of the HDFS metadata to obtain the access heat of the HDFS metadata; based on the access heat, store the HDFS metadata to a storage medium that matches the access heat; wherein different storage media are used to store data with different access heats, storage media with higher access heats are used to store data with high access heats, and storage media with lower access heats are used to store data with low access heats.

[0122] In one embodiment, the acquisition unit parses the access field of the HDFS metadata to obtain the access popularity of the HDFS metadata in the following manner: obtain the current scheduling time; based on the current scheduling time and the access field of the HDFS metadata, obtain the value of the access field of the HDFS metadata within a second preset time; based on the value of the access field of the HDFS metadata, determine the access popularity of the HDFS metadata.

[0123] In one embodiment, the generation unit generates a first data table based on HDFS metadata in the following manner: converting the HDFS metadata into a readable metadata table, wherein the readable metadata table records details of the HDFS metadata, including storage path, database name, table name, access time, whether the file has been accessed at a specified time, the partition to which the file belongs, and the time partition in which the file was generated; clustering the HDFS metadata details recorded in the readable metadata table according to the database field, access field, and partition field to obtain the first data table.

[0124] In one embodiment, the generation unit clusters the HDFS metadata details recorded in the readable metadata table according to database fields, access fields, and partition fields to obtain a first data table: in response to clustering according to database fields, table fields, partition fields, access fields, and time partition fields, a first data table aggregated according to partition granularity is obtained; in response to clustering according to database fields, table fields, access fields, and time partition fields, a first data table aggregated according to table granularity is obtained.

[0125] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0126] like Figure 8 As shown, Figure 8 : is a schematic diagram of an electronic device provided by an embodiment of the present invention. One embodiment of the present invention provides an electronic device 200. Among them, the electronic device 200 includes a memory 201, a processor 202, and an input / output (I / O) interface 203. Among them, the memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the file merging method of the embodiment of the present invention. Among them, the processor 202 is connected to the memory 201 and the I / O interface 203 respectively, for example, it can be connected through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the file merging method involved in the embodiment of the present invention, and the processor 202 executes various functional applications and data processing of the electronic device 200 by running the program stored in the memory 201.

[0127] In the embodiment of the present invention, the processor 202 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 202 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0128] The memory 201 in the embodiment of the present invention may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD) or solid-state drive (SSD).

[0129] In the embodiment of the present invention, the I / O interface 203 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the electronic device 200), and may also output various information to the outside (such as images or sounds). In the embodiment of the present invention, the I / O interface 203 may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.

[0130] It is to be understood that, although the operations are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood as requiring to perform these operations in the specific order shown or in a serial order, or requiring to perform all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0131] The methods and devices involved in the embodiments of the present invention can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0132] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0133] The foregoing description of the implementation of the present invention has been given for purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the invention to the exact form disclosed, and various variations and modifications are possible in accordance with the above teachings or may be obtained from the practice of the invention. These embodiments are selected and described in order to illustrate the principles of the invention and its practical application, so that those skilled in the art can utilize the invention in various embodiments and with various modifications suitable for the particular use contemplated.

Claims

1. A data processing method in a data warehouse, It is characterized in that include: Obtain a data warehouse Hive table in the data warehouse, where the HDFS metadata is stored; A first data table is generated based on the HDFS metadata, and lifecycle metadata corresponding to the HDFS metadata is obtained, wherein the HDFS metadata is clustered and stored in the first data table according to a database field, an access field, and a partition field, wherein the database field indicates a database to which the HDFS metadata belongs, the access field indicates whether the HDFS metadata is accessed at a specified time, and the partition field indicates a partition to which the HDFS metadata belongs; Based on the access field, the partition field and the lifecycle metadata, the HDFS metadata in the database corresponding to the database field in the first data table is cleaned or retained.

2. The method according to claim 1, It is characterized in that The cleaning or retaining of the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field, and the lifecycle metadata includes: Based on the current scheduling time and the lifecycle metadata, a retention strategy for the HDFS metadata in the database corresponding to the database field in the first data table is obtained, wherein the retention strategy includes a time interval for cleaning or retaining the HDFS metadata; If the retention strategy is cleaning, cleaning the HDFS metadata in the database corresponding to the database field in the first data table; If the retention strategy is a retention time interval, a partition list is determined based on the access field and the partition field, and the partition list is determined based on the first data table; In response to the partition list being within the retention time interval, retaining HDFS metadata in the database of the database field in the partition list; In response to the partition list not being within the retention time interval, clearing HDFS metadata in the database of the database field in the partition list.

3. The method according to claim 2, It is characterized in that The method further comprises: In response to satisfying the condition, updating the retention strategy of the HDFS metadata in the database corresponding to the database field in the first data table to clean up; The conditions include at least one of the following: detecting that the first data table has a first identifier, wherein the first identifier indicates that the first data table is confirmed not to be used; It is detected that the first data table is not associated with other data tables, and an access field of the first data table has not been accessed within a first preset time.

4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: In response to retaining the HDFS metadata in the database corresponding to the database field in the first data table, parsing the access field of the HDFS metadata to obtain the access popularity of the HDFS metadata; Based on the access heat, storing the HDFS metadata to a storage medium matching the access heat; Different storage media are used to store data with different access popularity. A storage medium with higher access popularity is used to store data with higher access popularity, and a storage medium with lower access popularity is used to store data with lower access popularity.

5. The method according to claim 4, It is characterized in that The parsing the access field of the HDFS metadata to obtain the access popularity of the HDFS metadata includes: Get the current scheduling time; Based on the current scheduling time and the access field of the HDFS metadata, obtaining a value of the access field of the HDFS metadata within a second preset time; Based on the value of the access field of the HDFS metadata, the access popularity of the HDFS metadata is determined.

6. The method according to claim 1, It is characterized in that The generating a first data table based on the HDFS metadata includes: Convert the HDFS metadata into a readable metadata table, in which the HDFS metadata details are recorded, including storage path, database name, table name, access time, whether it is accessed at a specified time, the partition to which the file belongs, and the time partition when the file was generated; The HDFS metadata details recorded in the readable metadata table are clustered according to the database field, the access field and the partition field to obtain a first data table.

7. The method according to claim 6, It is characterized in that The HDFS metadata details recorded in the readable metadata table are clustered according to the database field, the access field, and the partition field to obtain a first data table, including: In response to clustering according to the database field, the table field, the partition field, the access field and the time partition field, a first data table aggregated according to the partition granularity is obtained; In response to clustering according to the database field, the table field, the access field and the time partition field, a first data table aggregated according to the table granularity is obtained.

8. A data processing device in a data warehouse, It is characterized in that include: An acquisition unit is used to acquire a data warehouse Hive table in the data warehouse, where the HDFS metadata is stored; A generating unit is used to generate a first data table based on the HDFS metadata, and obtain lifecycle metadata corresponding to the HDFS metadata, wherein the HDFS metadata is clustered and stored in the first data table according to a database field, an access field, and a partition field, wherein the database field indicates a database to which the HDFS metadata belongs, the access field indicates whether the HDFS metadata is accessed at a specified time, and the partition field indicates a partition to which the HDFS metadata belongs; A processing unit is used to clean up or retain the HDFS metadata in the database corresponding to the database field in the first data table based on the access field, the partition field and the lifecycle metadata.

9. An electronic device, It is characterized in that include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the data processing method in the data warehouse according to any one of claims 1 to 7.

10. A storage medium, It is characterized in that The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data processing method in the data warehouse according to any one of claims 1 to 7.

Images