Data warehouse storage space recycling method and electronic device

Abstract

The application discloses a data warehouse storage space recycling method, comprising the steps of: in response to a recycling analysis task, obtaining upstream and downstream data table information of a data table according to the blood relationship of the data table in the data warehouse; determining a target data table that needs space recycling according to the latest update time of the data table and the latest update time of the upstream and downstream data tables; moving the target data table and the directory corresponding to the target data table to a temporary storage space; compressing the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and deleting the metadata of the target data table, the directory corresponding to the target data table and the file. The data warehouse storage resource recycling is automatically performed, manual operation is prevented from being not timely, and the situation that an error deletion or a missed deletion is caused due to insufficient business understanding ability is prevented, and corresponding systems, electronic equipment and computer readable storage media are provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, and in particular to a data warehouse storage space recycling method. In addition, the present application also relates to related systems, electronic devices and computer readable storage media. BACKGROUND

[0002] With the development of the Internet era, the amount of data has exploded, and big data technology is increasingly widely used. The data warehouse technology solution selection of enterprises has shifted from traditional databases to big data hive, spark and other offline computing solutions. As the company's business continues to develop and change, the data accumulated by the big data platform will continue to increase. Once the data warehouse resources are insufficient, it will directly affect the company's business, and the storage space needs to be continuously expanded to meet the daily needs. There are a large number of data in the data warehouse that are no longer valuable due to business changes, which not only increases resource costs, but also adversely affects the maintenance of the data warehouse. At present, the cleaning of the data warehouse mainly relies on manual maintenance. Because the size of the data warehouse is getting larger and larger, the relationship between the tables is complex, manual maintenance is not only labor-intensive, but also production accidents caused by insufficient consideration or misoperation, so a programmatic data warehouse storage resource recycling method is needed to avoid the complexity and error-prone problems of manual operation.

[0003] The content described in this background technology is only to facilitate the understanding of related technologies in this field, and is not regarded as an acknowledgement of the prior art. SUMMARY

[0004] Therefore, the embodiments of the present application intend to provide a data warehouse storage space recycling method, which can automatically recycle data warehouse storage resources, prevent manual operation from being not timely, and may cause errors due to insufficient business understanding ability, reduce manual costs and error problems. Specifically, the embodiments of the present application provide a data warehouse storage space recycling method, which includes the following steps:

[0005] In response to a recycling analysis task, upstream and downstream data table information of a data table is obtained according to the blood relationship of the data table in the data warehouse;

[0006] According to the latest update time of the data table and the latest update time of the upstream and downstream data tables, a target data table that needs to be recycled is determined;

[0007] The target data table and the directory corresponding to the target data table are moved to a temporary storage space;

[0008] The target data table and the directory corresponding to the target data table in the temporary storage space are compressed to generate a target compressed file, and the metadata of the target data table, the directory corresponding to the target data table and the file are deleted.

[0009] In some embodiments of the application, the method further comprises:

[0010] The recycling analysis task is generated when the storage resource usage of the data warehouse exceeds a threshold, or the recycling analysis task is generated according to a configured scheduling period.

[0011] In some embodiments of the application, the target data table requiring space recycling is determined according to the latest update time of the data table and the latest update time of the upstream data table.

[0012] The data table is determined as the target data table when the trigger time of the recycling analysis task minus the latest update time of the data table is greater than a recycling threshold time, and the trigger time of the recycling analysis task minus the latest update time of the upstream data table is greater than the recycling threshold time.

[0013] In some embodiments of the application, the target data table requiring space recycling is determined according to the latest update time of the data table and the latest update time of the upstream data table.

[0014] The data table is determined as the target data table when the trigger time of the recycling analysis task minus the latest update time of the data table is greater than a recycling threshold time, the trigger time of the recycling analysis task minus the latest update time of the upstream data table is greater than the recycling threshold time, and the data table belongs to a recyclable data table.

[0015] In some embodiments of the application, the method further comprises:

[0016] When the target data table is determined as a completely recyclable data table according to the pre-configuration, the target compressed file in the temporary storage space is cleaned up, and the recycling state of the target data table is set to completely recycled.

[0017] In some embodiments of the application, the method further comprises:

[0018] When the data table corresponding to the table query failure information is determined as the target data table in response to the table query failure information, the target compressed file is decompressed in the temporary storage space, a recovery data table and a directory corresponding to the recovery data table are generated;

[0019] The recovery data table and the directory corresponding to the recovery data table are moved to a preset directory.

[0020] In some embodiments of the application, the method further comprises:

[0021] The data table information, the data table blood relationship, the audit log, and the storage resource usage information are received.

[0022] According to the blood relationship and the audit log, a latest update time of the data table and a latest update time of an upstream data table of the data table are obtained.

[0023] In the embodiment of the application, a data warehouse storage space recycling system is also provided, comprising a big data platform subsystem, an information collection subsystem, a management subsystem, a task execution subsystem and a database, wherein,

[0024] The information collection subsystem is deployed to the big data platform subsystem and is configured to collect data table information, data table blood relationship, audit log and storage resource usage rate information of the big data platform subsystem;

[0025] The management subsystem is configured to analyze the blood relationship and the audit log reported by the information collection subsystem, obtain a latest update time of the data table, store the blood relationship of the data table and the latest update time of the data table into the database;

[0026] The management subsystem is further configured to, in response to a recycling analysis task, read the blood relationship of the data table in the data warehouse from the database, obtain upstream data table information of the data table, determine a target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream data table, and send the target data table information to the task execution subsystem;

[0027] The task execution subsystem is deployed to the big data platform subsystem and is configured to move the target data table and a directory corresponding to the target data table to a temporary storage space, compress the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and delete metadata of the target data table, the directory corresponding to the target data table and files.

[0028] In the embodiment of the application, a computer readable storage medium having a computer program stored thereon is provided, wherein the program is executed by a processor to implement the data warehouse storage space recycling method of any of the embodiments of the application.

[0029] In the embodiment of the application, an electronic device is provided, comprising a processor and a memory having a computer program stored thereon, wherein the processor is configured to execute the data warehouse storage space recycling method of any of the embodiments of the application when the computer program is run.

[0030] The embodiment of the present application provides a data warehouse storage space recycling method and system, which can automatically recycle data warehouse storage resources, can prevent manual operation from being not timely, and can prevent the case that error deletion or missing deletion is caused by insufficient business understanding ability, reduces manual cost and error problems; fine-grained storage resource recycling strategies are provided, storage resource recycling problems in different scenes are solved, and data table storage resources can be recycled in time and effectively; storage resource recycling can be completed without the participation of big data platform maintenance personnel, storage resource recycling efficiency is improved, and wasted storage space is released in time.

[0031] Other optional features and technical effects of the embodiment of the present application are described below, and some can be understood by reading this document. BRIEF DESCRIPTION OF DRAWINGS

[0032] Hereinafter, the embodiment of the present application will be described in detail with reference to the accompanying drawings, and the elements shown are not limited by the scale shown in the drawings, and the same or similar reference numerals in the drawings represent the same or similar elements, in which:

[0033] Figure 1 A system structure diagram of the data warehouse storage space recycling system is shown;

[0034] Figure 2 An exemplary flowchart of the data warehouse storage space recycling method according to the embodiment of the present application is shown;

[0035] Figure 3 A blood relationship diagram between data tables in the data warehouse storage space recycling method according to the embodiment of the present application is shown;

[0036] Figure 4 An exemplary flowchart of the data warehouse storage space recycling method according to some other embodiments of the present application is shown;

[0037] Figure 5 An exemplary flowchart of the data warehouse storage space recycling method according to some other embodiments of the present application is shown;

[0038] Figure 6 An exemplary flowchart of the data warehouse storage space recycling method according to some other embodiments of the present application is shown;

[0039] Figure 7 An exemplary flowchart of the data warehouse storage space recycling method according to some other embodiments of the present application is shown;

[0040] Figure 8 An exemplary flowchart of the data warehouse storage space recycling method according to some other embodiments of the present application is shown;

[0041] Figure 9An exemplary structural schematic diagram of a data warehouse storage space recycling device according to an embodiment of the present application is shown.

[0042] Figure 10 An exemplary structural schematic diagram of an electronic device capable of implementing a method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be given below in conjunction with specific embodiments and drawings. Herein, the exemplary embodiments of the present application and their descriptions are used to explain the present application but not to limit the present application.

[0044] In the embodiments of the present application, "hive" is a data warehouse tool based on Hadoop, used for data extraction, transformation and loading, and is a mechanism that can store, query and analyze large-scale data stored in Hadoop; "spark" refers to a fast general-purpose computing engine designed for large-scale data processing; and "Hadoop" refers to a distributed system infrastructure.

[0045] As shown in Figure 1 The data warehouse storage space recycling system 100 in the embodiments of the present application includes a big data platform subsystem 110, an information collection subsystem 120, a management subsystem 130, a task execution subsystem 140 and a database 150. The information collection subsystem 120 is responsible for collecting data table usage and reporting the collected information to the management subsystem 130. The management subsystem 130 is responsible for collecting, analyzing and displaying data table association and usage, providing rules and scheduling configuration functions, calling the task execution subsystem 140 to delete obsolete data table information and recycle storage space, and displaying storage space recycling records and progress. The task execution subsystem 140 is responsible for table metadata deletion and storage space recycling.

[0046] Specifically, the information collection subsystem 120 is deployed to the big data platform subsystem 110 and is configured to collect data table information, data table blood relationship, audit logs and storage resource usage information of the big data platform subsystem. The big data platform subsystem 110 runs a data warehouse, and the information collection subsystem 120 collects hive, spark table information and blood relationship, Hadoop audit logs and big data platform storage resource usage, and reports them to the management subsystem 130. The hive table blood relationship can be collected through a hive hook mechanism. The hive hook is a mechanism for extending and integrating external functions built into hive. The hive hook plug-in is integrated in the big data platform subsystem 110 to obtain blood relationship, execution records and other related information, thereby realizing information collection function.

[0047] The management subsystem 130 is configured to analyze the blood relationship and audit log reported by the information collection subsystem 120, obtain the latest update time of the data table, and store the blood relationship of the data table and the latest update time of the data table into the database 150. The management subsystem 130 analyzes the reported blood relationship and Hadoop audit log, synchronizes the latest update time of the data table and the Hadoop file directory, and stores the blood relationship and the update time into the database 150.

[0048] The management subsystem 130 is further configured to, in response to a recycling analysis task, read the blood relationship of the data table in the data warehouse from the database 150, obtain the upstream and downstream data table information of the data table, determine a target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream and downstream data tables, and send the target data table information to the task execution subsystem 140. The management subsystem 130 determines whether to trigger resource recycling according to the pre-configured storage resource recycling trigger ratio. When the information collection subsystem 120 reports that the storage resource usage rate has exceeded the threshold, the recycling analysis task is triggered. The management subsystem queries the blood relationship of the data table, determines whether the data table and the storage space are no longer used and need to be recycled according to the latest update time of the data table and the update time of the upstream and downstream data tables, and the pre-configured exclusion rules (the tables of the exclusion rules will not trigger recycling) and recycling rules, and records the data table metadata and recycling state information into the database. For example, if the update time of the data table and the upstream and downstream data tables are all within 60 days, the recycling instruction is submitted and is sent to the task execution subsystem 140 for execution.

[0049] In some embodiments, the management subsystem 130 provides scheduling configuration management, generates data table and storage space recycling analysis tasks according to the configured scheduling period, recycles the storage resources, and avoids triggering the threshold of the storage resource usage rate, thereby affecting the business use.

[0050] The task execution subsystem 140 is deployed to the big data platform subsystem 110 and is configured to move the target data table and the directory corresponding to the target data table to a temporary storage space, compress the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and delete the metadata of the target data table, the directory corresponding to the target data table, and the file. The task execution subsystem moves the data table to be recycled directory and file to the temporary storage space, stores them in a high compression format such as gzip, and then deletes the data table metadata and the directory and file corresponding to the data table. The management subsystem 130 updates the recycling information, updates the recycling state, and records the temporary storage space information, which is used for task rollback operation.

[0051] In some embodiments, the management subsystem 130 notifies the task execution subsystem to reclaim the storage space according to the configured data table reclaiming rule, for example, the data warehouse developer can set that the data table starting with temp and ending with the time in the format of year-month-day in the default database can be reclaimed after 60 days.

[0052] In some embodiments, the management subsystem 130 periodically schedules a task to scan the reclaiming record table, finds the data table that can be completely reclaimed according to the configured rule, and submits a cleaning instruction to the task execution subsystem 140, which cleans the temporary storage space and notifies the management subsystem 130 to update the reclaiming state to complete the reclaiming.

[0053] In some embodiments, after the management subsystem 130 receives the table query failure information reported by the information collection subsystem 120, it scans and analyzes the reclaiming record table, and if the data table query fails due to reclaiming, it notifies the task execution subsystem 140 to perform a reclaiming rollback operation and alerts the big data platform maintenance personnel. The task execution subsystem 140 creates a data table according to the reclaiming rollback instruction, moves the directories and files from the temporary storage space to the data table specification directory, and notifies the management subsystem 140 to update the reclaiming record state after the execution is completed.

[0054] The reclaiming system in the embodiments of the present application can manage the whole life cycle of data table storage resource reclaiming and support data table association relationship query; provide fine-grained storage resource reclaiming strategy to solve the storage resource reclaiming problem in different scenarios; all process information is retained to facilitate data warehouse developers and big data platform maintenance personnel to audit and troubleshoot problems; without too much manual intervention, the efficiency of storage resource reclaiming is effectively improved.

[0055] As shown in Figure 2 The present application provides a data warehouse storage space reclaiming method, which comprises the following steps:

[0056] S110, in response to a reclaiming analysis task, acquiring upstream and downstream data table information of a data table according to the blood relationship of the data table in the data warehouse; the reclaiming analysis task can be generated according to a preset configuration, and can be triggered to be generated when the storage resource utilization rate of the data warehouse exceeds a threshold value, or can be generated in a timing scheduling manner. The blood relationship is extracted from the data warehouse through a hive hook plug-in and a Hadoop audit log and stored in a database. When the reclaiming analysis task is responded to, the blood relationship is acquired from the database. The big data blood relationship refers to the link of data generation. The upstream and downstream data table information of the data table is acquired according to the blood relationship. As shown in Figure 3 As shown in Figure 3 , data table 2 is derived from data table 1, and data table 3 is derived from data table 2, so the upstream and downstream data tables of data table 2 are data table 1 and data table 3.

[0057] As shown in Figure 4 the method of the embodiment of the application further comprises:

[0058] S108, when the storage resource usage rate of the data warehouse exceeds the threshold value, a recycling analysis task is generated, for example, the threshold value is 75%, when it is detected that the storage resource usage rate of the data warehouse is 60%, no recycling analysis is performed, but the storage resource usage rate after continuous use is 80%, then the recycling analysis task is triggered, in order to ensure the integrity of each big data platform calculation, avoid the loss of calculation data caused by recycling, the collection of the storage resource usage rate is performed after each calculation is completed, and in the calculation, no collection is performed.

[0059] As shown in Figure 5 the method of the embodiment of the application further comprises:

[0060] S109, according to the configured scheduling period, a recycling analysis task is generated, for example, the configured period is 3 hours, then after the completion of the calculation task, and more than 3 hours after the last recycling analysis task is executed, the recycling analysis task is generated.

[0061] S120, according to the latest update time of the data table and the latest update time of the upstream and downstream data tables, a target data table needing space recycling is determined; the latest update time of the data table in the embodiment of the application is obtained from the data warehouse in a manner of information collection, specifically, the manner of a hive hook plug-in and an audit log is adopted, and after being obtained, is stored in a database, and can be stored in a table manner, for example, the storage format is as shown in the following table.

[0062]

[0063] The latest update time of the data table and the latest update time of the upstream and downstream data tables are obtained in a manner of querying the database.

[0064] In some embodiments of the present application, it is determined whether recovery is needed based on whether the update time exceeds a certain threshold. Specifically, it is determined that the data table is the target data table when the trigger time of the recovery analysis task minus the latest update time of the data table is greater than the recovery threshold time, and the trigger time of the recovery analysis task minus the latest update time of the upstream and downstream data tables is greater than the recovery threshold time. For example, the recovery threshold time is 60 days, the trigger time of the recovery analysis task is 2022-05-10, 19:00:02, and after subtracting the latest update time, neither of them exceeds 60 days, so recovery is not performed. When the trigger time of the recovery analysis task is 2022-06-08, 22:00:02, the latest update time is subtracted, which exceeds 60 days for data table 1, but does not exceed 60 days for data table 2, so recovery is not performed. When the trigger time of the recovery analysis task is 2022-06-12, 22:00:02, the latest update time is subtracted, which exceeds 60 days, so data table storage space recovery is performed, and data table 1, data table 2, and data table 3 are target data tables.

[0065] In some embodiments of the present application, in order to refine the management granularity, some joint data tables can be set not to be recovered. Specifically, it is determined that the data table is the target data table when the trigger time of the recovery analysis task minus the latest update time of the data table is greater than the recovery threshold time, the trigger time of the recovery analysis task minus the latest update time of the upstream and downstream data tables is greater than the recovery threshold time, and the data table belongs to a recoverable data table. For example, the recovery threshold time is 60 days, and when data table 1 is a recoverable data table, the trigger time of the task is 2022-06-12, 22:00:02, and recovery is performed on data table 1. When the data table is a non-recoverable data table, the trigger time of the task is 2022-06-18, 22:00:02, and recovery is not performed on data table 1.

[0066] S130, moving the target data table and the directory corresponding to the target data table to a temporary storage space; for example, moving data table 1 and its directory \financial data\data table 1 to a temporary storage space, for example, to a temporary space directory, and modifying the corresponding directory to \temporary space\financial data\data table 1.

[0067] S140, compressing the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and deleting the metadata of the target data table, the directory corresponding to the target data table, and the file. A high compression ratio method can be used for compression, for example, the compressed file is a gzip file, and then the metadata, directory, and file corresponding to the target data table are deleted.

[0068] In some embodiments, the data table recycling status can be recorded, for example, after data table 1 is recycled, its status is recorded as temporary recycling, and its directory information before recycling can also be recorded, facilitating subsequent recovery operations.

[0069] In some embodiments, in order to completely delete some permanently unused data tables, the method of the embodiment of the present application, as shown in Figure 6 , further comprises:

[0070] S150, according to the pre-configuration, when the target data table is a completely recyclable data table, cleaning the target compressed file in the temporary storage space, and setting the recycling status of the target data table as completely recycled. For example, the gzip file of the temporarily recycled data table 1 is deleted, and its status is recorded as completely recycled.

[0071] In order to ensure data security and prevent the function from being affected after data deletion, the embodiment of the present application further provides a recovery function for the temporarily recycled data table, specifically, as shown in Figure 7 , the method of the embodiment of the present application further comprises:

[0072] S160, in response to the table query failure information, when the data table corresponding to the table query failure information is the target data table, decompressing the target compressed file in the temporary storage space to generate a recovery data table and a directory corresponding to the recovery data table;

[0073] S170, moving the recovery data table and the directory corresponding to the recovery data table to a preset directory.

[0074] For example, for the gzip file of the temporarily recycled data table 1, decompression is performed to generate data table 1 in the temporary storage space, the corresponding directory is \temporary storage space\financial data\data table 1, and then data table 1 is moved to the original position, so that the corresponding directory is modified to \financial data\data table 1.

[0075] In some embodiments of the present application, the basic information of the data table is obtained in advance through data collection, facilitating the execution of recycling analysis tasks, specifically, as shown in Figure 8 , the method of the embodiment of the present application further comprises:

[0076] S101, receiving data table information, data table blood relationship, audit log and storage resource usage information;

[0077] S102, according to the blood relationship and the audit log, obtaining the latest update time of the data table and the latest update time of the upstream and downstream data tables of the data table.

[0078] The mechanism of the hive hook is used to integrate the hive hook plug-in in a data warehouse or a big data platform to obtain blood relationship and execution record and the like.

[0079] The big data platform data warehouse storage space recycling method provided by the embodiment of the application does not need manual operation for storage resource recycling, avoids misdeletion and omission of incomplete recycling of storage resources, provides a management platform for conveniently checking resource conditions and resource recycling conditions, provides a fine-grained storage resource recycling strategy, and solves the problem of storage resource recycling in different scenarios.

[0080] In the embodiment of the application, as shown in Figure 9 The data warehouse storage space recycling device 300 comprises:

[0081] The information acquisition module 310 is configured to acquire upstream and downstream data table information of a data table according to blood relationship of the data table in the data warehouse in response to a recycling analysis task.

[0082] The target data table determination module 320 is configured to determine a target data table that needs space recycling according to a latest update time of the data table and a latest update time of the upstream and downstream data tables.

[0083] The moving module 330 is configured to move the target data table and a directory corresponding to the target data table to a temporary storage space.

[0084] The recycling module 340 is configured to compress the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and delete metadata of the target data table, the directory corresponding to the target data table and files.

[0085] The device 300 in the embodiment of the application further comprises:

[0086] The recycling analysis task generation module 308 is configured to generate the recycling analysis task when a storage resource usage rate of the data warehouse exceeds a threshold value, or generate the recycling analysis task according to a configured scheduling period.

[0087] In some embodiments of the application, the target data table determination module 320 is specifically configured to:

[0088] When the trigger time of the recycling analysis task minus the latest update time of the data table is greater than a recycling threshold value time, and the trigger time of the recycling analysis task minus the latest update time of the upstream and downstream data tables is greater than the recycling threshold value time, the data table is determined as the target data table.

[0089] In some embodiments of the present application, the target data table determination module 320 is specifically configured to:

[0090] determine the target data table when the trigger time of the recycling analysis task minus the latest update time of the data table is greater than the recycling threshold time, the trigger time of the recycling analysis task minus the latest update time of the upstream and downstream data tables is greater than the recycling threshold time, and the data table belongs to the recyclable data table.

[0091] The device 300 in the embodiments of the present application further comprises:

[0092] The complete recycling module 350 is configured to, when the target data table is determined to be a completely recyclable data table according to the pre-configuration, clean up the target compressed file in the temporary storage space, and set the recycling state of the target data table to completely recycled.

[0093] The device 300 in the embodiments of the present application further comprises:

[0094] The recovery module 360 is configured to, in response to the table query failure information, when the data table corresponding to the table query failure information is the target data table, decompress the target compressed file in the temporary storage space to generate a recovery data table and a directory corresponding to the recovery data table.

[0095] The recovery data table and the directory corresponding to the recovery data table are moved to a preset directory.

[0096] The device 300 in the embodiments of the present application further comprises:

[0097] The preprocessing module is configured to receive data table information, data table blood relationship, audit logs and storage resource usage information; according to the blood relationship and the audit logs, obtain the latest update time of the data table and the latest update time of the upstream and downstream data tables of the data table.

[0098] In some embodiments, the data warehouse storage space recycling device can combine the features of the data warehouse storage space recycling method and system of any embodiment, and vice versa, which will not be repeated here.

[0099] In the embodiments of the present application, an electronic device is provided, comprising a processor and a memory storing a computer program, the processor being configured to execute the data warehouse storage space recycling method of any embodiment of the present application when running the computer program.

[0100] Figure 10A schematic diagram of an electronic device 1000 that can implement the method or realize the embodiments of the present application is shown, which can include more or fewer electronic devices than shown in some embodiments. In some embodiments, it can be implemented with a single or multiple electronic devices. In some embodiments, it can be implemented with cloud or distributed electronic devices.

[0101] As shown in Figure 10 The electronic device 1000 includes a processor (CPU) 1001 that can perform various appropriate operations and processes according to programs and / or data stored in a read only memory (ROM) 1002 or loaded from a storage section 1008 into a random access memory (RAM) 1003. The CPU 1001 can be one multi-core processor or can include a plurality of processors. In some embodiments, the CPU 1001 can include a general-purpose main processor and one or more special-purpose co-processors, such as a graphics processor (GPU), a neural network processor (NPU), a digital signal processor (DSP), and the like. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0102] The above processor and memory are used together to execute programs stored in the memory, which when executed by the computer can implement the steps or functions of the data warehouse storage space recycling method described in the above embodiments.

[0103] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, and the like; an output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage section 1008 including a hard disk, and the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, and the like. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 1010 as needed, so that a computer program read therefrom is installed in the storage section 1008 as needed. Figure 10 Only some components are shown schematically in the computer system 1000, and it is not meant to imply that the computer system 1000 only includes Figure 10 the components shown.

[0104] In some embodiments, the electronic device 1000 refers to a mobile terminal, including a mobile phone, a vehicle terminal, a smart TV, etc. Taking a mobile phone as an example, the electronic device 1000 further includes a display screen with a touch function, a loudspeaker, a gyroscope, a camera, a 4G / 5G antenna, and the like.

[0105] The system, apparatus, module or unit illustrated in the above embodiments can be implemented by a computer or its associated components. The computer can be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server, or a combination thereof.

[0106] Although not shown, in the embodiments of the present application, a storage medium is provided, which stores a computer program configured to perform the method of task scheduling monitoring of any of the embodiments of the present application when executed.

[0107] The storage medium of the embodiments of the present application includes permanent and non-permanent, movable and non-movable items that can store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0108] The methods, programs, systems, apparatuses, etc. of the embodiments of the present application can be executed or implemented in a single or multiple networked computers, or can be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks can be performed by remote processing devices connected through a communication network.

[0109] Those skilled in the art should understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, those skilled in the art can conceive that the implementation of the functional modules / units or controllers and related method steps illustrated in the above embodiments can be realized by software, hardware and a combination of software / hardware.

[0110] The acts or steps of the methods, programs, or procedures described according to embodiments of the present application need not necessarily be performed in the order described, and that the steps or acts of a method, or procedure can be performed in other orders or even at the same time, unless otherwise specified.

[0111] In this document, the terms "example" or "exemplary" are used as adjectives to indicate that something is an example of something else, but not necessarily the only example. For the context of this document, an "example" or "exemplary" feature or structure is just one of possible examples. Thus, unless otherwise specified, the use of "example" or "exemplary" in this document does not mean that a feature or structure is the only example of what could possibly be used in an embodiment of the present application. An "example" or "exemplary" feature or structure is just one of a possible

[0112] Exemplary systems and methods of the present application have been specifically illustrated and described herein, but various changes in the systems and methods described can be made without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A method for data warehouse storage space reclamation, the method comprising: The method comprises the following steps: in response to a recycling analysis task, obtaining upstream data table information of a data table according to the blood relationship of the data table in a data warehouse; determining a target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream data table, wherein whether recycling is needed is determined according to whether the latest update time of the data table and the latest update time of the upstream data table exceed a predetermined threshold value; moving the target data table and the directory corresponding to the target data table to a temporary storage space; compressing the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and deleting the metadata of the target data table, the directory corresponding to the target data table and the file.

2. The method of claim 1, wherein, Further comprising: generating the recycling analysis task when the storage resource usage rate of the data warehouse exceeds a threshold value; or, generating the recycling analysis task according to a configured scheduling period.

3. The method of claim 1, wherein, The determination of the target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream data table comprises: determining the data table as the target data table when the trigger time of the recycling analysis task minus the latest update time of the data table is greater than a recycling threshold time, and the trigger time of the recycling analysis task minus the latest update time of the upstream data table is greater than the recycling threshold time.

4. The method of claim 1, wherein, The determination of the target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream data table comprises: determining the data table as the target data table when the trigger time of the recycling analysis task minus the latest update time of the data table is greater than a recycling threshold time, the trigger time of the recycling analysis task minus the latest update time of the upstream data table is greater than the recycling threshold time, and the data table belongs to a recyclable data table.

5. The method of claim 1, characterized in that, Further comprising: when the target data table is determined to be a completely recyclable data table according to the pre-configuration, cleaning the target compressed file in the temporary storage space and setting the recycling state of the target data table to completely recycled.

6. The method of claim 1, characterized in that, Further comprising: in response to table query failure information, when the data table corresponding to the table query failure information is determined to be the target data table, decompressing the target compressed file in the temporary storage space to generate a restored data table and a directory corresponding to the restored data table; moving the restored data table and the directory corresponding to the restored data table to a preset directory.

7. The method of claim 1, wherein, Further comprising: receiving data table information, data table blood relationship, audit logs and storage resource usage information; obtaining the latest update time of the data table and the latest update time of the upstream data table of the data table according to the blood relationship and the audit logs.

8. A data warehouse storage space reclamation system, comprising: Comprising: a big data platform subsystem, an information collection subsystem, a management subsystem, a task execution subsystem and a database, wherein the information collection subsystem is deployed to the big data platform subsystem and is configured to collect data table information, data table blood relationship, audit logs and storage resource usage information of the big data platform subsystem; The management subsystem is configured to analyze the blood relationship and audit log reported by the information collection subsystem, obtain the latest update time of the data table, and store the blood relationship of the data table and the latest update time of the data table into the database. The management subsystem is further configured to read the blood relationship of the data table in the data warehouse from the database, obtain the upstream and downstream data table information of the data table, determine the target data table that needs to be recycled according to the latest update time of the data table and the latest update time of the upstream and downstream data tables, and send the target data table information to the task execution subsystem. The management subsystem is further configured to determine whether recycling is needed based on whether the latest update time of the data table and the latest update time of the upstream and downstream data tables exceed a predetermined threshold. The task execution subsystem is deployed to the big data platform subsystem and is configured to move the target data table and the directory corresponding to the target data table to a temporary storage space, compress the target data table and the directory corresponding to the target data table in the temporary storage space to generate a target compressed file, and delete the metadata of the target data table, the directory corresponding to the target data table, and the file.

9. A computer readable storage medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-7.

10. An electronic device, comprising: Comprise: A processor and a memory storing a computer program, the processor being configured to execute the method of any one of claims 1-7 when running the computer program.

Images