A storage cleaning method and device for Hadoop, a computer device and a storage medium

Abstract

The embodiment of the application belongs to the field of big data, and relates to a storage cleaning method and device for Hadoop, computer equipment and a storage medium. The method comprises the following steps: obtaining a SQL execution log; analyzing the SQL execution log to obtain historical access information of each data table in Hadoop; generating a cleaning strategy based on the historical access information, wherein the cleaning strategy is a cleaning strategy corresponding to each type of data table in Hadoop; cleaning each data table in Hadoop according to the cleaning strategy to clean the storage of Hadoop; generating cleaning feedback information based on the historical access information, and sending the cleaning feedback information to a preset terminal, wherein the cleaning feedback information is used for storage optimization of Hadoop. In addition, the application also relates to blockchain technology, and the SQL execution log can be stored in the blockchain. The application comprehensively cleans the storage of Hadoop, and reduces the waste of Hadoop storage resources.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a storage cleanup method, apparatus, computer device and storage medium for Hadoop. Background Technology

[0002] Hadoop is a big data software system framework. With the popularization of big data technology in recent years, various organizations have been using Hadoop more and more, accumulating a large amount of historical data. This historical data not only consumes a lot of storage resources, but also affects the operating efficiency of various components in the Hadoop ecosystem. Therefore, Hadoop storage cleanup has become particularly important.

[0003] However, current storage cleanup techniques for Hadoop mainly focus on merging the underlying data tables of MapReduce (table files generated during MapReduce execution). This single cleanup method can only slightly and temporarily alleviate the storage pressure on Hadoop and cannot effectively clean up Hadoop's storage. Summary of the Invention

[0004] The purpose of this application is to provide a storage cleanup method, apparatus, computer device, and storage medium for Hadoop, so as to effectively clean up Hadoop storage and reduce the waste of Hadoop storage resources.

[0005] To address the aforementioned technical problems, this application provides a storage cleanup method for Hadoop, employing the following technical solution:

[0006] Get the SQL execution log;

[0007] The SQL execution logs are parsed to obtain historical access information for each data table in Hadoop;

[0008] A cleanup strategy is generated based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop.

[0009] The data tables in the Hadoop system are cleaned according to the cleanup strategy to perform storage cleanup on the Hadoop system.

[0010] Based on the historical access information, a cleanup feedback message is generated and sent to a preset terminal. The cleanup feedback message is used to optimize the storage of the Hadoop system.

[0011] To address the aforementioned technical problems, this application also provides a storage cleanup device for Hadoop, employing the following technical solution:

[0012] The log acquisition module is used to acquire SQL execution logs;

[0013] The information generation module is used to parse the SQL execution log to obtain historical access information for each data table in Hadoop;

[0014] The strategy generation module is used to generate a cleanup strategy based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop.

[0015] The data table cleanup module is used to clean up each data table in the Hadoop according to the cleanup strategy, so as to perform storage cleanup on the Hadoop.

[0016] The feedback generation module is used to generate cleanup feedback information based on the historical access information and send the cleanup feedback information to a preset terminal, wherein the cleanup feedback information is used to optimize the storage of the Hadoop.

[0017] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0018] Get the SQL execution log;

[0019] The SQL execution logs are parsed to obtain historical access information for each data table in Hadoop;

[0020] A cleanup strategy is generated based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop.

[0021] The data tables in the Hadoop system are cleaned according to the cleanup strategy to perform storage cleanup on the Hadoop system.

[0022] Based on the historical access information, a cleanup feedback message is generated and sent to a preset terminal. The cleanup feedback message is used to optimize the storage of the Hadoop system.

[0023] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0024] Get the SQL execution log;

[0025] The SQL execution logs are parsed to obtain historical access information for each data table in Hadoop;

[0026] A cleanup strategy is generated based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop.

[0027] The data tables in the Hadoop system are cleaned according to the cleanup strategy to perform storage cleanup on the Hadoop system.

[0028] Based on the historical access information, a cleanup feedback message is generated and sent to a preset terminal. The cleanup feedback message is used to optimize the storage of the Hadoop system.

[0029] Compared with existing technologies, the embodiments of this application have the following main advantages: SQL execution logs are obtained, which record the execution process of user-submitted SQL statements. Parsing these logs reveals the specific access status of each data table in Hadoop, thus obtaining historical access information. Based on this historical access information, data tables can be classified, and specific cleanup strategies for each type of data table can be generated. Each data table can be cleaned specifically according to the cleanup strategies, thereby reducing the storage resources occupied by Hadoop. Cleanup feedback information is generated based on historical access information. This feedback information is the result of user analysis and can be sent to a preset terminal to provide feedback to the user, enabling storage optimization of Hadoop and avoiding the waste of Hadoop storage resources caused by design flaws, thereby greatly saving Hadoop storage resources. Attached Figure Description

[0030] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0032] Figure 2 This is a flowchart of an embodiment of the storage cleanup method for Hadoop according to this application;

[0033] Figure 3 This is a schematic diagram of a storage cleanup apparatus for Hadoop according to an embodiment of this application;

[0034] Figure 4This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0038] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0040] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0041] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103. Server 105 can perform storage cleanup on the Hadoop cluster.

[0042] It should be noted that the storage cleanup method for Hadoop provided in this application embodiment is generally executed by the server, and correspondingly, the storage cleanup device for Hadoop is generally set in the server.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a storage cleanup method for Hadoop according to this application. The storage cleanup method for Hadoop includes the following steps:

[0045] Step S201: Obtain the SQL execution log.

[0046] In this embodiment, the storage cleanup method for Hadoop runs on electronic devices (e.g., Figure 1 The server shown can communicate with the terminal via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0047] Specifically, the server first retrieves the SQL execution log. During task execution on the big data platform, data tables can be read and written using SQL statements. SQL (Structured Query Language) is a programming language, a database query and programming language, used to access, query, update, and manage relational database systems. The SQL execution log is the historical record of SQL statement execution, used to record the execution process of SQL statements.

[0048] It should be emphasized that, to further ensure the privacy and security of the SQL execution logs, they can also be stored in a blockchain node.

[0049] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0050] Furthermore, before step S201 above, the process may include: obtaining the SQL statement submitted by the user; executing the SQL statement and monitoring the execution process of the SQL statement to obtain the SQL execution log.

[0051] Specifically, the server retrieves the SQL statements submitted by the user. In one embodiment, the IDE (Integrated Development Environment) serves as the entry point for users to submit SQL statements and process the data. The server submits the user-submitted SQL statements to Hadoop Yarn for scheduling, where Yarn is the component in Hadoop responsible for resource scheduling and task management. The SQL statements submitted by the user and executed in Yarn are recorded during the execution process, and the generated historical records are stored in the underlying HDFS audit file. Then, through a scheduled task, the HDFS audit file is stored in a Hive table to obtain the SQL execution log.

[0052] In this embodiment, the execution process is monitored when SQL statements are executed to obtain SQL execution logs, which can then be used to perform storage cleanup on Hadoop.

[0053] Step S202: Parse the SQL execution log to obtain historical access information for each data table in Hadoop.

[0054] Specifically, Hadoop stores a large number of data tables. The server parses the SQL execution logs to obtain the past access status of each data table, that is, the history of being read and written by SQL statements, thereby obtaining the historical access information of each data table.

[0055] Furthermore, step S202 above may include: parsing the SQL execution log to obtain the data tables associated with each SQL statement, as well as the access time and access count for each data table; and generating historical access information for each data table in Hadoop based on the obtained data tables and their corresponding access times and access counts.

[0056] Specifically, the server can parse the SQL execution log to obtain the data tables associated with each SQL statement during execution. Here, association refers to which data tables in which databases the SQL statement is involved in reading and writing operations. At the same time, it can also obtain the access time and number of accesses for each data table.

[0057] Based on the parsed data tables and their corresponding access times and counts, historical access information for each data table in Hadoop can be generated. This historical access information can include not only the table name, database, access time, and count, but also some attribute information about the table, such as whether it's a full table or a partitioned table, whether it's an offline Hive table or a real-time Hive table, which parts of the table are historical partitions and which are real-time partitions, and the access counts and times for both historical and real-time partitions. It can also include information calculated according to a preset calculation method, such as the number of times the table was accessed within a preset time period.

[0058] In one embodiment, the SQL execution logs are parsed using the SQL parser built into Alibaba's open-source tool Druid.

[0059] In this embodiment, the SQL execution log is parsed to obtain the associated data tables and their corresponding access times and access counts, thereby obtaining the historical access information of each data table and preparing data for storage cleanup based on historical access information.

[0060] Step S203: Generate a cleanup strategy based on historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in Hadoop.

[0061] Specifically, historical access information can include details of read and write operations for each data table and its different partitions, as well as the table's attribute information. Based on this historical access information, data tables in Hadoop can be categorized into several predefined table types, each with its own corresponding cleanup strategy. The cleanup strategy specifies how to clean up the data tables.

[0062] In one embodiment, each data table has a cleanup strategy. For data tables of the same type, their cleanup strategies are similar, but the specific parameters in the cleanup strategy may differ due to the different composition of the data tables.

[0063] Step S204: Clean up each data table in Hadoop according to the cleanup strategy to perform storage cleanup on Hadoop.

[0064] Specifically, for each data table in Hadoop, cleanup is performed according to the corresponding cleanup strategy. This cleanup can be performed on the entire data table or on a part of the data table. Cleanup is not limited to deletion operations; it can also include operations such as format conversion of the data in the data table.

[0065] Step S205: Generate cleanup feedback information based on historical access information and send the cleanup feedback information to a preset terminal. The cleanup feedback information is used to optimize Hadoop storage.

[0066] Specifically, user analysis can be performed based on historical access information. This analysis can involve analyzing the SQL statements executed by users to determine if there are any vulnerabilities in the SQL statements during execution or if there are any inappropriate aspects in the user-designed data tables. After the user analysis is completed, a cleanup feedback result is generated and sent to a preset terminal for display to the user or developers. The cleanup feedback result is fed back to the user, thereby optimizing Hadoop storage and avoiding the waste of Hadoop storage resources due to design flaws.

[0067] In this embodiment, SQL execution logs are obtained. These logs record the execution process of user-submitted SQL statements. Parsing them reveals the specific access status of each data table in Hadoop, thus obtaining historical access information. Based on this historical access information, the data tables can be categorized, and specific cleanup strategies for each type of data table can be generated. Each data table is then cleaned according to the cleanup strategies, reducing the storage resources occupied by Hadoop. Cleanup feedback information is generated based on the historical access information. This feedback information represents the user's analysis and can be sent to a preset terminal to provide feedback to the user. This allows for storage optimization of Hadoop, avoiding waste of Hadoop storage resources caused by design flaws, and significantly saving Hadoop storage resources.

[0068] Furthermore, the steps described above for cleaning up each data table in Hadoop according to the cleaning strategy may include: reading each cleaning strategy separately; deleting the data table associated with the cleaning strategy when the cleaning strategy is a table cleaning strategy; and compressing or merging the data table associated with the system cleaning strategy when the cleaning strategy is a system cleaning strategy.

[0069] Specifically, each generated cleanup strategy is read individually. Cleanup strategies can be categorized into table cleanup strategies and system cleanup strategies. Table cleanup strategies involve deleting data tables. Invalid or low-frequency data tables are identified through historical access information. Invalid data tables can be those that have never been accessed or are isolated from any task; low-frequency data tables can be those that have been accessed too infrequently within a preset time period. For table cleanup strategies, the associated data tables are deleted, thus achieving cleanup.

[0070] The system cleanup strategy can refer to the compression or merging of data tables. To do this, obtain the data tables associated with the system cleanup strategy and perform compression or merging operations on the data tables to complete the cleanup.

[0071] In this embodiment, the cleanup strategy includes a table deletion cleanup strategy and a system cleanup strategy of compression or merging, thereby performing targeted cleanup of the data tables in different ways.

[0072] Furthermore, when the cleanup strategy is a system cleanup strategy, the steps for compressing or merging the data tables associated with the system cleanup strategy may include: when the system cleanup strategy belongs to a first system cleanup strategy, merging the data tables associated with the first system cleanup strategy, wherein the data tables are MR underlying data tables; when the system cleanup strategy belongs to a second system cleanup strategy, compressing the data tables associated with the second system cleanup strategy, wherein the data tables are offline Hive data tables; when the system cleanup strategy belongs to a third system cleanup strategy, compressing the historical partitions in the data tables associated with the third system cleanup strategy, wherein the data tables are real-time Hive data tables.

[0073] Specifically, the system cleanup strategy can be divided into three types: the first system cleanup strategy, the second system cleanup strategy, and the third system cleanup strategy. Among them, the first system cleanup strategy targets the underlying data tables of MapReduce. The underlying data tables of MapReduce refer to the files generated at the lower level during the execution of MapReduce. These are small files. Although MapReduce data tables are useful, they can affect MapReduce performance, so they need to be merged periodically.

[0074] The second system cleanup strategy targets offline Hive tables, which are typically in TextFile format. This format can be converted to ORC format, thus compressing the tables. ORC format is columnar storage, which improves network transmission performance and SQL pruning capabilities, while reducing storage space usage.

[0075] The third system's cleanup strategy targets real-time Hive tables. Real-time Hive tables can include real-time partitions and historical partitions. Data in real-time partitions can change in real time, while data in historical partitions does not. Therefore, according to the third system's cleanup strategy, historical partitions can be compressed, that is, the data in historical partitions in textfile format can be converted to orc format.

[0076] In this embodiment, the system cleanup strategy performs merging, compression, or partial compression operations on data tables according to their different types, providing a variety of cleanup methods and saving storage resources.

[0077] Furthermore, step S205 may include: identifying invalid data tables and their corresponding task chains in Hadoop based on historical access information, and generating chain cleanup information; generating historical partition cleanup information for partitioned data tables in Hadoop based on historical access information; generating table transformation information for all data tables in Hadoop based on historical access information; generating statement feedback information based on SQL statements in historical access information; generating cleanup feedback information based on chain cleanup information, historical partition cleanup information, table transformation information, and statement feedback information, and sending the cleanup feedback information to a preset terminal.

[0078] Specifically, user analysis can be performed based on historical access information to uncover design flaws in Hadoop. First, invalid data tables in Hadoop can be identified, including low-frequency data tables, and the task chains associated with these tables can be queried. Since each task chain contains multiple tasks, the tasks within these chains may also be invalid due to the presence of invalid data tables. Therefore, chain cleanup information can be generated to remind users or the server to check or clean up the task chain.

[0079] For partitioned data tables in Hadoop (i.e., partitioned tables in the general sense), if historical access information reveals that a historical partition has not been accessed for a long time, a historical partition cleanup message can be generated to remind these historical partitions to be cleaned up.

[0080] If historical access information reveals that a full data table with a long time span (i.e., a full table in the usual sense) has been accessed, table conversion information is generated to remind users to convert the full data table into a partitioned data table. For example, if historical access information shows that a full data table containing data generated within two years has been read and written, since such a large time span of data is not usually used, a suggestion can be made to split the full data table and generate a partitioned data table.

[0081] Furthermore, the SQL statements in historical access information can be analyzed. When accessing partitioned data tables, if the filtering conditions do not include a partition field, a large amount of historical data will be accessed, resulting in a waste of computing resources. If this situation is discovered through historical access information, statement feedback information can be generated to remind users to correct vulnerabilities in the SQL statements.

[0082] Link cleanup information, historical partition cleanup information, table conversion information, and statement feedback information can be merged to generate cleanup feedback information, which is then sent to a preset terminal to inform the user of vulnerabilities in Hadoop.

[0083] In this embodiment, user analysis is performed based on historical access information to generate link cleanup information, historical partition cleanup information, table conversion information, and statement feedback information, thereby discovering vulnerabilities in Hadoop and providing alerts to avoid resource waste.

[0084] Furthermore, the aforementioned storage cleanup methods for Hadoop may also include: adjusting Hadoop based on cleanup feedback information, including link cleanup, historical partition cleanup, and table transformation.

[0085] Specifically, Hadoop can be adjusted based on the cleanup feedback information, including link cleanup, historical partition cleanup, and table transformation. Link cleanup can involve deleting or optimizing task links related to invalid data tables; historical partition cleanup can involve adding data retention time to historical partitions in partition tables and cleaning up expired historical data using scripts; table transformation can involve splitting the full data table and building its corresponding partition data table, thereby improving access efficiency.

[0086] The server can automatically adjust Hadoop based on cleanup feedback information, or it can be adjusted by the user.

[0087] In this embodiment, Hadoop is adjusted based on the cleanup feedback information, thereby reducing resource waste in Hadoop storage and usage.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0089] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0090] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a storage cleanup device for Hadoop, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0091] like Figure 3 As shown, the storage cleanup device 300 for Hadoop described in this embodiment includes: a log acquisition module 301, an information generation module 302, a policy generation module 303, a data table cleanup module 304, and a feedback generation module 305, wherein:

[0092] The log acquisition module 301 is used to acquire SQL execution logs.

[0093] The information generation module 302 is used to parse the SQL execution log to obtain historical access information for each data table in Hadoop.

[0094] The strategy generation module 303 is used to generate cleanup strategies based on historical access information. The cleanup strategies are the cleanup strategies corresponding to various data tables in Hadoop.

[0095] The data table cleanup module 304 is used to clean up each data table in Hadoop according to the cleanup strategy, so as to perform storage cleanup on Hadoop.

[0096] The feedback generation module 305 is used to generate cleanup feedback information based on historical access information and send the cleanup feedback information to a preset terminal. The cleanup feedback information is used to optimize Hadoop storage.

[0097] In this embodiment, SQL execution logs are obtained. These logs record the execution process of user-submitted SQL statements. Parsing them reveals the specific access status of each data table in Hadoop, thus obtaining historical access information. Based on this historical access information, the data tables can be categorized, and specific cleanup strategies for each type of data table can be generated. Each data table is then cleaned according to the cleanup strategies, reducing the storage resources occupied by Hadoop. Cleanup feedback information is generated based on the historical access information. This feedback information represents the user's analysis and can be sent to a preset terminal to provide feedback to the user. This allows for storage optimization of Hadoop, avoiding waste of Hadoop storage resources caused by design flaws, and significantly saving Hadoop storage resources.

[0098] In some optional implementations of this embodiment, the storage cleanup device 300 for Hadoop may further include: a statement acquisition module and an execution monitoring module, wherein:

[0099] The statement retrieval module is used to retrieve the SQL statements submitted by the user.

[0100] The execution monitoring module is used to execute SQL statements and monitor the execution process of SQL statements to obtain SQL execution logs.

[0101] In this embodiment, the execution process is monitored when SQL statements are executed to obtain SQL execution logs, which can then be used to perform storage cleanup on Hadoop.

[0102] In some optional implementations of this embodiment, the information generation module 302 may include: a log parsing submodule and an information generation submodule, wherein:

[0103] The log parsing submodule is used to parse the SQL execution log to obtain the data tables associated with each SQL statement, as well as the access time and number of accesses to each data table.

[0104] The information generation submodule is used to generate historical access information for each data table in Hadoop based on the obtained data tables and their corresponding access times and access counts.

[0105] In this embodiment, the SQL execution log is parsed to obtain the associated data tables and their corresponding access times and access counts, thereby obtaining the historical access information of each data table and preparing data for storage cleanup based on historical access information.

[0106] In some optional implementations of this embodiment, the data table cleanup module 304 may include: a strategy reading submodule, a first cleanup submodule, and a second cleanup submodule, wherein:

[0107] The policy reading submodule is used to read each cleanup policy separately.

[0108] The first cleanup submodule is used to delete the data tables associated with the cleanup strategy when the cleanup strategy belongs to the table cleanup strategy.

[0109] The second cleanup submodule is used to compress or merge the data tables associated with the system cleanup strategy when the cleanup strategy belongs to the system cleanup strategy.

[0110] In this embodiment, the cleanup strategy includes a table deletion cleanup strategy and a system cleanup strategy of compression or merging, thereby performing targeted cleanup of the data tables in different ways.

[0111] In some optional implementations of this embodiment, the second cleanup submodule may include: a first cleanup unit, a second cleanup unit, and a third cleanup unit, wherein:

[0112] The first cleanup unit is used to merge the data tables associated with the first system cleanup strategy when the system cleanup strategy belongs to the first system cleanup strategy. The data tables are the underlying data tables of MR.

[0113] The second cleanup unit is used to compress the data tables associated with the second system cleanup strategy when the system cleanup strategy belongs to the second system cleanup strategy. The data tables are offline Hive data tables.

[0114] The third cleanup unit is used to compress the historical partitions in the data table associated with the third system cleanup strategy when the system cleanup strategy belongs to the third system cleanup strategy. The data table is a real-time Hive data table.

[0115] In this embodiment, the system cleanup strategy performs merging, compression, or partial compression operations on data tables according to their different types, providing a variety of cleanup methods and saving storage resources.

[0116] In some optional implementations of this embodiment, the feedback generation module 305 may include: a link information generation submodule, a partition information generation submodule, a conversion information generation submodule, a statement information generation submodule, and a feedback information generation submodule, wherein:

[0117] The link information generation submodule is used to identify invalid data tables and their corresponding task links in Hadoop based on historical access information, so as to generate link cleanup information.

[0118] The partition information generation submodule is used to generate historical partition cleanup information for partitioned data tables in Hadoop based on historical access information.

[0119] The transformation information generation submodule is used to generate table transformation information for all data tables in Hadoop based on historical access information.

[0120] The statement information generation submodule is used to generate statement feedback information based on the SQL statements in the historical access information.

[0121] The feedback information generation submodule is used to generate cleanup feedback information based on link cleanup information, historical partition cleanup information, table conversion information, and statement feedback information, and send the cleanup feedback information to the preset terminal.

[0122] In this embodiment, user analysis is performed based on historical access information to generate link cleanup information, historical partition cleanup information, table conversion information, and statement feedback information, thereby discovering vulnerabilities in Hadoop and providing alerts to avoid resource waste.

[0123] In some optional implementations of this embodiment, the storage cleanup device 300 for Hadoop may further include: an adjustment module, wherein:

[0124] The tuning module is used to adjust Hadoop based on cleanup feedback information. Tuning includes link cleanup, historical partition cleanup, and table transformation.

[0125] In this embodiment, Hadoop is adjusted based on the cleanup feedback information, thereby reducing resource waste in Hadoop storage and usage.

[0126] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0127] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0128] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0129] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for storage cleanup methods for Hadoop. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0130] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the storage cleanup method for Hadoop.

[0131] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0132] The computer device provided in this embodiment can execute the above-described storage cleanup method for Hadoop. The storage cleanup method for Hadoop described here can be any of the storage cleanup methods for Hadoop described in the various embodiments above.

[0133] In this embodiment, SQL execution logs are obtained. These logs record the execution process of user-submitted SQL statements. Parsing them reveals the specific access status of each data table in Hadoop, thus obtaining historical access information. Based on this historical access information, the data tables can be categorized, and specific cleanup strategies for each type of data table can be generated. Each data table is then cleaned according to the cleanup strategies, reducing the storage resources occupied by Hadoop. Cleanup feedback information is generated based on the historical access information. This feedback information represents the user's analysis and can be sent to a preset terminal to provide feedback to the user. This allows for storage optimization of Hadoop, avoiding waste of Hadoop storage resources caused by design flaws, and significantly saving Hadoop storage resources.

[0134] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the storage cleanup method for Hadoop as described above.

[0135] In this embodiment, SQL execution logs are obtained. These logs record the execution process of user-submitted SQL statements. Parsing them reveals the specific access status of each data table in Hadoop, thus obtaining historical access information. Based on this historical access information, the data tables can be categorized, and specific cleanup strategies for each type of data table can be generated. Each data table is then cleaned according to the cleanup strategies, reducing the storage resources occupied by Hadoop. Cleanup feedback information is generated based on the historical access information. This feedback information represents the user's analysis and can be sent to a preset terminal to provide feedback to the user. This allows for storage optimization of Hadoop, avoiding waste of Hadoop storage resources caused by design flaws, and significantly saving Hadoop storage resources.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0137] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A storage cleanup method for Hadoop, characterized in that, Includes the following steps: Get the SQL execution log; The SQL execution logs are parsed to obtain historical access information for each data table in Hadoop; A cleanup strategy is generated based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop. The data tables in the Hadoop system are cleaned according to the cleanup strategy to perform storage cleanup on the Hadoop system. Based on the historical access information, cleanup feedback information is generated and sent to a preset terminal. The cleanup feedback information is used to optimize the storage of the Hadoop system. The step of cleaning each data table in the Hadoop according to the cleaning strategy includes: Read each cleanup policy separately; When the cleanup strategy is a table cleanup strategy, delete the data table associated with the cleanup strategy; When the cleanup strategy is a system cleanup strategy, the data tables associated with the system cleanup strategy are compressed or merged. When the cleanup strategy is a system cleanup strategy, the steps of compressing or merging the data tables associated with the system cleanup strategy include: When the system cleanup strategy belongs to the first system cleanup strategy, the data tables associated with the first system cleanup strategy are merged, wherein the data tables are the underlying data tables of MR. When the system cleanup strategy belongs to the second system cleanup strategy, the data table associated with the second system cleanup strategy is compressed, wherein the data table is an offline Hive data table; When the system cleanup strategy belongs to the third system cleanup strategy, the historical partitions in the data table associated with the third system cleanup strategy are compressed, wherein the data table is a real-time Hive data table.

2. The storage cleanup method for Hadoop according to claim 1, characterized in that, Before the step of obtaining the SQL execution log, the following is also included: Retrieve the SQL statement submitted by the user; The SQL statement is executed, and the execution process is monitored to obtain the SQL execution log.

3. The storage cleanup method for Hadoop according to claim 1, characterized in that, The step of parsing the SQL execution log to obtain historical access information for each data table in Hadoop includes: The SQL execution log is parsed to obtain the data tables associated with each SQL statement, as well as the access time and number of accesses to each data table; Based on the obtained data tables and their corresponding access times and access counts, historical access information for each data table in Hadoop is generated.

4. The storage cleanup method for Hadoop according to claim 1, characterized in that, The step of generating cleanup feedback information based on the historical access information and sending the cleanup feedback information to a preset terminal includes: Based on the historical access information, invalid data tables in the Hadoop system and their corresponding task chains are identified to generate chain cleanup information. Based on the historical access information, generate historical partition cleanup information for the partitioned data table in the Hadoop; Based on the historical access information, generate table transformation information for all data tables in the Hadoop dataset; Based on the SQL statements in the historical access information, generate statement feedback information; Cleanup feedback information is generated based on the link cleanup information, the historical partition cleanup information, the table conversion information, and the statement feedback information, and the cleanup feedback information is sent to a preset terminal.

5. The storage cleanup method for Hadoop according to claim 4, characterized in that, The method further includes: The Hadoop system is adjusted based on the cleanup feedback information. The adjustments include link cleanup, historical partition cleanup, and table transformation.

6. A storage cleanup device for Hadoop, characterized in that, include: The log acquisition module is used to acquire SQL execution logs; The information generation module is used to parse the SQL execution log to obtain historical access information for each data table in Hadoop; The strategy generation module is used to generate a cleanup strategy based on the historical access information, wherein the cleanup strategy is the cleanup strategy corresponding to various data tables in the Hadoop. The data table cleanup module is used to clean up each data table in the Hadoop according to the cleanup strategy, so as to perform storage cleanup on the Hadoop. The feedback generation module is used to generate cleanup feedback information based on the historical access information and send the cleanup feedback information to a preset terminal, wherein the cleanup feedback information is used to optimize the storage of the Hadoop. The data table cleanup module includes a strategy reading submodule, a first cleanup submodule, and a second cleanup submodule. The strategy reading submodule is used to read each cleanup strategy separately; The first cleanup submodule is used to delete the data table associated with the cleanup strategy when the cleanup strategy belongs to the table cleanup strategy; The second cleaning submodule is used to compress or merge the data tables associated with the system cleaning strategy when the cleaning strategy belongs to the system cleaning strategy; The second cleaning submodule includes a first cleaning unit, a second cleaning unit, and a third cleaning unit; The first cleaning unit is used to merge the data tables associated with the first system cleaning strategy when the system cleaning strategy belongs to the first system cleaning strategy, wherein the data table is the underlying data table of MR. The second cleanup unit is used to compress the data table associated with the second system cleanup strategy when the system cleanup strategy belongs to the second system cleanup strategy, wherein the data table is an offline Hive data table; The third cleanup unit is used to compress the historical partitions in the data table associated with the third system cleanup strategy when the system cleanup strategy belongs to the third system cleanup strategy, wherein the data table is a real-time Hive data table.

7. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the storage cleanup method for Hadoop as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the storage cleanup method for Hadoop as described in any one of claims 1 to 5.

Images