Cold data management method applied to storage and calculation separation and related equipment

Abstract

The embodiment of the invention belongs to the technical field of data processing, and relates to a cold data management method applied to storage and calculation separation and related equipment, and the method comprises the steps: obtaining a cold data management request carrying a target system identifier; calling a target HDFS file system corresponding to the target system identifier, and exporting a mirror image file of the target HDFS file system according to the hdfs oiv command; performing format conversion operation on the mirror image file to obtain a standard mirror image file; creating a first Hive table, and setting partitions in the first Hive table by taking day as a unit; importing the standard mirror image file into a corresponding partition of the first Hive table according to a day as a unit; performing cold data query operation on the first Hive table to obtain a cold data path; and creating a second Hive table, and moving the standard mirror text corresponding to the cold data path into the second Hive table. According to the method, hundreds of millions of metadata of the HDFS are analyzed by using the super-large-scale parallel data processing capability of the big data platform, so that the reliability and the expandability are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a cold data management method and related equipment applied to storage and computing separation. Background Art

[0002] At present, the industry usually adopts the method of maintaining the full path record of the HDFS directory tree in the data scheduling system, monitoring the changes of HDFS metadata, and synchronizing HDFS metadata information in real time. In the data scheduling system, the HDFS directory tree information is periodically traversed to search for paths that meet the cold data (for example, no access records for more than 30 days are considered cold data).

[0003] However, the data scheduling system usually uses a relational database to store the metadata of HFDS, but the upper limit of a single table in a relational database is often not high. For example, the data of a single MYSQL table exceeds 5 million rows. Even if an index is added, the query performance is still a bottleneck. In actual production, especially in larger clusters, the cold data in HDFS often exceeds hundreds of millions. This is a huge performance bottleneck for the scheduling system using a relational database.

[0004] It can be seen that the traditional data scheduling system has performance limitations. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a cold data management method and related equipment applied to storage and computing separation to solve the problem of performance limitations in traditional data scheduling systems.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a cold data management method applied to storage and computing separation, which adopts the following technical solutions:

[0007] Obtain a cold data management request carrying a target system identifier;

[0008] Calling a target HDFS file system corresponding to the target system identifier, and exporting an image file of the target HDFS file system according to an hdfs oiv command;

[0009] Performing a format conversion operation on the image file to obtain a standard image file;

[0010] Create a first Hive table, and set partitions in the first Hive table in units of days;

[0011] Importing the standard image file into the corresponding partition of the first Hive table on a daily basis;

[0012] Perform a cold data query operation on the first Hive table to obtain a cold data path;

[0013] A second Hive table is created, and the standard image text corresponding to the cold data path is moved to the second Hive table.

[0014] Furthermore, the step of performing a cold data query operation on the first Hive table to obtain a cold data path specifically includes the following steps:

[0015] Obtaining the access time of each standard image file from the first Hive table;

[0016] The flag bit of the standard image file whose access time exceeds the preset number of days threshold is set to 0;

[0017] The flag bit of the standard image file whose access time does not exceed the preset number of days threshold is set to 1;

[0018] The storage path of the standard image file with a flag bit of 0 is determined as the cold data path.

[0019] Furthermore, after the step of setting the flag of the standard image file whose access time exceeds the preset number of days as cold data, the method further includes the following steps:

[0020] Calculate the sum of the flag parameters of each partition respectively;

[0021] The partitions whose flag bit parameters sum to 0 are identified as cold data partitions.

[0022] Furthermore, the step of performing a cold data query operation on the first Hive table to obtain a cold data path specifically includes the following steps:

[0023] Obtain metadata information of all files in the first Hive table;

[0024] The metadata information is classified into hot and cold data according to preset cold data judgment rules and a decision tree algorithm to obtain the cold data path.

[0025] Furthermore, the step of creating a second Hive table and moving the standard image text corresponding to the cold data path to the second Hive table specifically includes the following steps:

[0026] Analyzing the association patterns between the cold data paths according to an association rule mining algorithm to obtain a cold data file group with similar characteristics;

[0027] Count the total amount of data and average file size indicators of each cold data file group to determine the cold data scale threshold;

[0028] If the size of the cold data file group exceeds a threshold, all file paths in the cold data file group are added to a cold data migration task queue;

[0029] According to the cold data file path in the migration task queue, the cold data file is transferred to the second Hive table.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides a cold data management device applied to storage and computing separation, which adopts the following technical solution:

[0031] A request acquisition module is used to acquire a cold data management request carrying a target system identifier;

[0032] A file export module, used to call a target HDFS file system corresponding to the target system identifier, and export an image file of the target HDFS file system according to an hdfs oiv command;

[0033] A format conversion module, used for performing a format conversion operation on the image file to obtain a standard image file;

[0034] A first Hive table creation module, used to create a first Hive table and set partitions in the first Hive table in units of days;

[0035] A file import module, used for importing the standard image file into the corresponding partition of the first Hive table based on days;

[0036] A cold data query module, used to perform a cold data query operation on the first Hive table to obtain a cold data path;

[0037] The second Hive table creation module is used to create a second Hive table and move the standard image text corresponding to the cold data path to the second Hive table.

[0038] Furthermore, the cold data query module includes:

[0039] An access time acquisition submodule, used to obtain the access time of each standard image file from the first Hive table;

[0040] A first flag setting submodule, used to set the flag of the standard image file whose access time exceeds a preset number of days to 0;

[0041] A second flag bit setting submodule, used to set the flag bit of the standard image file whose access time does not exceed the preset day threshold to 1;

[0042] The cold data path determination submodule is used to determine the storage path of the standard image file whose flag bit is 0 as the cold data path.

[0043] Furthermore, the cold data query module also includes:

[0044] The parameter and calculation submodule is used to calculate the sum of the flag parameters of each partition respectively;

[0045] The cold data partition determination submodule is used to determine the partition whose flag bit parameters sum to 0 as a cold data partition.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0047] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the cold data management method applied to storage and computing separation as described above are implemented.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0049] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the cold data management method applied to storage and computing separation as described above are implemented.

[0050] The present application provides a cold data management method for storage and computing separation, including: obtaining a cold data management request carrying a target system identifier; calling a target HDFS file system corresponding to the target system identifier, and exporting the image file of the target HDFS file system according to the hdfs oiv command; performing a format conversion operation on the image file to obtain a standard image file; creating a first Hive table, and setting a partition in units of days in the first Hive table; importing the standard image file into the corresponding partition of the first Hive table according to days; performing a cold data query operation on the first Hive table to obtain a cold data path; creating a second Hive table, and moving the standard image text corresponding to the cold data path to the second Hive table. Compared with the prior art, the present application uses the ultra-large-scale parallel data processing capabilities of the big data platform to analyze the metadata of hundreds of millions of HDFS. Both the scale of data and the processing speed far exceed general data management systems, such as Intel's open source HDFS SmartStorageManager; this solution has high reliability and good scalability. Even if the metadata of HDFS continues to expand, it can be easily dealt with using the large-scale processing capabilities of Hive, Spark, and MapReduce. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0053] Figure 2 This is a flowchart of the implementation of the cold data management method for storage and computing separation provided in Example 1 of the present application;

[0054] Figure 3 It is a structural diagram of a cold data management device for storage and computing separation provided in Embodiment 2 of the present application;

[0055] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0061] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0062] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0063] It should be noted that the cold data management method applied to storage and computing separation provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the cold data management device applied to storage and computing separation is generally set in the server / terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] Embodiment 1

[0066] Continue to refer Figure 2 , shows a flow chart of an embodiment of a cold data management method for storage and computing separation according to the present application. The cold data management method for storage and computing separation includes: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0067] In step S201, a cold data management request carrying a target system identifier is obtained.

[0068] In an embodiment of the present application, the request includes an identifier of a target system where the cold data to be managed is located, wherein the identifier may be a name, a number, or any information that can uniquely identify the target system.

[0069] In step S202, the target HDFS file system corresponding to the target system identifier is called, and the image file of the target HDFS file system is exported according to the hdfs oiv command.

[0070] In the embodiment of the present application, according to the target system identifier in the request, the system needs to locate and connect to the corresponding HDFS file system. This involves looking up configuration information to determine the HDFS cluster address, port and authentication information of the target system.

[0071] In the embodiment of the present application, the Offline Image Viewer (oiv) tool of HDFS can be used to export the image of the target HDFS file system. This image file contains the metadata information of the file system.

[0072] In step S203, a format conversion operation is performed on the image file to obtain a standard image file.

[0073] In the embodiment of the present application, the exported image file may not be in a format that can be directly used for subsequent processing, so format conversion is required. This may include converting the file from XML to CSV, Parquet, or other formats that are more suitable for Hive processing.

[0074] In step S204, a first Hive table is created, and partitions are set in the first Hive table in units of days.

[0075] In the embodiment of the present application, a new table is created in Hive to store the converted image file data. In order to improve query efficiency, the table is designed to be partitioned by day, so that data of a specific date can be located more quickly.

[0076] In step S205, the standard image file is imported into the corresponding partition of the first Hive table based on the day.

[0077] In the embodiment of the present application, the converted standard image file data is imported into the corresponding partition of the Hive table. This usually involves writing a Hive load (LOAD) statement to associate the data with a specific partition.

[0078] In step S206, a cold data query operation is performed on the first Hive table to obtain a cold data path.

[0079] In an embodiment of the present application, a query is executed to identify cold data stored in a Hive table, wherein the cold data can be determined based on the access time or modification time of the data. The query result will return the path of the file or directory that is considered to be cold data.

[0080] In step S207, a second Hive table is created, and the standard image text corresponding to the cold data path is moved to the second Hive table.

[0081] In the embodiment of the present application, a new Hive table is created to specifically store cold data. Then, the relevant records identified as cold data in the first Hive table are moved to the second Hive table. Specifically, a Hive INSERT statement can be written to achieve data movement.

[0082] In an embodiment of the present application, a cold data management method for storage and computing separation is provided, including: obtaining a cold data management request carrying a target system identifier; calling a target HDFS file system corresponding to the target system identifier, and exporting the image file of the target HDFS file system according to the hdfs oiv command; performing a format conversion operation on the image file to obtain a standard image file; creating a first Hive table, and setting partitions in units of days in the first Hive table; importing the standard image file into the corresponding partition of the first Hive table according to days; performing a cold data query operation on the first Hive table to obtain a cold data path; creating a second Hive table, and moving the standard image text corresponding to the cold data path to the second Hive table. Compared with the prior art, the present application uses the ultra-large-scale parallel data processing capabilities of the big data platform to analyze the metadata of hundreds of millions of HDFS. Both the scale of data and the processing speed far exceed general data management systems, such as Intel's open source HDFS SmartStorageManager; this solution has high reliability and good scalability. Even if the metadata of HDFS continues to expand, it can be easily dealt with using the large-scale processing capabilities of Hive, Spark, and MapReduce.

[0083] In some optional implementations of the embodiments of the present application, the above step S206 specifically includes the following steps:

[0084] Get the access time of each standard image file from the first Hive table;

[0085] Set the flag bit of the standard image file whose access time exceeds the preset number of days to 0;

[0086] Set the flag bit of the standard image file whose access time does not exceed the preset number of days to 1;

[0087] The storage path of the standard image file with the flag bit set to 0 is determined as the cold data path.

[0088] In the embodiment of the present application, a Hive query needs to be executed to retrieve the access time of each standard image file from the first Hive table. This access time may be a field that records the date and time when the file was last accessed. The result of the query will be a list containing the file path and the corresponding access time.

[0089] In the embodiment of the present application, it is necessary to traverse the query results and compare the access time of each file with the current time. If the access time of a file exceeds a preset threshold of days (e.g., 30 days, 60 days, or 90 days, etc.), the flag of the file is set to 0. This flag is a field used to identify whether the file is cold data, and 0 indicates cold data.

[0090] In the embodiment of the present application, for files whose access time does not exceed the preset day threshold, their flag bits are set to 1. This means that these files are not cold data and may still be frequently accessed or used.

[0091] In the embodiment of the present application, the query results are traversed again to filter out files with a flag bit of 0. The storage paths of these files are cold data paths, which contain standard mirror files that are considered to be cold data. These paths will be used for subsequent processing, such as moving cold data to a dedicated storage location or archiving it.

[0092] In the embodiment of the present application, the system can effectively identify cold data in HDFS and classify it according to the access time of the file, which helps to optimize the use of storage resources and improve the efficiency of data management.

[0093] In some optional implementations of the embodiments of the present application, after the step of setting the flag bit of the standard image file whose access time exceeds the preset day threshold as cold data, the following steps are also included:

[0094] Calculate the sum of the flag parameters of each partition respectively;

[0095] The partitions whose flag bit parameters sum to 0 are identified as cold data partitions.

[0096] In an embodiment of the present application, the system needs to sum the flag parameters of each partition in the Hive table. The flag parameter is previously set for each standard image file and is used to identify whether the file is cold data (0 means cold data, 1 means not). Therefore, by summing the flags of all files in each partition, the number of cold data files in the partition can be obtained.

[0097] In the embodiment of the present application, after calculating the sum of the flag parameters of each partition, the system needs to determine which partitions are cold data partitions. If the sum of the flag parameters of a partition is 0, it means that all files in the partition are marked as cold data. Therefore, such a partition can be confirmed as a cold data partition.

[0098] In the embodiment of the present application, the system can effectively identify which partitions are cold data partitions, that is, all files in those partitions have not been accessed for a long time. This is very useful for data management and storage optimization, because the system can move these cold data partitions to lower-cost storage devices, or archive them to free up more expensive storage space for more needed data. At the same time, this also helps to reduce the cost of data backup and recovery, because the backup frequency of cold data partitions can be lower.

[0099] In some optional implementations of the embodiments of the present application, the above step S206 specifically includes the following steps:

[0100] Get the metadata information of all files in the first Hive table;

[0101] According to the preset cold data judgment rules and decision tree algorithm, the metadata information is classified into hot and cold data to obtain the cold data path.

[0102] In the embodiment of the present application, metadata information related to all files is retrieved from the Hive table. Metadata is data about data, which describes the characteristics of the data, such as file size, creation time, modification time, access time, file type, storage location, etc. In this scenario, metadata is crucial to determine which files are cold data.

[0103] In an embodiment of the present application, the system uses preset cold data judgment rules to analyze the metadata information retrieved from the Hive table. These rules may be based on the access time, modification time, usage frequency or other relevant indicators of the file. In order to classify the data more accurately, the system may also apply a decision tree algorithm. The decision tree algorithm is a machine learning algorithm that can learn and make decisions based on a given data set and a series of rules. In this case, the decision tree algorithm will help the system classify files as hot data (frequently accessed data) or cold data (data that has not been accessed for a long time) based on the metadata information.

[0104] In the embodiment of the present application, the system determines which files are cold data based on the classification results of the decision tree algorithm and retrieves the storage paths of these files. These paths are the so-called "cold data paths", which point to the physical or logical locations where the cold data is stored. With this information, the system can further manage the cold data, such as moving, archiving or deleting, to optimize the use of storage resources.

[0105] In the embodiment of the present application, the data is classified using preset rules and decision tree algorithms, and the storage path of the cold data is finally determined, which is crucial for implementing effective data storage and management strategies, especially when processing large amounts of data.

[0106] In some optional implementations of the embodiments of the present application, the above step S207 specifically includes the following steps:

[0107] Analyze the association patterns between cold data paths according to the association rule mining algorithm to obtain cold data file groups with similar characteristics;

[0108] Count the total amount of data and average file size indicators of each cold data file group to determine the cold data scale threshold;

[0109] If the size of the cold data file group exceeds the threshold, all file paths in the cold data file group are added to the cold data migration task queue;

[0110] According to the cold data file path in the migration task queue, the cold data file is transferred to the second Hive table.

[0111] In the present application embodiment, association rule mining is a data mining technique used to discover interesting relationships between variables in large data sets. In this scenario, the algorithm will try to find cold data files with similar characteristics (such as access time, file type, size, storage location, etc.) and group them together. In this way, the system can manage these cold data more effectively because they have similar attributes and characteristics.

[0112] In the embodiment of the present application, after obtaining the cold data file groups, the system needs to count the total amount of data and average file size indicators of each group. These indicators are very important for understanding the scale and characteristics of cold data. Based on these statistical information, the system can set a cold data scale threshold to determine which cold data file groups are large enough to be migrated or perform other management operations.

[0113] In the embodiment of the present application, the system checks whether the size of each cold data file group exceeds a previously set threshold. If the size of a group exceeds the threshold, the paths of all files in the group will be added to the cold data migration task queue. This queue is a list of pending tasks that contains the path information of the cold data files that need to be migrated.

[0114] In an embodiment of the present application, the system transfers the cold data files from the current Hive table (the first Hive table) to another Hive table (the second Hive table) according to the file path information in the migration task queue. This transfer process may involve copying, moving, or reorganizing the data to ensure that the cold data is effectively managed and stored. By migrating these cold data to a dedicated Hive table, the system can optimize the use of storage resources and reduce data management costs.

[0115] In the embodiment of the present application, an association rule mining algorithm is used to group cold data, count the size and characteristics of each group, determine the size threshold, and migrate cold data files that exceed the threshold to another Hive table. This helps to achieve a more efficient data storage and management strategy, especially when dealing with a large amount of cold data.

[0116] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0117] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0119] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0120] Embodiment 2

[0121] Further references Figure 3 , as a response to the above Figure 2The present application provides an embodiment of a cold data management device for storage and computing separation. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0122] like Figure 3 As shown, the cold data management device 200 applied to storage and computing separation in the embodiment of the present application includes:

[0123] The request acquisition module 210 is used to acquire a cold data management request carrying a target system identifier;

[0124] The file export module 220 is used to call the target HDFS file system corresponding to the target system identifier and export the image file of the target HDFS file system according to the hdfs oiv command;

[0125] The format conversion module 230 is used to perform a format conversion operation on the image file to obtain a standard image file;

[0126] A first Hive table creation module 240 is used to create a first Hive table and set partitions in the first Hive table in units of days;

[0127] A file import module 250, used to import the standard image file into the corresponding partition of the first Hive table based on the day;

[0128] A cold data query module 260 is used to perform a cold data query operation on the first Hive table to obtain a cold data path;

[0129] The second Hive table creation module 270 is used to create a second Hive table and move the standard image text corresponding to the cold data path to the second Hive table.

[0130] In an embodiment of the present application, a cold data management device 200 for storage and computing separation is provided, including: a request acquisition module 210, used to acquire a cold data management request carrying a target system identifier; a file export module 220, used to call a target HDFS file system corresponding to the target system identifier, and export the image file of the target HDFS file system according to the hdfs oiv command; a format conversion module 230, used to perform a format conversion operation on the image file to obtain a standard image file; a first Hive table creation module 240, used to create a first Hive table, and set a partition in the first Hive table in units of days; a file import module 250, used to import the standard image file into the corresponding partition of the first Hive table in units of days; a cold data query module 260, used to perform a cold data query operation on the first Hive table to obtain a cold data path; a second Hive table creation module 270, used to create a second Hive table, and move the standard image text corresponding to the cold data path to the second Hive table. Compared with the prior art, the present application uses the ultra-large-scale parallel data processing capability of the big data platform to analyze the metadata of hundreds of millions of HDFS. Both the data size and processing speed far exceed those of general data management systems, such as Intel's open source HDFS SmartStorageManager. This solution is highly reliable and scalable. Even if the metadata of HDFS continues to expand, it can be easily handled using the large-scale processing capabilities of Hive, Spark, and MapReduce.

[0131] In some optional implementations of the embodiments of the present application, the cold data query module includes:

[0132] An access time acquisition submodule, used to obtain the access time of each standard image file from the first Hive table;

[0133] The first flag setting submodule is used to set the flag of the standard image file whose access time exceeds a preset number of days to 0;

[0134] The second flag setting submodule is used to set the flag of the standard image file whose access time does not exceed the preset number of days threshold to 1;

[0135] The cold data path determination submodule is used to determine the storage path of the standard image file whose flag bit is 0 as the cold data path.

[0136] In some optional implementations of the embodiments of the present application, the cold data query module further includes:

[0137] The parameter and calculation submodule is used to calculate the sum of the flag parameters of each partition respectively;

[0138] The cold data partition determination submodule is used to determine the partition whose flag bit parameters sum to 0 as a cold data partition.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0140] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for the cold data management method for storage and computing separation. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or are to be output.

[0143] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as running the computer-readable instructions of the cold data management method for storage and computing separation.

[0144] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0145] The computer device provided by this application uses the ultra-large-scale parallel data processing capabilities of the big data platform to analyze the metadata of hundreds of millions of HDFS. Both the scale of data and the processing speed far exceed general data management systems, such as Intel's open source HDFS SmartStorageManager; this solution is highly reliable and scalable. Even if the metadata of HDFS continues to expand, it can be easily handled using the large-scale processing capabilities of Hive, Spark, and MapReduce.

[0146] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the cold data management method applied to storage and computing separation as described above.

[0147] The computer-readable storage medium provided by this application uses the ultra-large-scale parallel data processing capabilities of the big data platform to analyze metadata of hundreds of millions of HDFS. Both the scale of data and the processing speed far exceed general data management systems, such as Intel's open source HDFS SmartStorageManager; this solution is highly reliable and scalable, and even if the metadata of HDFS continues to expand, it can be easily handled using the large-scale processing capabilities of Hive, Spark, and MapReduce.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A cold data management method applied to storage and computing separation, characterized in that: The steps include: Obtain a cold data management request carrying a target system identifier; Calling a target HDFS file system corresponding to the target system identifier, and exporting an image file of the target HDFS file system according to the hdfs oiv command; Performing a format conversion operation on the image file to obtain a standard image file; Create a first Hive table, and set partitions in the first Hive table in units of days; Importing the standard image file into the corresponding partition of the first Hive table on a daily basis; Perform a cold data query operation on the first Hive table to obtain a cold data path; A second Hive table is created, and the standard image text corresponding to the cold data path is moved to the second Hive table.

2. The cold data management method for storage and computing separation according to claim 1 is characterized in that: The step of performing a cold data query operation on the first Hive table to obtain a cold data path specifically includes the following steps: Obtaining the access time of each standard image file from the first Hive table; The flag bit of the standard image file whose access time exceeds the preset number of days threshold is set to 0; The flag bit of the standard image file whose access time does not exceed the preset number of days threshold is set to 1; The storage path of the standard image file with a flag bit of 0 is determined as the cold data path.

3. The cold data management method for storage and computing separation according to claim 2 is characterized in that: After the step of setting the flag of the standard image file whose access time exceeds the preset number of days as cold data, the method further includes the following steps: Calculate the sum of the flag parameters of each partition respectively; The partitions whose flag bit parameters sum to 0 are identified as cold data partitions.

4. The cold data management method for storage and computing separation according to claim 1 is characterized in that: The step of performing a cold data query operation on the first Hive table to obtain a cold data path specifically includes the following steps: Obtain metadata information of all files in the first Hive table; The metadata information is classified into hot and cold data according to preset cold data judgment rules and a decision tree algorithm to obtain the cold data path.

5. The cold data management method for storage and computing separation according to claim 4 is characterized in that: The step of creating a second Hive table and moving the standard image text corresponding to the cold data path to the second Hive table specifically includes the following steps: Analyzing the association patterns between the cold data paths according to an association rule mining algorithm to obtain a cold data file group with similar characteristics; Count the total amount of data and average file size indicators of each cold data file group to determine the cold data scale threshold; If the size of the cold data file group exceeds a threshold, all file paths in the cold data file group are added to a cold data migration task queue; According to the cold data file path in the migration task queue, the cold data file is transferred to the second Hive table.

6. A cold data management device for storage and computing separation, characterized in that: include: A request acquisition module is used to acquire a cold data management request carrying a target system identifier; A file export module, used for calling a target HDFS file system corresponding to the target system identifier, and exporting an image file of the target HDFS file system according to an hdfsoiv command; A format conversion module, used for performing a format conversion operation on the image file to obtain a standard image file; A first Hive table creation module, used to create a first Hive table and set partitions in the first Hive table in units of days; A file import module, used for importing the standard image file into the corresponding partition of the first Hive table based on days; A cold data query module, used to perform a cold data query operation on the first Hive table to obtain a cold data path; The second Hive table creation module is used to create a second Hive table and move the standard image text corresponding to the cold data path to the second Hive table.

7. The cold data management device for storage and computing separation according to claim 6 is characterized in that: The cold data query module includes: An access time acquisition submodule, used to obtain the access time of each standard image file from the first Hive table; A first flag setting submodule, used to set the flag of the standard image file whose access time exceeds a preset number of days to 0; A second flag setting submodule, used to set the flag of the standard image file whose access time does not exceed a preset number of days threshold to 1; The cold data path determination submodule is used to determine the storage path of the standard image file whose flag bit is 0 as the cold data path.

8. The cold data management device for storage and computing separation according to claim 6 is characterized in that: The cold data query module also includes: The parameter and calculation submodule is used to calculate the sum of the flag parameters of each partition respectively; The cold data partition determination submodule is used to determine the partition whose flag bit parameters sum to 0 as a cold data partition.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the cold data management method applied to storage and computing separation as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the cold data management method for storage and computing separation according to any one of claims 1 to 5.

Images