Cold data retrieval method and apparatus, electronic device, and storage medium

Abstract

The application provides a cold data retrieval method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining data information to be searched; constructing a hive external table according to the data information to be searched; determining cold data, a data type of the cold data and a target storage partition storing the cold data according to an access time of the cold data to be searched and the hive external table; determining target hot data and a distribution of the target hot data in each storage file in the target storage partition according to the data type of the cold data; determining a display order of each storage file in the target storage partition according to the distribution of the target hot data in each storage file in the target storage partition; and displaying the cold data in each storage file according to the display order of each storage file in the target storage partition. The method provided by the application can quickly determine the distribution of the cold data without invasion, and display the retrieval order of the cold data for the user according to the distribution, thereby improving the accuracy of cold data retrieval.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a cold data retrieval method, apparatus, electronic device and storage medium. Background Technology

[0002] In the field of big data processing, as the amount of data stored in distributed file systems continues to grow, significant differences in access frequency will appear. In the distributed file system, data that is not frequently used is called cold data, while data that is frequently used (accessed frequently) is called hot data. The existence of a large amount of cold data is a waste of system resources and will also affect the stability of system storage space.

[0003] When the stability of the system storage space deteriorates, the system users can only be notified to identify the cold data themselves and perform cleanup or transfer operations. Specifically, the method of traversing the file list is used to recursively obtain the directory structure tree of the entire distributed file system to obtain the list of all files, thereby determining the information of cold data in all file lists, and thus finding the storage location of the corresponding cold data in the system.

[0004] However, this approach involves too much manual intervention, and it is difficult for users to determine which data is cold data when searching for data, resulting in the retrieval of cold data being less effective and efficient than expected. Summary of the Invention

[0005] This application provides a cold data retrieval method, apparatus, electronic device, and storage medium to solve the problem of low retrieval efficiency in the process of retrieving cold data.

[0006] Firstly, this application provides a cold data retrieval method, including:

[0007] Retrieve the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item.

[0008] Based on the data to be searched, construct a Hive external table;

[0009] Based on the access time of the cold data to be searched and the Hive external table, determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored;

[0010] Based on the data type of the cold data, determine the target hot data and the distribution of the target hot data in each storage file in the target storage partition. The data type of the target hot data is the same as that of the cold data.

[0011] The display order of each storage file in the target storage partition is determined based on the distribution of the target hot data in each storage file in the target storage partition.

[0012] Based on the display order of the storage files in the target storage partition, display the cold data in each storage file.

[0013] In this embodiment of the application, obtaining the data information to be searched includes:

[0014] Receive data information from the data storage server, which is the server that needs to search for cold data information in the data information;

[0015] The data information on the data storage server is parsed to obtain the data information to be searched.

[0016] In this embodiment of the application, receiving data information from the data storage server includes:

[0017] Determine the file search server's permission information and transmission port address information. The file search server is the server that executes cold data information search instructions. The file search server is different from the data storage server.

[0018] Based on the permission information and transmission port address information of the data storage server and file retrieval server, the information transmission statement is determined so that the file retrieval server can obtain the data information to be searched from the data storage server according to the information transmission statement.

[0019] In this embodiment of the application, the data information of the data storage server is parsed to obtain the data information to be searched, including:

[0020] Determine the files in the data information of the data storage server, and their storage addresses;

[0021] Based on the preset offline image file viewer, the file and file storage address in the data information of the data storage server are converted into a format to generate the data information to be searched.

[0022] In this embodiment of the application, determining the target hot data and the distribution of the target hot data among various storage files in the target storage partition, based on the data type of the cold data, includes:

[0023] Based on the preset access time, determine the hot data in the target storage partition and the type of hot data;

[0024] Based on the data type of the cold data and the hot data in the target storage partition, as well as the type of the hot data, determine the target hot data and its distribution among various storage files in the target storage partition.

[0025] In this embodiment of the application, the target hot data and its distribution among various storage files in the target storage partition are determined based on the data type of the cold data, the hot data in the target storage partition, and the type of the hot data, including:

[0026] Determine the target hot data based on the data type of the cold data;

[0027] Based on the target thermal data, determine the corresponding storage amount of the target thermal data;

[0028] Based on the storage volume of the target hot data, determine the distribution of the target hot data among the various storage files in the target storage partition.

[0029] In this embodiment of the application, cold data in each storage file is displayed according to the display order of each storage file in the target storage partition, including:

[0030] Determine the preset number of search results to display in the user interface;

[0031] Based on the number of search results and the display order of each storage file in the target storage partition, cold data in each storage file is displayed.

[0032] In this embodiment of the application, cold data in each storage file is displayed according to the number of search results displayed and the display order of each storage file in the target storage partition, including:

[0033] Get the preset access time;

[0034] By comparing the preset access time with the access time of cold data in each storage file in the target storage partition, the display order of cold data in each storage file in the target storage partition is determined.

[0035] Based on the number of search results and the order in which cold data is displayed in each storage file within the target storage partition, display the cold data in each storage file.

[0036] Secondly, this application provides a cold data retrieval device, comprising:

[0037] The acquisition module is used to acquire the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item.

[0038] The module is used to build Hive external tables based on the data to be searched.

[0039] The first determination module is used to determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored, based on the access time of the cold data to be searched and the Hive external table.

[0040] The second determining module is used to determine the target hot data and the distribution of the target hot data in each storage file in the target storage partition based on the data type of the cold data. The data type of the target hot data is the same as the data type of the cold data.

[0041] The third determining module is used to determine the display order of each storage file in the target storage partition based on the distribution of the target hot data in each storage file in the target storage partition;

[0042] The display module is used to display cold data in each storage file according to the display order of each storage file in the target storage partition.

[0043] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0044] The memory stores the instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory to implement the cold data retrieval method of this application embodiment.

[0046] Fourthly, a computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the cold data retrieval method of the embodiments of this application.

[0047] The cold data retrieval method, apparatus, electronic device, and storage medium provided in this application acquire data information to be searched, including the storage partition path, file access time, and file directory of each data item; construct a Hive external table based on the data information to be searched; determine the cold data, its data type, and the target storage partition containing the cold data based on the access time of the cold data to be searched and the Hive external table; determine the target hot data and its distribution among various storage files in the target storage partition based on the data type of the cold data, wherein the data type of the target hot data is the same as that of the cold data; determine the display order of the various storage files in the target storage partition based on the distribution of the target hot data among various storage files in the target storage partition; and display the cold data in each storage file according to the display order of the various storage files in the target storage partition. This allows the cold data retrieval operation to be performed on a file search server, achieving a non-intrusive retrieval of cold data from data storage servers. Furthermore, by utilizing the characteristics related to the same type of cold and hot data storage methods, the priority of the cold data retrieval results is determined by the distribution of hot data in each storage partition, improving the accuracy of cold data retrieval. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] Figure 1A flowchart illustrating a cold data retrieval method provided in an embodiment of this application;

[0050] Figure 2 A flowchart illustrating another cold data retrieval method provided in this application embodiment;

[0051] Figure 3 A detailed flowchart illustrating another cold data retrieval method provided in this application embodiment;

[0052] Figure 4 This is a schematic diagram of the structure of a cold data retrieval device provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] In existing technologies, when the stability of system storage space deteriorates, system users are notified to manually identify cold data and perform cleanup or transfer operations. However, this method involves too much manual intervention, and users find it difficult to determine which data is cold, leading to subjectivity and a lack of unified rules. This results in less than expected cleanup effectiveness and efficiency. With the continuous increase in data volume and the emergence of application automation, data management also requires related automated management technologies and methods to improve efficiency and reduce maintenance costs. Therefore, providing an accurate and efficient cold data retrieval method has become an urgent technical problem to be solved.

[0057] This application can transfer the cold data retrieval operation to a file search server, thereby achieving the effect of non-intrusive retrieval of cold data in the data storage server. At the same time, by utilizing the characteristics of the same type of cold and hot data storage methods, the priority of the cold data retrieval results in the corresponding storage partition is determined by the distribution of hot data in each storage partition, providing users with a basis for judging whether to perform cold data processing, thereby solving the above problems.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0059] This application provides a cold data retrieval method, apparatus, electronic device, and storage medium, applied in a file search server. This file search server is connected to a data storage server via a network interface and has data transmission, data processing, and data display functions. The file search server can be a server within a Hadoop system, but it is not in the same enterprise production environment resource pool as the data storage server. This embodiment does not impose any particular restriction on the type of file search server selected, as long as it can obtain the information of the data to be searched, including the storage partition path, file access time, and file directory of each data item; construct a Hive external table based on the information of the data to be searched; determine the cold data, its data type, and the target storage partition containing the cold data based on the access time of the cold data to be searched and the Hive external table; determine the target hot data and the distribution of the target hot data in each storage file within the target storage partition based on the data type of the cold data, with the data type of the target hot data being the same as that of the cold data; determine the display order of each storage file in the target storage partition based on the distribution of the target hot data in each storage file within the target storage partition; and display the cold data in each storage file according to the display order of the storage files in the target storage partition.

[0060] This application provides a cold data retrieval method, in which the executing entity can be a file search server, such as... Figure 1 As shown, the cold data retrieval method may include the following steps:

[0061] S101. Obtain the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item.

[0062] One method to obtain the data information to be searched is to parse the data mirror in the data storage server, obtain the latest data information, then export the latest metadata information and store this data in a server that is not associated with the data storage server in shared resources, providing basic data for executing server instructions, and finally parse this basic data to obtain the data information to be searched.

[0063] The data storage server can be an HDFS metadata server. The architecture of an HDFS metadata server includes a metadata node (NameNode(A), a backup metadata node (NameNode(S)), and data block storage nodes (DataNodes). The NameNode's internal structure consists of an editlog and an fsimage file. The editlog records the operation log, and the fsimage file records the image file. The fsimage file contains the serialization information (ID, type, directory, owner, user permissions, timestamp, etc.) of all directories and file inodes of the HDFS metadata server. Therefore, the exported latest metadata information can be the information from the fsimage file.

[0064] In this embodiment of the application, the method for obtaining the data information to be searched may include:

[0065] Receive data information from the data storage server, which is the server that needs to search for cold data information in the data information;

[0066] The data information on the data storage server is parsed to obtain the data information to be searched.

[0067] One method for parsing data information is to deserialize the file. Deserialization refers to converting a binary file into a state file of an object, which makes the file easier to read. The binary stream format is convenient for storage on disk, transmission over a network, or persistence in memory; while the state format of the object can be transferred between different computers, processes, or networks and restored when needed.

[0068] In this embodiment of the application, the method for receiving data information from a data storage server may include:

[0069] Determine the file search server's permission information and transmission port address information. The file search server is the server that executes cold data information search instructions. The file search server is different from the data storage server.

[0070] Based on the permission information and transmission port address information of the data storage server and file retrieval server, the information transmission statement is determined so that the file retrieval server can obtain the data information to be searched from the data storage server according to the information transmission statement.

[0071] The file search server permission information and transmission port address information can be the hostname and IP address information of the file search server, which are used to perform security authentication of the file search server when connecting to the data storage server.

[0072] Information transmission statements can be SCP (secure copy) commands, which are secure remote file copy commands in Linux systems based on SSH (Secure Shell) login. For example, a client of a file search server sends a connection request to a data storage server. The data storage server checks the received request packet and IP address. If the information is correct, it sends a connection key to the file search server, and a connection is established using the key.

[0073] In this embodiment of the application, the method for parsing the data information of the data storage server to obtain the data information to be searched may include:

[0074] Determine the files in the data information of the data storage server, and their storage addresses;

[0075] Based on the preset offline image file viewer, the file and file storage address in the data information of the data storage server are converted into a format to generate the data information to be searched.

[0076] The data information on the data storage server can be stored in binary form. The files and file storage addresses in the data information are displayed in binary string format, which has low readability. Therefore, format conversion is required.

[0077] Offline image file viewer refers to OIV (Offline Image Viewer), which can deserialize the serialized content of HDFS fsimage files, convert it into a human-readable format, and provide a read-only web API interface, enabling relatively fast processing of very large fsimage files.

[0078] S102. Based on the data information to be searched, construct a Hive external table.

[0079] In this embodiment of the application, the method for constructing a Hive external table based on the data to be searched may include:

[0080] Identify the individual pieces of information in the data to be searched, as well as the preset Hive tables;

[0081] Load each piece of information from the data to be searched into a Hive table to obtain the Hive external relation table corresponding to the data information and the table structure.

[0082] Hive tables can be built using the Hive data warehouse tool. Like tables in a relational database, regardless of the number of databases (data warehouses) or tables within them, all Hive tables only store metadata such as location, type, and attributes. The actual data stored in all tables is stored on a data storage server. Hive tables can include managed tables and external tables. The storage location of managed tables is fixed; modifications to a managed table are synchronized with the metadata, and deleting a managed table directly deletes both the metadata and the stored data. External tables can be stored in a specified location. Modifications to the structure and partitions of an external table are not directly synchronized with the metadata, and deleting an external table only deletes the metadata, not the original file storage data on the server.

[0083] S103. Based on the access time of the cold data to be searched and the Hive external table, determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored.

[0084] In this embodiment of the application, the method for determining cold data, the data type of the cold data, and the target storage partition storing the cold data may include:

[0085] Determine the preset access time for the cold data to be searched;

[0086] Construct a cold data query statement based on the access time of the cold data to be searched;

[0087] Based on the cold data query statement, query the Hive external table to determine the cold data storage information;

[0088] Based on the cold data storage information, determine the data type and target storage partition of the cold data.

[0089] The access time for cold data to be searched can be set based on information such as file type, attributes, and last modification time, and the access time for cold data to be searched is the same as the preset access time.

[0090] Cold data query statements can be HQL (Hibernate Query Language) provided by the Hive data warehouse tool. You only need to write HQL statements for the target object, and Hive will automatically convert the statements into MapReduce programs to process the structured data on the corresponding server.

[0091] For example, data accessed before time point T0 can be defined as cold data. By using HQL to generate a cold data query statement, the cold data can be queried in the Hive external table. By traversing the access time of the data in the Hive external table, all files accessed before time point T0 can be identified, and the storage information of the data in these files can be recorded. This allows us to determine the types of all cold data and the partition locations where they are stored.

[0092] S104. Based on the data type of the cold data, determine the target hot data and the distribution of the target hot data in each storage file in the target storage partition. The data type of the target hot data is the same as that of the cold data.

[0093] Hot data can be online data that is frequently accessed by the server host, while cold data can be offline data that is not frequently accessed. For ease of data storage, different storage partitions are created according to the different types of data. For example, step S103 identifies user types A and B, and the cold data of users A and B are stored in storage partition 1 and storage partition 2, respectively.

[0094] In this embodiment of the application, the method for determining the target hot data and the distribution of the target hot data among various storage files in the target storage partition based on the data type of the cold data may include:

[0095] Based on the preset access time, determine the hot data in the target storage partition and the type of hot data;

[0096] Based on the data type of the cold data and the hot data in the target storage partition, as well as the type of the hot data, determine the target hot data and its distribution among various storage files in the target storage partition.

[0097] One method for determining hot data in the target storage partition based on a preset access time is to iterate through the access times of each data item in the target storage partition and determine the data that is accessed after the preset access time as the hot data of the corresponding target storage partition.

[0098] For example, given that the preset access time T0 is March 1st, and the storage partitions corresponding to cold data include storage partition 1 and storage partition 2, we iterate through the data in storage partition 1 and storage partition 2 to determine that data accessed after March 1st is hot data. Then, we classify the hot data in storage partition 1 and storage partition 2 respectively. The classification results are as follows: in storage partition 1, the hot data corresponding to user category includes hot data of users A and C, and the hot data corresponding to date category includes hot data of March 9th and March 10th; in storage partition 2, the hot data corresponding to user category includes hot data of users B and C, and the hot data corresponding to date category includes hot data of March 21st and April 2nd.

[0099] In this embodiment of the application, the method for determining the target hot data and its distribution among various storage files in the target storage partition based on the data type of the cold data, the hot data in the target storage partition, and the type of the hot data may include:

[0100] Determine the target hot data based on the data type of the cold data;

[0101] Based on the target thermal data, determine the corresponding storage amount of the target thermal data;

[0102] Based on the storage volume of the target hot data, determine the distribution of the target hot data among the various storage files in the target storage partition.

[0103] For example, the method to determine the target hot data based on the data type of cold data can be as follows: if the types of cold data include user categories (user A, user B, user C, and user D) and date categories (February 10th, February 20th), and the storage partitions of the cold data include storage partition 1, storage partition 2, and storage partition 3, then based on the user categories of the cold data, it can be determined that the hot data corresponding to user A is in storage partition 1, the hot data corresponding to user B is in storage partition 2, the hot data corresponding to user C is in both storage partition 1 and storage partition 2, and the data for user D is all cold data.

[0104] Specifically, based on the storage volume of the target hot data, the distribution of the target hot data in each storage file in the target storage partition is determined. Taking user categories in storage partition 1 as an example: storage partition 1 includes storage file a, storage file b, and storage file c. The number of hot data stored for user A in storage partition 1 is determined in order of maximum to minimum as storage file a, storage file b, and storage file c; the number of hot data stored for user C in storage partition 1 is determined in order of maximum to minimum as storage file a, storage file c, and storage file b.

[0105] Specifically, based on the storage volume of the target hot data, the distribution of the target hot data in each storage file within the target storage partition is determined. Taking user C as an example: if there are storage partition 1 and storage partition 2, the storage quantity of hot data in storage partition 1 and storage partition 2 needs to be recorded respectively (the recording result is: the storage quantity of hot data in storage partition 1 is higher than that in storage partition 2); then the storage quantity of hot data in the two storage partitions is determined respectively, and the result is: the storage quantity of hot data in storage partition 1 is sorted from most to least as storage file a, storage file c, storage file b; the storage quantity of hot data in storage partition 2 is sorted from most to least as storage file a, storage file d, storage file f.

[0106] S105. Based on the distribution of the target hot data in each storage file in the target storage partition, determine the display order of each storage file in the target storage partition.

[0107] In this case, the display order of cold data can be such that the storage file with the most hot data is displayed later.

[0108] For example, based on the distribution: User A's hot data storage quantity in storage partition 1, sorted from most to least, is storage file a, storage file b, storage file c; User C's hot data storage quantity in storage partition 1, sorted from most to least, is storage file a, storage file c, storage file b, it can be determined that: User A's storage files in storage partition 1 are displayed in the order of storage file c, storage file b, storage file a; User C's storage files in storage partition 1 are displayed in the order of storage file b, storage file c, storage file a.

[0109] For example, based on the distribution: the distribution of storage partition 1 corresponding to user C is higher than that of storage partition 2, the display order can be determined as: storage partition 2, storage partition 1. Then, based on the distribution of the number of hot data stored in the two storage partitions, the display order can be determined as: storage partition 2 - (storage file b, storage file c, storage file a), storage partition 1 - (storage file f, storage file d, storage file a).

[0110] S106. Display the cold data in each storage file according to the display order of each storage file in the target storage partition.

[0111] In this embodiment of the application, the method for displaying cold data in each storage file according to the display order of each storage file in the target storage partition may include:

[0112] Determine the preset number of search results to display in the user interface;

[0113] Based on the number of search results and the display order of each storage file in the target storage partition, cold data in each storage file is displayed.

[0114] In this embodiment of the application, the method for displaying cold data in each storage file based on the number of search results displayed and the display order of each storage file in the target storage partition may include:

[0115] Get the preset access time;

[0116] By comparing the preset access time with the access time of cold data in each storage file in the target storage partition, the display order of cold data in each storage file in the target storage partition is determined.

[0117] Based on the number of search results and the order in which cold data is displayed in each storage file within the target storage partition, display the cold data in each storage file.

[0118] For example, given that the display order for user C is: storage partition 2 - (storage file b, storage file c, storage file a), storage partition 1 - (storage file f, storage file d, storage file a), and the number of search results displayed is 4, assuming that cold data information in storage partition 2 has 1 piece of information in storage file b, 2 pieces of information in storage file c, and 1 piece of information in storage file a, the display order in the user interface would be storage file b, storage file c / 1, storage file c / 2, storage file a in storage partition 2, and so on for storage partition 1. Another feasible display order could be: assuming that cold data information in storage partition 2 has 1 piece of information in storage file b, 2 pieces of information in storage file c, and 1 piece of information in storage file a; and that cold data information in storage partition 1 has 1 piece of information in storage file f, 2 pieces of information in storage file d, and 1 piece of information in storage file a, the display order in the user interface would be storage file b, storage file c / 1 in storage partition 2, and storage file f, storage file d / 1 in storage partition 1.

[0119] The cold data retrieval method provided in this application embodiment can utilize a file search server to search for cold data without affecting the normal data service implementation on the data storage server. This protects the stored data on the data storage server from damage during the search process, demonstrating the non-intrusive nature of this method. Furthermore, by using the access time of the cold data, data corresponding to the access time can be retrieved from the Hive external table, thereby identifying the cold data and improving the retrieval efficiency. In addition, by utilizing the characteristics related to the same type of cold and hot data storage methods, the priority of the cold data retrieval results in the corresponding storage partition is determined based on the distribution of hot data in each storage partition. This provides users with a basis for judging the processing of cold data and allows users to selectively process only a portion of the cold data.

[0120] Figure 2 A flowchart illustrating another cold data retrieval method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0121] S201. Obtain the fsimage file from the NameNode and copy the fsimage file to a separate server using the copy command.

[0122] In this scenario, an enterprise might have a Hadoop production cluster. The fsimage metadata file is copied to a separate server using the SCP command on the NameNode of the Hadoop production cluster. This way, analyzing the fsimage file on the separate server will not consume the resources of the Hadoop NameNode.

[0123] A Hadoop production cluster is a cluster of servers that run actual jobs. Each server is equivalent to a node. The more nodes there are, the larger the amount of data that can be processed. Therefore, in practical applications, the number of servers in a Hadoop production cluster depends on the amount of data that needs to be processed. In a Hadoop production cluster, each NameNode is an independent server. At any given time, only one NameNode is active, while the other is in standby mode.

[0124] The fsimage file refers to a complete image of the metadata in the NameNode. It stores the latest metadata checkpoint and is loaded when the Hadoop Distributed File System starts. It contains information about all directories and files in the entire file system.

[0125] S202. Obtain a file with a different format than the fsimage file through deserialization instructions.

[0126] The process involves using Hadoop's built-in OIV command to deserialize the fsimage command uploaded to the server in the first step, resulting in a plaintext file in the following format:

[0127] HDFS_DIR: HDFS file path and directory path; MODIFICATION_TIME: Modification time; ACCESS_TIME: Access time; FILE_SIZE: File size; USER_NAME: Username.

[0128] The OIV command is used to view fsimage files in Hadoop, and can also be used to save formatted files and call a process to perform decoding operations.

[0129] Deserialization can be achieved through OIV commands and writing Java programs. When two processes communicate remotely, they can send various types of data, including text, images, audio, and video. This data is transmitted over the network in the form of binary sequences. Since Java is an object-oriented development approach, everything is a Java object. To transmit Java objects over the network, serialization and deserialization can be used. The sender needs to convert the Java object into a byte sequence and then transmit it over the network. After receiving the character sequence, the receiver will use deserialization to restore the byte sequence into a Java object.

[0130] The process of saving a Java object as a series of bytes in a disk file, or the process of saving the state of a Java object, is called serialization. Serialization can permanently save data on disk (usually in a file), corresponding to the metadata file in the embodiments of this application, that is, the read metadata file is the serialized file; the process of converting the Java bytecode saved in the disk file back into a Java object is called deserialization.

[0131] S203. Create a hive table and load the deserialized file.

[0132] The statement used to create the Hive table is shown below:

[0133]

[0134] Hive is an open-source data warehouse software built on a distributed file system. It can map structured and semi-structured data files stored in distributed files to a database table. Based on the table, it provides a SQL-like query model called Hive Query Language (HQL) for accessing and analyzing large datasets stored in the distributed file system. The core of Hive is to convert HQL into MapReduce programs and then submit the programs to a distributed cluster for execution.

[0135] S204. Obtain the cold data size under multiple dimensions, including obtaining the cold data size of the entire cluster, obtaining the cold data size of each user, and obtaining the cold data size for each day.

[0136] Specifically, to obtain the size of cold data across multiple dimensions, and defining data accessed before cold_time as cold data, the following records are made:

[0137] 1) Obtain the total size of cold data in the entire cluster.

[0138] SELECT SUM(filesize)

[0139] from cold_data_table

[0140] where accesstime <cold_time

[0141] 2) Obtain the cold data size for each user.

[0142] SELECT username, SUM(filesize)

[0143] from cold_data_table

[0144] where accesstime <cold_time

[0145] Group by username

[0146] 3) Obtain the daily cold data size

[0147] select day(accesstime),sum(filesize)

[0148] from cold_data_table

[0149] where accesstime <cold_time

[0150] Grouped by day (access time)

[0151] After identifying cold data, it can be deleted and cleaned up based on the search information, thereby effectively ensuring the stability of HDFS.

[0152] The cold data retrieval method provided in this application copyes the metadata file to a server unrelated to the enterprise's Hadoop production cluster to perform cold data lookup operations. This does not affect the normal data service implementation of the enterprise's Hadoop production cluster, thus protecting the stored data in the NameNode from corruption during the lookup process, demonstrating the non-intrusive nature of this method. Furthermore, by searching for files corresponding to the access times of cold data in the Hive table, cold data can be generated, improving the efficiency of cold data retrieval. Compared to methods using the HDFS interface in the enterprise's Hadoop production cluster for scanning, the cold data lookup method provided in this application can accurately locate cold data within minutes, quickly process the cold data list, and ensure the stability of the distributed file system.

[0153] Figure 3 This is a schematic diagram illustrating the specific process of another cold data retrieval method provided in an embodiment of this application. For example... Figure 3 As shown, the fsimage metadata file stored in the NameNode of the data storage server is collected and copied to a server that is different from the running data storage server. On this server, the metadata file is deserialized to form a plaintext file. Then, a Hive table is created and the plaintext file is loaded into the Hive table to obtain the mapping relationship between each file in the plaintext file. The cold data definition is defined and HQL is used to calculate the cold data in a distributed manner. The files corresponding to the cold data list are then compiled to obtain the cold data and thus process the cold data.

[0154] Figure 4 This is a schematic diagram of the cold data retrieval device provided in an embodiment of this application. Figure 4 As shown, the cold data retrieval device 40 includes: an acquisition module 401, an establishment module 402, a first determination module 403, a second determination module 404, a third determination module 405, and a display module 406. Wherein:

[0155] The acquisition module 401 is used to acquire the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item.

[0156] Module 402 is used to construct a Hive external table based on the data to be searched.

[0157] The first determining module 403 is used to determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored, based on the access time of the cold data to be searched and the Hive external table.

[0158] The second determining module 404 is used to determine the target hot data and the distribution of the target hot data in each storage file in the target storage partition according to the data type of the cold data. The data type of the target hot data is the same as the data type of the cold data.

[0159] The third determining module 405 is used to determine the display order of each storage file in the target storage partition based on the distribution of the target hot data in each storage file in the target storage partition.

[0160] The display module 406 is used to display cold data in each storage file according to the display order of each storage file in the target storage partition.

[0161] In this embodiment of the application, the acquisition module 401 can also be used for:

[0162] Receive data information from the data storage server, which is the server that needs to search for cold data information in the data information;

[0163] The data information on the data storage server is parsed to obtain the data information to be searched.

[0164] In this embodiment of the application, the acquisition module 401 can also be used for:

[0165] Determine the file search server's permission information and transmission port address information. The file search server is the server that executes cold data information search instructions. The file search server is different from the data storage server.

[0166] Based on the permission information and transmission port address information of the data storage server and file retrieval server, the information transmission statement is determined so that the file retrieval server can obtain the data information to be searched from the data storage server according to the information transmission statement.

[0167] In this embodiment of the application, the acquisition module 401 can also be used for:

[0168] Determine the files in the data storage server's data information, and their storage addresses;

[0169] Based on the preset offline image file viewer, the file and file storage address in the data information of the data storage server are converted into a format to generate the data information to be searched.

[0170] In this embodiment of the application, the second determining module 404 can also be used for:

[0171] Based on the preset access time, determine the hot data in the target storage partition and the type of hot data;

[0172] Based on the data type of the cold data and the hot data in the target storage partition, as well as the type of the hot data, determine the target hot data and its distribution among various storage files in the target storage partition.

[0173] In this embodiment of the application, the second determining module 404 can also be used for:

[0174] Determine the target hot data based on the data type of the cold data;

[0175] Based on the target thermal data, determine the corresponding storage amount of the target thermal data;

[0176] Based on the storage volume of the target hot data, determine the distribution of the target hot data among the various storage files in the target storage partition.

[0177] In this embodiment of the application, the display module 406 can also be used for:

[0178] Determine the preset number of search results to display in the user interface;

[0179] Based on the number of search results and the display order of each storage file in the target storage partition, cold data in each storage file is displayed.

[0180] In this embodiment of the application, the display module 406 can also be used for:

[0181] Get the preset access time;

[0182] By comparing the preset access time with the access time of cold data in each storage file in the target storage partition, the display order of cold data in each storage file in the target storage partition is determined.

[0183] Based on the number of search results and the order in which cold data is displayed in each storage file within the target storage partition, display the cold data in each storage file.

[0184] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes:

[0185] The electronic device 50 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a network interface 503, and other components. The processor 501, memory 502, and network interface 503 are connected via a bus 504.

[0186] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the cold data retrieval method described above.

[0187] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0188] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0189] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0190] A network interface can be a wireless network interface or a wired network interface. This interface is typically used to establish communication connections between electronic devices and other electronic devices. For example, a network interface is used to connect an electronic device to an external terminal via a network, establishing data transmission channels and communication connections between the electronic device and the external terminal.

[0191] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0192] In some embodiments, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described cold data retrieval methods.

[0193] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in any computer-readable storage medium and loaded and executed by a processor.

[0194] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the cold data retrieval methods provided in embodiments of this application.

[0195] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0196] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0197] Since the instructions stored in the storage medium can execute the steps of any of the cold data retrieval methods provided in the embodiments of this application, the beneficial effects that any of the cold data retrieval methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0198] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0199] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0200] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A cold data retrieval method, characterized in that, Applied to a file search server, the method includes: Obtain the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item; Based on the data to be searched, construct a Hive external table; Based on the access time of the cold data to be searched and the Hive external table, determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored; Based on the data type of the cold data, the target hot data and the distribution of the target hot data in each storage file in the target storage partition are determined, wherein the data type of the target hot data is the same as the data type of the cold data. Based on the distribution of the target hot data among the various storage files in the target storage partition, determine the display order of each storage file in the target storage partition; Display cold data in each storage file according to the display order of the storage files in the target storage partition.

2. The method according to claim 1, characterized in that, The process of obtaining the data information to be searched includes: Receive data information from a data storage server, wherein the data storage server is a server that needs to search for cold data information in the data information; The data information of the data storage server is parsed to obtain the data information to be searched.

3. The method according to claim 2, characterized in that, The data information received from the data storage server includes: The file search server's permission information and transmission port address information are determined. The file search server is the server that executes cold data information search instructions, and the file search server is different from the data storage server. Based on the permission information of the data storage server and the file search server, and the transmission port address information, an information transmission statement is determined so that the file search server can obtain the data information to be searched from the data storage server according to the information transmission statement.

4. The method according to claim 2, characterized in that, The step of parsing the data information of the data storage server to obtain the data information to be searched includes: Determine the files and their storage addresses within the data information of the data storage server; Using a preset offline image file viewer, the file and file storage address in the data information of the data storage server are converted into a format to generate the data information to be searched.

5. The method according to claim 1, characterized in that, The step of determining the target hot data and the distribution of the target hot data among various storage files in the target storage partition based on the data type of the cold data includes: Based on the preset access time, determine the hot data in the target storage partition and the type of the hot data; Based on the data type of the cold data and the hot data in the target storage partition, as well as the type of the hot data, the distribution of the target hot data and its various storage files in the target storage partition is determined.

6. The method according to claim 5, characterized in that, The step of determining the target hot data and its distribution among various storage files in the target storage partition based on the data type of the cold data, the hot data in the target storage partition, and the type of the hot data includes: The target hot data is determined based on the data type of the cold data; Based on the target thermal data, determine the storage amount corresponding to the target thermal data; Based on the storage volume of the target hot data, determine the distribution of the target hot data among the various storage files in the target storage partition.

7. The method according to claim 1, characterized in that, The step of displaying cold data in each storage file according to the display order of each storage file in the target storage partition includes: Determine the preset number of search results to display in the user interface; Based on the number of search results displayed and the display order of each storage file in the target storage partition, cold data in each storage file is displayed.

8. The method according to claim 7, characterized in that, The step of displaying cold data in each storage file based on the number of search results displayed and the display order of each storage file in the target storage partition includes: Get the preset access time; By comparing the preset access time with the access time of cold data in each storage file of the target storage partition, the display order of cold data in each storage file of the target storage partition is determined. Based on the number of search results displayed and the order in which cold data is displayed in each storage file in the target storage partition, cold data in each storage file is displayed.

9. A cold data retrieval device, characterized in that, include: The acquisition module is used to acquire the data information to be searched, which includes the storage partition path, file access time, and file directory of each data item. A module is established to construct a Hive external table based on the data to be searched. The first determining module is used to determine the cold data, the data type of the cold data, and the target storage partition where the cold data is stored, based on the access time of the cold data to be searched and the Hive external table. The second determining module is used to determine the target hot data and the distribution of the target hot data in each storage file in the target storage partition according to the data type of the cold data, wherein the data type of the target hot data is the same as the data type of the cold data; The third determining module is used to determine the display order of each storage file in the target storage partition based on the distribution of the target hot data in each storage file in the target storage partition; The display module is used to display cold data in each storage file according to the display order of each storage file in the target storage partition.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the cold data retrieval method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the cold data retrieval method as described in any one of claims 1 to 8.

Images