Data processing method and apparatus

Abstract

The present application relates to the field of storage, and discloses a data processing method and apparatus. The method is applied to a server of a distributed file system. The method comprises: firstly, receiving a data governance request for M directories of a distributed file system; in response to the data governance request, respectively reading M latest reading time points in the M directories; then, on the basis of the latest reading time point of each directory, respectively determining a read popularity of each directory; and finally, on the basis of each read popularity, determining whether to perform data governance on the directory corresponding to each read popularity. Thus, the access complexity during data governance is reduced from the file level to the directory level, thereby improving the data governance efficiency.

Description

Data processing method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 14, 2023, with application number 202310869573.7 and application name “A method, device and other equipment for data processing”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 30, 2023, with application number 202311435047.6 and application name “Data processing method and device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage, and in particular to a data processing method and device. Background Art

[0003] With the rapid development of computer technology, user demand for server services is increasing. A single server can no longer meet these demands, leading to the emergence of distributed systems. A distributed system utilizes high-speed computer networks to connect multiple physically dispersed server nodes into a logically unified cluster. The basic idea is to distribute centralized data across multiple network-connected server nodes to achieve greater storage capacity and higher concurrent access. To ensure high availability of distributed systems, regular data governance is required to delete or transfer less frequently accessed files.

[0004] When performing data governance on files stored in a distributed system, users need to determine the metadata of all files in the entire distributed system and obtain the read popularity information of each file. This puts a lot of pressure on the server access and results in low efficiency of data governance tasks.

[0005] Summary of the Invention

[0006] The present application provides a data processing method and device, which reduces the access complexity during data management from the file level to the directory level, thereby improving data management efficiency.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, a data processing method is provided, which is applied to the server side of a distributed file system. The execution subject of the method can be the server side, or a component or device (such as a processor, chip, or chip system) applied to the server side, or a logic module or software that can implement all or part of the server side functions. The method includes: first receiving a data governance request for M directories of the distributed file system; then, in response to the data governance request, reading the M latest read time points recorded in the M directories respectively; then, determining the read heat of each directory based on the latest read time point of each directory; and finally, determining whether to perform data governance on the directory corresponding to each read heat based on each read heat.

[0009] In the first aspect, data governance is performed by accessing the latest read time point of the directory. Compared with current data governance, this reduces the complexity of data access from the file level to the directory level, greatly improving data governance efficiency.

[0010] In one implementation, the read heat of each directory is determined based on the latest read time point of each directory, including: comparing each read heat with a heat threshold, and when the read heat is higher than the heat threshold, determining the directory as a directory with high read heat; when the read heat is lower than or equal to the heat threshold, determining the directory as a directory with low read heat.

[0011] In this implementation, by comparing the heat threshold and the read heat, it is possible to accurately determine whether the directory has low read heat or high read heat.

[0012] In one implementation, whether to perform data management on the directory corresponding to each read heat is determined based on each read heat, including: giving up access to the file heat in the directory with low read heat; and accessing the file heat in the directory with high read heat, and performing data management on the directory with high read heat based on the file heat.

[0013] In this implementation, targeted data governance is performed on directories based on their read popularity. The data governance process only needs to access the file popularity in directories with high read popularity, which improves data governance efficiency.

[0014] In one implementation, data governance includes one or more of the following: deleting low-hot files, storing low-hot files infrequently, or archiving low-hot files.

[0015] In this implementation, multiple data governance implementation methods are provided to effectively carry out data governance.

[0016] In one implementation, the distributed file system also includes multiple home storage nodes, and the method also includes: respectively confirming the H home storage nodes where the M directories are located; recording the M latest reading time points in the H home storage nodes respectively, where H is a positive integer greater than or equal to.

[0017] In this implementation, the latest reading time point of the directory is recorded in the corresponding home storage node, which can be used to determine the reading popularity of the directory in the home storage node.

[0018] In the second aspect, a data processing device is provided, which is applied to the server side of a distributed file system. The communication device can be the server side or a chip or system on chip in the server side. The communication device can implement the functions performed by the server side in the above-mentioned first aspect or the possible design of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the communication device includes: a receiving module for receiving a data governance request for M directories of a distributed file system; an acquisition module for reading the M latest reading time points recorded in the M directories respectively in response to the data governance request; a determination module for determining the reading heat of each directory according to the latest reading time point of each directory; and the determination module is also used to determine whether to perform data governance on the directory corresponding to each reading heat according to each reading heat.

[0019] In one implementation, the determination module is specifically used to: compare each read heat with the heat threshold respectively, and when the read heat is higher than the heat threshold, determine the directory as a directory with high read heat; when the read heat is lower than or equal to the heat threshold, determine the directory as a directory with low read heat.

[0020] In one implementation, the determination module is specifically used to: abandon access to file heat in a directory with low read heat; and access file heat in a directory with high read heat, and perform data management on the directory with high read heat according to the file heat.

[0021] In one implementation, data governance includes one or more of the following: deleting low-hot files, storing low-hot files infrequently, or archiving low-hot files.

[0022] In one implementation, the distributed file system also includes multiple home storage nodes, and the device also includes a recording module; the determination module is further used to separately confirm the H home storage nodes where the M directories are located; the recording module is used to record the M latest reading time points in the H home storage nodes respectively, where H is a positive integer greater than or equal to.

[0023] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each of the at least one computing device comprising at least one processor and at least one memory, wherein computer-readable instructions are stored in the at least one memory; the at least one processor executes the computer-readable instructions so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0024] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, causes the computing device cluster to execute the method described in the first aspect or any possible implementation of the first aspect.

[0025] In a fifth aspect, the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0026] Among them, the beneficial effects described in the second to fifth aspects of this application can refer to the analysis of the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG1 is a schematic diagram of the structure of a distributed system provided in an embodiment of the present application;

[0028] FIG2 is a schematic diagram of the structure of another distributed system provided in an embodiment of the present application;

[0029] FIG3 is a flow chart of a data processing method provided in an embodiment of the present application;

[0030] FIG4 is a flow chart of another data processing method provided in an embodiment of the present application;

[0031] FIG5 is a flow chart of another data processing method provided in an embodiment of the present application;

[0032] FIG6 is a flow chart of another data processing method provided in an embodiment of the present application;

[0033] FIG7 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0034] FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0035] FIG9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0036] FIG10 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0038] It should be noted that the terms "first" and "second" in the specification, claims, and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0039] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, "more than one" refers to two or more, "at least two (items)" refers to two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. In addition, the "connection" in the embodiments of the present application refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices, and the embodiments of the present application do not impose any limitation on this.

[0040] It should be noted that the terms "first" and "second" in the specification, claims, and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0041] Before introducing the data processing method provided in the embodiments of the present application, some terms involved in the embodiments of the present application are explained.

[0042] Low-frequency storage: Offers low cost and high reliability, ensuring data availability and integrity. However, low-frequency storage has relatively slow read and write speeds and requires prefetching and warming up, making it unsuitable for frequently accessed data.

[0043] Archive storage: This offers the lowest price, but also has the slowest read and write speeds, requiring prefetching and warming up, and requiring a certain amount of latency when accessing data. Archive storage offers high reliability and security, ensuring data integrity and confidentiality. Therefore, it's suitable for data that requires long-term retention but not for frequently accessed data.

[0044] Distributed system: also known as a distributed cluster. It is a group of interconnected computers or servers that work together to perform common tasks. Computers or servers in a distributed system are usually located in different physical locations and are interconnected through a network. Distributed systems are generally used to provide high availability, scalability and fault tolerance for applications and services. In a distributed system, each computer or server can be called a node, and multiple nodes constitute a distributed system. Exemplarily, a distributed system can be as shown in Figure 1, which is a structural diagram of the data access system provided by this application. The data access system includes a distributed system 100, and the distributed system 100 includes: a network device 130 and multiple server nodes: node 111, node 112 and node 113. A server node is used to process part of the data in the distributed system 100, and the server node communicates with the host (client) through a network or hardware communication channel. It should be understood that Figure 1 is only an example provided in this embodiment, and the distributed system 100 may include more or fewer server nodes, which is not limited by this application.

[0045] The distributed system 100 can support multiple hosts (also known as clients) to process and access files, data, etc. in the server node. The multiple hosts may include host 1, host 2, and host 3 shown in Figure 1. It is worth noting that the distributed system 100 can also support more or fewer hosts for data processing and access. Among them, the host, or client, user end, etc., can be a computer running an application. The computer running the application can be a physical machine or a virtual machine. For example, if the computer running the application is a physical computing device, the physical computing device can be a host or a terminal.

[0046] In addition to the distributed clusters described above, the distributed clusters in the embodiments of the present application can also be cloud servers, such as public cloud servers, private cloud servers, hybrid cloud servers, etc. Although a cloud server is not a traditional distributed cluster (because a cloud server is not a group of interconnected physical servers), it can be considered a virtual distributed cluster. This is because cloud servers are usually hosted on a hypervisor, which is a virtualization platform that allows multiple virtual machines to run on a single physical server. Each virtual machine is equivalent to a node, and multiple nodes constitute a distributed cluster.

[0047] As shown in Figure 2, Figure 2 is a schematic diagram of the structure and organization of the distributed system provided by this application. The hardware implementation in the distributed system 200 can refer to the relevant content in Figure 1 and is not described in detail here. As shown in Figure 2, the distributed system 200 includes one or more nodes, and the data in a node is processed and accessed by a server node.

[0048] A distributed system can also be called a file system. A file system is a structured data file storage and organization format. As shown in Figure 2, a node contains multiple directories, such as Directory 1, Directory 2, and Directory 3. A directory stores multiple files, each of which can be used to store one or more sets of data.

[0049] The above describes a distributed system from the perspective of hardware structure and software implementation. To ensure high availability, distributed system applications require regular data governance to delete or dump files with low access popularity. When performing data governance on files stored in a distributed system, it is necessary to determine the access popularity of each file in the entire distributed system so that low-access files can be deleted or dumped. Generally, the number of directories is far smaller than the number of files. However, the directory metadata table does not record file-level read popularity information in the directory, but rather records directory-level read popularity information. Currently, the update scenarios for the read popularity information atime in the directory metadata table are shown in Table 1. The read popularity information atime is only updated when directories are listed. In other words, the current read popularity information atime cannot indicate the access popularity of files in the directory. As a result, current data governance requires accessing the metadata of all files in the entire distributed system at every turn, which is very inefficient.

[0050] Table 1

[0051] To address the above issues, embodiments of the present application provide a data processing method, which is applied to the server side of a distributed system. This method performs data management by accessing the latest read time point of a directory. Compared to current data management, this method reduces the complexity of data access from the file level to the directory level, significantly improving data management efficiency.

[0052] FIG3 shows a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG3 , the method may include the following steps:

[0053] Step S310: Receive a data management request for M directories of a distributed file system.

[0054] The data management request may be initiated by a user on the client and sent by the client to the server to request the management of M directories.

[0055] Step S320: Read the M latest reading time points recorded in the M directories respectively in response to the data governance request.

[0056] Wherein, each directory corresponds to a latest read time point, M is a positive integer greater than 1 or equal to 1. The latest read time point of each directory is the latest time point among the N time points at which the server reads each file in the directory based on N file read requests for at least one file in each directory, N is a positive integer greater than 1 or equal to 1. Exemplarily, N is 3, and the time points corresponding to the N file read requests are: T1, T2 and T3 respectively. T3 is later than T1 and T2, then the latest time point of the directory is T3. The files requested by the N file read requests can be the same file or different files. The latest read time point is determined based on multiple read time points generated by the client. Specifically, when the client receives a read request for a file (indicating a file and a file storage address, the file storage address includes the directory to which the file belongs and the server to which the directory belongs, and exemplary, the file storage address can include a server identifier and a directory identifier inode), it records the corresponding read time point and sends the read time point and the corresponding file storage address to the server. The read request may request one or more files in a directory, or may request one or more files in different directories. Different directories may be stored on the same server or on different servers, without limitation.

[0057] After receiving multiple sets of read time points and corresponding file storage addresses, the server can determine multiple read time points corresponding to the same directory, and determine the latest read time point of the directory from the multiple read time points. For example, within a preset period, the server continuously receives read requests for files, requesting file A, file B, and file C (corresponding to read times T1, T2, and T3, respectively). The file storage addresses corresponding to file A, file B, and file C are: server P1-directory I, server P1-directory I, server P2-directory U. Then the read time points of directory I include T1 and T2, and the read time points of directory U include T3. If T2 is later than T1, then the latest read time point of directory I is T2, and the latest read time point of directory U is T3. Based on this, the M latest read time points recorded in the M directories can be read.

[0058] In one embodiment, a client can continuously receive file read requests within a preset period to obtain multiple file read requests. In other words, the client can aggregate read time points within a preset period and then send them to the server. In this implementation, when the client aggregates read time points within a preset period, it can determine the latest read time point for the same directory and use this latest read time point as the content sent to the server. The specific determination process can refer to the principle of determining the latest read time point by the server and will not be described in detail. Clients may have multiple read requests. When the server receives multiple latest read time points for the same directory from multiple clients, it can determine the final latest read time point from these multiple latest read time points. This operation reduces the communication overhead caused by the real-time transmission of file popularity between the client and the server. The preset period can be flexibly set according to the application scenario. For example, if file popularity accumulates quickly, the preset period can be set shorter; otherwise, the preset period can be set longer. For example, the preset period can be set to 100ms.

[0059] Step S330 : determining the read heat of each directory according to the latest read time point of each directory.

[0060] When determining the read popularity of each directory, the latest read time point of the directory can be directly used as the read popularity. The latest read time point can also be converted into other data types that can be quantified and compared, without limitation.

[0061] Step S340: Determine whether to perform data management on the directory corresponding to each read heat according to each read heat.

[0062] Among them, after determining the reading popularity of each directory, it can be determined whether to perform data governance on the directory based on it.

[0063] In the embodiment of the present application, data management is performed by accessing the latest read time point of the directory. Compared with the current data management, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data management.

[0064] In one embodiment, step S330 may include:

[0065] Each read heat is compared with the heat threshold. If the read heat is higher than the heat threshold, the directory is determined to be a high read heat directory. If the read heat is lower than or equal to the heat threshold, the directory is determined to be a low read heat directory.

[0066] Among them, the heat threshold can be set flexibly. For example, the threshold time point obtained by subtracting a preset time from the current time can be used as the heat threshold. Specifically, assuming that the current time is 15:00 on September 1, 2022, and the preset time is 72 hours, the heat threshold is 15:00 on August 29, 2022. The read heat of directory A is 12:00 on August 30, 2022, which is later than 15:00 on August 29, 2022 (equivalent to the read heat being higher than the heat threshold), then directory A is a directory with high read heat. The read heat of directory B is 15:00 on August 28, 2022, which is earlier than 15:00 on August 29, 2022 (equivalent to the read heat being lower than or equal to the heat threshold), then directory B is a directory with low read heat.

[0067] In the embodiment of the present application, by comparing the heat threshold and the read heat, it is possible to accurately determine whether the directory has low read heat or high read heat.

[0068] In one embodiment, as shown in FIG4 , step S340 may include:

[0069] Step S410: Abandon accessing the file heat in the directory with low read heat.

[0070] A low read popularity in a directory indicates that the files in the directory have not been used recently. In this case, file popularity access is no longer necessary for the files in the low-read popularity directory. Instead, the following actions are taken on the low-read popularity directory at the directory level: deletion, low-frequency storage, or archiving of all files in the low-read popularity directory.

[0071] Step S420: Access the file heat in the directory with high read heat.

[0072] A high directory read heat indicates that some files in the directory have been used recently. In this case, you can access the file heat of the directory with high read heat to differentiate and manage the files in that directory. The file heat is recorded in the file attribute information. Similar to the directory read heat, the file heat is determined based on the latest read time of the file. The specific process for determining file heat can be referenced in the directory read heat determination process and will not be repeated here.

[0073] Step S430: Perform data management on directories with high read popularity based on file popularity.

[0074] Based on file popularity, we can determine which files in a directory with high read popularity have high and low file popularity, and then conduct data management. Specifically, for low-popularity files, we can use one or more of the following methods to manage data: deleting low-popularity files, storing low-popularity files less frequently, or archiving low-popularity files. High-popularity files can continue to be stored on the server.

[0075] In an embodiment of the present application, targeted data management is performed on directories according to their different read popularity. The data management process only needs to access the file popularity of directories with high read popularity, thereby improving data management efficiency.

[0076] In one embodiment, as shown in FIG5 , the distributed file system further includes a plurality of home storage nodes, and the method further includes:

[0077] Step S510: confirm the H home storage nodes where the M directories are located respectively.

[0078] Where H is a positive integer greater than or equal to 1. The server node storing a directory is called the home storage node for that directory. A server node can store one or more directories. The M directories may be stored on the same home storage node or on different home storage nodes. The server executing steps S510-S520 may be the same node as the directory's home storage node or different nodes.

[0079] A file read request may be a request for one file or multiple files, and the multiple files requested may be stored in the same home node or in different home nodes. In a scenario where the files requested by the read request are stored in different home nodes, upon receiving the file read request, the access node will send read requests to the different home nodes respectively to obtain all the files requested by the read request. For example, as shown in FIG4 , if the files requested by the read request are stored in home node 41 and home node 42 respectively, the access node 40 will send read requests to home node 41 and home node 42 respectively to obtain all the files requested by the read request.

[0080] Step S520: Record the M latest reading time points in the H home storage nodes respectively.

[0081] Wherein, assuming that there are three directories A, B, and C, then M is 3, and the latest reading time points corresponding to directory A, directory B, and directory C are T1, T2, and T3, respectively. As shown in Figure 6, the storage node to which directory A and directory B belong is node 1, and the storage node to which directory C belongs is node 2. Both node 1 and node 2 are not the server ends that execute steps S510-S520. Then T1 and T2 are recorded in the corresponding directory attribute information in node 1, and T3 is recorded in the corresponding directory attribute information in node 2. Exemplarily, the latest reading time point can be recorded in the metadata table of the directory in the form of ext_dir_atime. In other words, once a file in a directory is read, the ext_dir_atime in the directory will be updated. The update scenario of ext_dir_atime in the metadata table is shown in Table 2.

[0082] Table 2

[0083] In the embodiment of the present application, the latest reading time point of the directory is recorded in the corresponding home storage node, which can be used to determine the reading popularity of the directory in the home storage node.

[0084] In summary, the embodiment of the present application performs data management by accessing the latest read time point of the directory. Compared with the current data management, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data management.

[0085] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the execution logic of each step. It is understandable that, in order to realize the above functions, the server includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the algorithm steps of each example described in the embodiment disclosed herein, the method of the embodiment of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0086] Figure 7 shows a structural diagram of a data processing device 700, which is applied to the server. Each module in the device shown in Figure 7 has the function of the corresponding step in the method embodiment and can achieve its corresponding technical effect. The beneficial effects of the execution steps of each module can refer to the description of the corresponding steps in the method embodiment and will not be repeated here. The function can be implemented by hardware or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the data processing device includes: a receiving module 710, which is used to receive a data governance request for M directories of a distributed file system; an acquisition module 720, which is used to read the M latest reading time points recorded in the M directories respectively in response to the data governance request; a determination module 730, which is also used to determine the reading heat of each directory according to the latest reading time point of each directory; and a determination module, which is also used to determine whether to perform data governance on the directory corresponding to each reading heat according to each reading heat.

[0087] Each of the modules described above can be implemented by software or hardware. For example, the implementation of the receiving module 710 will be described below using the receiving module 710 as an example. Similarly, the implementation of the obtaining module 720 and the determining module 730 can refer to the implementation of the receiving module 710.

[0088] As an example of a software functional unit, the receiving module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the receiving module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0089] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0090] As an example of a hardware functional unit, receiving module 710 may include at least one computing device, such as a server. Alternatively, receiving module 710 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0091] The multiple computing devices included in the receiving module 710 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 710 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 710 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0092] It should be noted that, in other embodiments, the receiving module 710 can be used to execute any step in the data processing method, the acquisition module 720 and the determination module 730 can be used to execute any step in the data processing method, and the steps that the receiving module 710, the acquisition module 720 and the determination module 730 are responsible for implementing can be specified as needed. The full functions of the data processing device are realized by respectively implementing different steps in the data processing method through the receiving module 710, the acquisition module 720 and the determination module 730.

[0093] This application also provides a computing device 100. As shown in FIG8 , computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0094] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 shows a single bus line, but this does not imply a single bus or type of bus. Bus 104 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0095] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0096] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0097] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the aforementioned receiving module 710, acquisition module 720, and determination module 730, thereby implementing the data processing method. In other words, the memory 106 stores instructions for executing the data processing method.

[0098] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned data processing apparatus, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0099] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0100] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0101] As shown in Figure 9, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0102] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data processing method.

[0103] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions, each for executing a portion of the functions of the data processing apparatus. In other words, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of one or more of the receiving module 710, the obtaining module 720, and the determining module 730.

[0104] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 10 illustrates one possible implementation. As shown in FIG. 10 , two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is achieved via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the receiving module 710 . Simultaneously, the memory 106 in the computing device 100B stores instructions for executing the functions of the acquiring module 720 and the determining module 730 .

[0105] The connection method between the computing device clusters shown in Figure 10 may be based on the needs of the data processing method provided in this application, so it is considered to hand over the functions implemented by the acquisition module 720 and the determination module 730 to the computing device 100B for execution.

[0106] It should be understood that the functions of the computing device 100A shown in FIG10 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0107] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 9 and 10. However, the memory 106 in one or more computing devices 100 in this computing device cluster can store the same instructions for executing the data processing method.

[0108] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data processing method.

[0109] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing part of the functions of the data detection system. In other words, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of the data processing device.

[0110] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the computer program product causes the at least one computing device to perform a data processing method or a data processing method.

[0111] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a server of a distributed file system, and the method comprises: Receiving a data management request for M directories of the distributed file system; In response to the data governance request, read the M latest reading time points recorded in the M directories respectively, wherein each directory record has a latest reading time point, and M is a positive integer greater than 1 or equal to 1; Determine the read heat of each directory according to the latest read time point of each directory, wherein the latest read time point of each directory is the latest time point among N time points at which the server reads each file in the directory based on N file read requests of at least one file in each directory, and N is a positive integer greater than 1 or equal to 1; Determine whether to perform data management on the directory corresponding to each read heat according to each read heat.

2. The method according to claim 1, characterized in that Determining the read heat of each directory according to the latest read time point of each directory includes: Each read heat is compared with a heat threshold value respectively. When the read heat is higher than the heat threshold value, the directory is determined to be a directory with high read heat. When the read heat is lower than or equal to the heat threshold value, the directory is determined to be a directory with low read heat.

3. The method according to claim 2, characterized in that The determining, according to each read heat, whether to perform data management on the directory corresponding to each read heat includes: Abandon accessing the file temperature in the directory with low read temperature; and Access the file heat in the directory with high read heat, and perform data management on the directory with high read heat according to the file heat.

4. The method according to any one of claims 1 to 3, characterized in that: The data governance includes one or more of the following: Delete low-frequency files, store low-frequency files infrequently, or archive low-frequency files.

5. The method according to any one of claims 1 to 4, characterized in that: The distributed file system further includes a plurality of home storage nodes, and the method further includes: Respectively confirm the H belonging storage nodes where the M directories are located; The M latest reading time points are recorded in the H belonging storage nodes respectively, where H is a positive integer greater than 1 or equal to 1.

6. A data processing device, characterized in that: The device is applied to a server of a distributed file system, and the device includes: A receiving module, configured to receive a data management request for M directories of the distributed file system; an acquisition module, configured to respectively determine, in response to the data governance request, M latest reading time points corresponding to the M directories, wherein each directory corresponds to a latest reading time point, and M is a positive integer greater than or equal to 6; The determination module is further used to determine the reading heat of each directory according to the latest reading time point of each directory, wherein the latest reading time point of each directory is the latest time point among N time points at which the server reads each file in the directory based on N file read requests of at least one file in each directory, and N is a positive integer greater than 6 or equal to 6; The determination module is further used to determine whether to perform data management on the directory corresponding to each reading heat according to each reading heat.

7. The device according to claim 6, characterized in that The determination module is specifically used for: Each read heat is compared with a heat threshold value respectively. When the read heat is higher than the heat threshold value, the directory is determined to be a directory with high read heat. When the read heat is lower than or equal to the heat threshold value, the directory is determined to be a directory with low read heat.

8. The device according to claim 7, characterized in that The determination module is specifically used for: Abandon accessing the file heat in the directory with low read heat; and access the file heat in the directory with high read heat, and perform data management on the directory with high read heat according to the file heat.

9. The device according to any one of claims 6 to 8, characterized in that The data governance includes one or more of the following: Delete low-frequency files, store low-frequency files infrequently, or archive low-frequency files.

10. The device according to any one of claims 6 to 9, characterized in that The distributed file system further includes a plurality of home storage nodes, the device further includes a recording module, and the determination module is further used to respectively confirm the H home storage nodes where the M directories are located; The recording module is used to record the M latest reading time points in the H belonging storage nodes respectively, where H is a positive integer greater than 6 or equal to 6.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 5.

Images