Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the field of storage. The method is applied to a server of the distributed file system, and comprises the following steps: firstly, receiving a data governance request for M directories of the distributed file system; respectively reading M latest reading time points in the M directories in response to the data governance request; according to the latest reading time point of each catalogue, the reading heat of each catalogue is determined; and finally, according to each reading popularity, determining whether to carry out data management on the directory corresponding to each reading popularity. In this way, the access complexity during data governance is reduced from the file level to the directory level, and the data governance efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of storage, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of computer technology, users have an increasing demand for services provided by servers. A single server can no longer meet users' large service needs, so distributed systems came into being. Distributed systems use high-speed computer networks to connect multiple physically dispersed server nodes to form a logically unified cluster. The basic idea is to store the original centralized data in multiple server nodes connected by the network to obtain larger storage capacity and higher concurrent access volume. In order to ensure the high availability of distributed systems, it is necessary to regularly manage data in distributed systems to delete or dump files with low access popularity in distributed systems.

[0003] When performing data management on files stored in a distributed system, users need to determine the metadata of all files in the entire distributed system and obtain the read popularity information of each file, which puts a lot of pressure on the access of the server and leads to low efficiency of data management tasks. Summary of the invention

[0004] The present application provides a data processing method and device, which reduces the access complexity during data management from the file level to the directory level, thereby improving the efficiency of data management.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In the first aspect, a data processing method is provided, which is applied to the server of a distributed file system. The execution subject of the method can be the server, or a component or device (such as a processor, a chip, or a chip system, etc.) applied to the server, or a logic module or software that can realize all or part of the server functions. The method includes: first receiving a data governance request for M directories of a distributed file system; then reading the M latest reading time points recorded in the M directories respectively in response to the data governance request; then determining the reading heat of each directory respectively according to the latest reading time point of each directory; and finally determining whether to perform data governance on the directory corresponding to each reading heat according to each reading heat.

[0007] In the first aspect, data governance is performed by accessing the latest read time point of the directory. Compared with the current data governance, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data governance.

[0008] In one implementation, the read heat of each directory is determined according to the latest read time point of each directory, including: comparing each read heat with a heat threshold, and when the read heat is higher than the heat threshold, determining the directory to be a directory with high read heat; when the read heat is lower than or equal to the heat threshold, determining the directory to be a directory with low read heat.

[0009] In this implementation, by comparing the heat threshold and the read heat, it is possible to accurately determine whether the directory has low read heat or high read heat.

[0010] In one implementation, whether to perform data management on the directory corresponding to each read heat is determined based on each read heat, including: giving up access to file heat in directories with low read heat; and accessing file heat in directories with high read heat, and performing data management on the directories with high read heat based on the file heat.

[0011] In this implementation, targeted data management is performed on directories according to their different read popularity. The data management process only needs to access the file popularity in directories with high read popularity, which improves data management efficiency.

[0012] In one implementation, data governance includes one or more of the following: deleting low-temperature files, storing low-temperature files infrequently, or archiving and storing low-temperature files.

[0013] In this implementation, a variety of data governance implementation methods are provided to enable effective data governance.

[0014] In one implementation, the distributed file system also includes multiple belonging storage nodes, and the method also includes: respectively confirming the H belonging storage nodes where the M directories are located; recording the M latest reading time points in the H belonging storage nodes respectively, where H is a positive integer greater than or equal to.

[0015] In this implementation, the latest reading time point of the directory is recorded in the corresponding home storage node, which can be used to determine the reading popularity of the directory in the home storage node.

[0016] In the second aspect, a data processing device is provided, which is applied to the server of a distributed file system. The communication device can be a server or a chip or system on chip in the server. The communication device can implement the functions performed by the server in the above-mentioned first aspect or the possible design of the first aspect. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. For example, the communication device includes: a receiving module for receiving a data governance request for M directories of a distributed file system; an acquisition module for reading the M latest reading time points recorded in the M directories respectively in response to the data governance request; a determination module for determining the reading heat of each directory according to the latest reading time point of each directory; the determination module is also used to determine whether to perform data governance on the directory corresponding to each reading heat according to each reading heat.

[0017] In one implementation, the determination module is specifically used to: compare each read heat with a heat threshold respectively, and when the read heat is higher than the heat threshold, determine the directory as a directory with high read heat; when the read heat is lower than or equal to the heat threshold, determine the directory as a directory with low read heat.

[0018] In one implementation, the determination module is specifically used to: abandon access to file temperatures in directories with low read temperatures; and access file temperatures in directories with high read temperatures, and perform data management on the directories with high read temperatures based on the file temperatures.

[0019] In one implementation, data governance includes one or more of the following: deleting low-temperature files, storing low-temperature files infrequently, or archiving and storing low-temperature files.

[0020] In one implementation, the distributed file system also includes multiple belonging storage nodes, and the device also includes a recording module; the determination module is also used to separately confirm the H belonging storage nodes where the M directories are located; the recording module is used to record the M latest reading time points in the H belonging storage nodes respectively, where H is a positive integer greater than or equal to.

[0021] In a third aspect, the present application provides a computing device cluster, wherein the computing device cluster includes at least one computing device, each of the at least one computing device includes at least one processor and at least one memory, and the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, causes the computing device cluster to execute the method described in the first aspect or any possible implementation of the first aspect.

[0023] In a fifth aspect, the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0024] Among them, the beneficial effects described in the second to fifth aspects of the present application can refer to the analysis of the beneficial effects of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the structure of a distributed system provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of another distributed system provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0028] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;

[0029] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;

[0030] Figure 6 A flowchart of another data processing method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0034] Fig.10 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0036] It should be noted that the terms "first" and "second" in the specification, claims and drawings of the present application are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0037] It should be understood that in the embodiments of the present application, "at least one (item)" refers to one or more, "multiple" refers to two or more, "at least two (items)" refers to two or three and more than three, and "and / or" is used to describe the association relationship of the associated objects, indicating that there can be three relationships. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following (items)" or similar expressions refers to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. In addition, the "connection" in the embodiments of the present application refers to various connection modes such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitation on this.

[0038] It should be noted that the terms "first" and "second" in the specification, claims and drawings of the present application are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0039] Before introducing the data processing method provided in the embodiment of the present application, some terms involved in the embodiment of the present application are explained.

[0040] Low-frequency storage: It has low price and high reliability, and the availability and integrity of data are also guaranteed. However, the read and write speed of low-frequency storage is relatively slow, and pre-fetching and pre-heating operations are required, so it is not suitable for frequently accessed data.

[0041] Archive storage: The lowest price, but the slowest read and write speed, requires pre-fetching and pre-warming operations, and requires a certain amount of waiting when accessing. Archive storage has high reliability and high security, and the integrity and confidentiality of data are guaranteed. Therefore, archive storage is suitable for data that needs to be retained for a long time, but not for data that is frequently accessed.

[0042] Distributed system: Also known as a distributed cluster. It is a group of interconnected computers or servers that work together to perform common tasks. Computers or servers in a distributed system are usually located in different physical locations and are connected to each other through a network. Distributed systems are often used to provide high availability, scalability, and fault tolerance for applications and services. In a distributed system, each computer or server can be called a node, and multiple nodes constitute a distributed system. For example, a distributed system can be as follows: Figure 1 As shown, Figure 1 The data access system provided by the present application is a structural diagram, which includes a distributed system 100, and the distributed system 100 includes: a network device 130 and multiple server nodes: node 111, node 112 and node 113. A server node is used to process part of the data in the distributed system 100, and the server node communicates with the host (client) through a network or hardware communication channel. It should be understood that Figure 1 This is only an example provided for this embodiment. The distributed system 100 may include more or fewer server nodes, which is not limited in this application.

[0043] The distributed system 100 can support multiple hosts (also called clients) to process and access files, data, etc. in the server node. The multiple hosts may include Figure 1 It is worth noting that the distributed system 100 can also support more or fewer hosts for data processing and access, as shown in the host 1, host 2 and host 3. Among them, the host, or client, user end, etc., can be a computer running an application, and the computer running the application can be a physical machine or a virtual machine. For example, if the computer running the application is a physical computing device, the physical computing device can be a host or a terminal.

[0044] In addition to the distributed clusters described above, the distributed clusters in the embodiments of the present application may also be cloud servers, such as public cloud servers, private cloud servers, hybrid cloud servers, etc.; although a cloud server is not a traditional distributed cluster (because a cloud server is not a group of interconnected physical servers), it can be considered a virtual distributed cluster. This is because cloud servers are usually hosted on a hypervisor, which is a virtualization platform that allows multiple virtual machines to run on a single physical server. Each virtual machine is equivalent to a node, and multiple nodes constitute a distributed cluster.

[0045] like Figure 2 As shown, Figure 2 The structure and organization diagram of the distributed system provided in this application. The hardware implementation in the distributed system 200 can refer to Figure 1 The relevant contents in the article will not be elaborated here. Figure 2 As shown, the distributed system 200 includes one or more nodes, and the data in a node is processed and accessed by a server node.

[0046] A distributed system can also be called a file system. A file system is a structured data file storage and organization form. Figure 2 As shown, a node includes multiple directories, such as Figure 2 The shown directories are directory 1, directory 2, and directory 3. Multiple files are stored in one directory, and each file can be used to store one or more groups of data.

[0047] The above introduces the distributed system from the perspective of hardware structure and software implementation. In the application process of the distributed system, in order to ensure the high availability of the distributed system, it is necessary to perform data governance on the distributed system regularly to delete or dump files with low access popularity in the distributed system. When performing data governance on files stored in the distributed system, it is necessary to determine the access popularity of each file in the entire distributed system so as to delete or dump files with low access popularity. In general, the number of directories is much smaller than the number of files. However, the directory metadata table does not record the file-level read heat information in the directory, but records the directory-level read heat information. At present, the update scenario of the read heat information atime in the directory metadata table is shown in Table 1. The read heat information atime is updated only when the directory is listed. In other words, the current read heat information atime cannot indicate the access heat of files in the directory. This leads to the current data governance, which requires access to the metadata of all files in the entire distributed system every time, and the data governance efficiency is very low.

[0048] Table 1

[0049]

[0050] In order to solve the above problems, an embodiment of the present application provides a data processing method, which is applied to the server of a distributed system. The method performs data management by accessing the latest reading time point of the directory. Compared with the current data management, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data management.

[0051] Figure 3 FIG. 1 is a flow chart showing a data processing method provided by an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0052] Step S310: receiving a data management request for M directories of a distributed file system.

[0053] The data management request may be initiated by a user on the client and sent by the client to the server to request the management of M directories.

[0054] Step S320: Read the M latest reading time points recorded in the M directories respectively in response to the data governance request.

[0055] Wherein, each directory corresponds to a latest reading time point, and M is a positive integer greater than 1 or equal to 1. The latest reading time point of each directory is the latest time point among the N time points at which the server reads each file in the directory based on N file reading requests of at least one file in each directory, and N is a positive integer greater than 1 or equal to 1. Exemplarily, N is 3, and the time points corresponding to the N file reading requests are respectively: T1, T2 and T3, T3 is later than T1 and T2, then the latest time point of the directory is T3. The files requested by the N file reading requests can be the same file or different files. The latest reading time point is determined according to multiple reading time points generated by the client. Specifically, when the client receives a read request for a file (indicating a file and a file storage address, the file storage address includes the directory to which the file belongs and the server to which the directory belongs, and exemplary, the file storage address can include a server identifier and a directory identifier inode), it records the corresponding reading time point and sends the reading time point and the corresponding file storage address to the server. The read request may request one or more files in a directory, or may request one or more files in different directories. Different directories may be stored on the same server or on different servers, without limitation.

[0056] After receiving multiple sets of read time points and corresponding file storage addresses, the server can determine multiple read time points corresponding to the same directory, and determine the latest read time point of the directory from the multiple read time points. For example, the read requests for files are continuously received within a preset period, and file A, file B, and file C are requested respectively (corresponding to read times T1, T2, and T3, respectively). The file storage addresses corresponding to file A, file B, and file C are: server P1-directory I, server P1-directory I, server P2-directory U. Then the read time points of directory I include T1 and T2, and the read time points of directory U include T3. T2 is later than T1, so the latest read time point of directory I is T2, and the latest read time point of directory U is T3. Based on this, the M latest read time points recorded in the M directories can be read.

[0057] In one embodiment, the client can continuously receive the read request of the file within a preset period to obtain multiple read requests of the file. In other words, the client can aggregate the read time points of a preset period and then send it to the server. In this implementation, when the client aggregates the read time points of a preset period, it can determine the latest read time point of the same directory, and use the determined latest read time point as the content sent to the server. The specific determination process can refer to the principle of determining the latest read time point by the server, and will not be repeated. The client may have multiple, and when the server receives multiple latest read time points of the same directory from multiple clients, the final latest read time point can be determined from the multiple latest read time points. In this way, the communication overhead caused by the real-time transmission of file heat between the client and the server is reduced. The preset period can be flexibly set according to the application scenario. Exemplarily, if the file heat accumulates faster, the preset period can be set shorter, otherwise, the preset period can be set longer. For example, the preset period can be set to 100ms.

[0058] Step S330, determining the reading heat of each directory according to the latest reading time point of each directory.

[0059] When determining the read heat of each directory, the latest read time point of the directory can be directly used as the read heat. The latest read time point can also be converted into other data types that can be quantified and compared, without limitation.

[0060] Step S340, determining whether to perform data management on the directory corresponding to each read heat according to each read heat.

[0061] Among them, after determining the reading popularity of each directory, it can be determined whether to perform data governance on the directory based on it.

[0062] In the embodiment of the present application, data management is performed by accessing the latest reading time point of the directory. Compared with the current data management, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data management.

[0063] In one embodiment, step S330 may include:

[0064] Each read heat is compared with the heat threshold respectively. When the read heat is higher than the heat threshold, the directory is determined to be a directory with high read heat. When the read heat is lower than or equal to the heat threshold, the directory is determined to be a directory with low read heat.

[0065] Among them, the heat threshold can be set flexibly. Exemplarily, the threshold time point obtained by subtracting a preset period of time from the current time can be used as the heat threshold. Specifically, assuming that the current time is 15:00 on September 1, 2022, and the preset period of time is 72 hours, the heat threshold is 15:00 on August 29, 2022. The read heat of directory A is 12:00 on August 30, 2022, which is later than 15:00 on August 29, 2022 (equivalent to the read heat being higher than the heat threshold), then directory A is a directory with high read heat. The read heat of directory B is 15:00 on August 28, 2022, which is earlier than 15:00 on August 29, 2022 (equivalent to the read heat being lower than or equal to the heat threshold), then directory B is a directory with low read heat.

[0066] In the embodiment of the present application, by comparing the heat threshold and the read heat, it is possible to accurately determine whether the directory has low read heat or high read heat.

[0067] In one embodiment, if Figure 4 As shown, step S340 may include:

[0068] Step S410, abandon access to the file heat in the directory with low read heat.

[0069] If the read heat of a directory is low, it means that the files in the directory have not been used recently. In this case, for the directory with low read heat, it is not necessary to perform file heat access on the files in it. Instead, the directory with low read heat is processed as follows: delete, infrequently store, or archive all files in the directory with low read heat.

[0070] Step S420, access the file heat in the directory with high read heat.

[0071] Among them, if the read heat of the directory is high, it means that some files in the directory have been used recently. At this time, the file heat of the directory with high read heat can be accessed to differentiate and manage the files in the directory. The attribute information of the file records the file heat of the file, which is similar to the read heat of the directory. The file heat of the file is determined according to the latest read time point of the file. The specific determination process of the file heat can refer to the determination process of the read heat of the directory, which will not be repeated here.

[0072] Step S430: Perform data management on directories with high read popularity according to file popularity.

[0073] Among them, according to the file popularity, it is possible to determine which files in the directory with high read popularity have high file popularity and which files have low file popularity, and then perform data management. Specifically, for files with low file popularity, one or more of the following means can be used for data management: deleting low-temperature files, storing low-temperature files infrequently, or archiving low-temperature files. For files with high file popularity, they can continue to be stored on the server.

[0074] In the embodiment of the present application, targeted data management is performed on the directories according to the different reading heats of the directories. The data management process only needs to access the file heats of the directories with high reading heats, thereby improving data management efficiency.

[0075] In one embodiment, if Figure 5 As shown, the distributed file system also includes multiple home storage nodes, and the method also includes:

[0076] Step S510, respectively confirming the H storage nodes to which the M directories belong.

[0077] Wherein, H is a positive integer greater than 1 or equal to 1, and the server node storing the directory is called the home storage node of the directory. The server node can store one or more directories, and the M directories may be stored in the same home storage node, or in different home storage nodes. The server performing steps S510 to S520 and the home storage node of the directory may be the same node or different nodes.

[0078] The file read request may be for one file or multiple files, and the multiple files requested may be stored in the same home node or in different home nodes. In the scenario where the files requested by the read request are stored in different home nodes, when the access node receives the file read request, it will send read requests to different home nodes respectively to obtain all the files requested by the read request. For example, Figure 4 As shown, the files requested by the read request are stored in the home node 41 and the home node 42 respectively, and the access node 40 will send read requests to the home node 41 and the home node 42 respectively to obtain all the files requested by the read request.

[0079] Step S520: Record the M latest reading time points in the H home storage nodes respectively.

[0080] Assume that there are three directories A, B, and C, then M is 3, and the latest reading time points corresponding to directories A, B, and C are T1, T2, and T3, respectively. Figure 6As shown, the storage nodes belonging to directories A and B are node 1, and the storage node belonging to directory C is node 2. Both node 1 and node 2 are not servers that execute steps S510 to S520. Then T1 and T2 are recorded in the corresponding directory attribute information in node 1, and T3 is recorded in the corresponding directory attribute information in node 2. Exemplarily, the latest reading time point can be recorded in the metadata table of the directory in the form of ext_dir_atime. In other words, once a file in a directory is read, the ext_dir_atime in the directory will be updated. The update scenario of ext_dir_atime in the metadata table is shown in Table 2.

[0081] Table 2

[0082]

[0083] In the embodiment of the present application, the latest reading time point of the directory is recorded in the corresponding home storage node, which can be used to determine the reading popularity of the directory in the home storage node.

[0084] In summary, the embodiment of the present application performs data management by accessing the latest reading time point of the directory. Compared with the current data management, the complexity of data access is reduced from the file level to the directory level, which greatly improves the efficiency of data management.

[0085] The above mainly introduces the scheme provided by the embodiment of the present application from the perspective of the execution logic of each step. It is understandable that, in order to realize the above functions, the service end includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiment disclosed herein, the method of the embodiment of the present application can be implemented in the form of hardware, software, or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0086] Figure 7 A structural diagram of a data processing device 700 is shown, and the data processing device is applied to a server. Figure 7Each module in the device shown has the function of the corresponding step in the method embodiment and can achieve its corresponding technical effect. The corresponding beneficial effects of the execution steps of each module can refer to the description of the corresponding steps in the method embodiment and will not be repeated here. The function can be implemented by hardware or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the data processing device includes: a receiving module 710, which is used to receive a data governance request for M directories of a distributed file system; an acquisition module 720, which is used to read the M latest reading time points recorded in the M directories respectively in response to the data governance request; a determination module 730, which is also used to determine the reading heat of each directory according to the latest reading time point of each directory; the determination module is also used to determine whether to perform data governance on the directory corresponding to each reading heat according to each reading heat.

[0087] Among them, each module described above can be implemented by software or by hardware. Exemplarily, the implementation of the receiving module 710 is introduced below by taking the receiving module 710 as an example. Similarly, the implementation of the acquisition module 720 and the determination module 730 can refer to the implementation of the receiving module 710.

[0088] As an example of a software functional unit, the receiving module 710 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the receiving module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0089] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0090] As an example of a hardware functional unit, the receiving module 710 may include at least one computing device, such as a server, etc. Alternatively, the receiving module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0091] The multiple computing devices included in the receiving module 710 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 710 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 710 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0092] It should be noted that, in other embodiments, the receiving module 710 can be used to execute any step in the data processing method, the acquisition module 720 and the determination module 730 can be used to execute any step in the data processing method, and the steps that the receiving module 710, the acquisition module 720 and the determination module 730 are responsible for implementing can be specified as needed. The full functions of the data processing device are realized by respectively implementing different steps in the data processing method through the receiving module 710, the acquisition module 720 and the determination module 730.

[0093] The present application also provides a computing device 100. Figure 8 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0094] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 104 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0095] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0096] The memory 106 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0097] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned receiving module 710, the acquisition module 720, and the determination module 730, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0098] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned data processing apparatus, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0099] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0100] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0101] like Fig. 9 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0102] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the data processing method.

[0103] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data processing apparatus. That is, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of one or more of the receiving module 710, the obtaining module 720 and the determining module 730.

[0104] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the receiving module 710. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the obtaining module 720 and the determining module 730.

[0105] Fig.10 The connection method between the computing device clusters shown may be based on the needs of the data processing method provided in the present application, and therefore it is considered that the functions implemented by the acquisition module 720 and the determination module 730 are handed over to the computing device 100B for execution.

[0106] It should be understood that Fig.10The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0107] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0108] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the data processing method.

[0109] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions for executing partial functions of the data detection system. That is, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of the data processing device.

[0110] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a data processing method or a data processing method.

[0111] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a server of a distributed file system, and the method comprises: Receiving a data management request for M directories of the distributed file system; In response to the data governance request, read the M latest reading time points recorded in the M directories respectively, wherein each directory record has a latest reading time point, and M is a positive integer greater than 1 or equal to 1; Determine the read heat of each directory according to the latest read time point of each directory, wherein the latest read time point of each directory is the latest time point among N time points at which the server reads each file in the directory based on N file read requests of at least one file in each directory, and N is a positive integer greater than 1 or equal to 1; Determine whether to perform data management on the directory corresponding to each read heat according to each read heat.

2. The method according to claim 1, characterized in that Determining the read heat of each directory according to the latest read time point of each directory includes: Each read heat is compared with a heat threshold value respectively. When the read heat is higher than the heat threshold value, the directory is determined to be a directory with high read heat. When the read heat is lower than or equal to the heat threshold value, the directory is determined to be a directory with low read heat.

3. The method according to claim 2, characterized in that The determining, according to each read heat, whether to perform data management on the directory corresponding to each read heat includes: Abandon accessing the file temperature in the directory with low read temperature; and Access the file heat in the directory with high read heat, and perform data management on the directory with high read heat according to the file heat.

4. The method according to any one of claims 1 to 3, characterized in that: The data governance includes one or more of the following: Delete low-frequency files, store low-frequency files infrequently, or archive low-frequency files.

5. The method according to any one of claims 1 to 4, characterized in that: The distributed file system further includes a plurality of home storage nodes, and the method further includes: Respectively confirm the H belonging storage nodes where the M directories are located; The M latest reading time points are recorded in the H belonging storage nodes respectively, where H is a positive integer greater than 1 or equal to 1.

6. A data processing device, characterized in that: The device is applied to a server of a distributed file system, and the device includes: A receiving module, configured to receive a data management request for M directories of the distributed file system; an acquisition module, configured to respectively determine, in response to the data governance request, M latest reading time points corresponding to the M directories, wherein each directory corresponds to a latest reading time point, and M is a positive integer greater than or equal to 6; The determination module is further used to determine the reading heat of each directory according to the latest reading time point of each directory, wherein the latest reading time point of each directory is the latest time point among N time points at which the server reads each file in the directory based on N file read requests of at least one file in each directory, and N is a positive integer greater than 6 or equal to 6; The determination module is further used to determine whether to perform data management on the directory corresponding to each reading heat according to each reading heat.

7. The device according to claim 6, characterized in that The determining module is specifically used for: Each read heat is compared with a heat threshold value respectively. When the read heat is higher than the heat threshold value, the directory is determined to be a directory with high read heat. When the read heat is lower than or equal to the heat threshold value, the directory is determined to be a directory with low read heat.

8. The device according to claim 7, characterized in that The determining module is specifically used for: Abandon accessing the file heat in the directory with low read heat; and access the file heat in the directory with high read heat, and perform data management on the directory with high read heat according to the file heat.

9. The device according to any one of claims 6 to 8, characterized in that The data governance includes one or more of the following: Delete low-frequency files, store low-frequency files infrequently, or archive low-frequency files.

10. The device according to any one of claims 6 to 9, characterized in that The distributed file system further includes a plurality of home storage nodes, the device further includes a recording module, and the determination module is further used to respectively confirm the H home storage nodes where the M directories are located; The recording module is used to record the M latest reading time points in the H belonging storage nodes respectively, where H is a positive integer greater than 6 or equal to 6.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 5.

Images