Distributed file system metadata caching method, device, terminal and storage medium

Abstract

The present invention discloses a distributed file system metadata caching method, device, terminal and storage medium. A three-level cache mechanism is designed to select cache, and metadata to be used is placed in the first and second level cache queues, and metadata with low usage rate is placed in the third level cache queue. When eliminating, metadata with the longest cache time in the third level cache queue is eliminated, so as to retain the metadata to be used, ensure the hit rate of metadata, reduce memory usage, shorten the metadata access IO path, reduce network communication pressure, disk access IO pressure, and set a reasonable cache amount threshold, so as to effectively realize the optimization of metadata cache performance on the basis of controllable memory.

Description

Technical Field

[0001] The present invention relates to the field of distributed file system metadata caching, and in particular to a distributed file system metadata caching method, device, terminal and storage medium. Background Art

[0002] In the era of big data, unstructured data is growing explosively and rapidly. Massive data is saved, managed, and used in file formats, resulting in a huge number of files. Metadata is data that describes data, which refers to system data used to describe the characteristics of a file, such as access rights, file owners, and distribution information of file data blocks.

[0003] For small file scenarios in distributed file systems (object storage), improving the utilization of metadata cache can reduce the number of communications between the client and the metadata server and the number of IOs with the underlying disk, thereby reducing network pressure and disk pressure. In actual scenarios, due to user search operations, such as directory indexing, a large amount of useless metadata cache will be occupied, making the hit rate of the metadata cache very low; on the other hand, in small file high-performance read and write scenarios, due to the high requirements for business performance, it is often necessary to fully cache all metadata caches per unit time, which also includes useless metadata caches created by search operations, resulting in very high memory usage of the storage system. Summary of the invention

[0004] To solve the above problems, the present invention provides a distributed file system metadata caching method, device, terminal and storage medium, designs a three-level cache mechanism, performs selective cache, places metadata to be used in the first and second level cache queues, places metadata with low usage rate in the third level cache queue, and eliminates metadata with the longest cache time from the third level cache queue during elimination, thereby ensuring the hit rate of metadata and reducing memory usage.

[0005] In a first aspect, the technical solution of the present invention provides a distributed file system metadata caching method, comprising the following steps:

[0006] Configure the third-level cache;

[0007] When there is new metadata cache, the new metadata is inserted into the first-level cache queue;

[0008] When the cache amount in the first-level cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the first-level cache queue is taken out until the cache amount in the first-level cache queue is less than its cache amount threshold, and the taken out metadata is used as the metadata to be moved;

[0009] The metadata to be read in the metadata to be moved is moved to the L2 cache queue, and the other metadata is moved to the L3 cache queue;

[0010] When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata cached the longest in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold;

[0011] When the cache amount in the third-level cache queue is greater than its cache amount threshold, metadata cached the longest in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold.

[0012] Furthermore, the metadata to be read in the metadata to be moved is moved to the secondary cache queue, and the other metadata is moved to the tertiary cache queue, specifically including:

[0013] Determine whether the file corresponding to the metadata to be moved has been read or written or retrieved separately;

[0014] If so, the metadata to be moved is moved to the L2 cache queue; otherwise, it is moved to the L3 cache queue.

[0015] Furthermore, when there is new metadata cache, the new metadata is inserted into the first-level cache queue, which specifically includes:

[0016] When there is new metadata cache, the new metadata is inserted into the head of the first-level cache queue.

[0017] Furthermore, when there is new metadata cache, the new metadata is inserted into the first-level cache queue, which specifically includes:

[0018] When there is new metadata cached, determine whether the new metadata has been cached in the first-level cache queue, the second-level cache queue, or the third-level cache queue;

[0019] If so, the cached metadata is taken out and inserted into the head of the first-level cache queue.

[0020] Furthermore, when the cache amount in the secondary cache queue is greater than its cache amount threshold, the metadata cached the longest in the secondary cache queue is moved to the third-level cache queue until the cache amount in the secondary cache queue is less than its cache amount threshold, specifically:

[0021] When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata at the tail of the L2 cache queue is moved to the head of the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold;

[0022] When the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold. Specifically:

[0023] When the cache amount in the third-level cache queue is greater than its cache amount threshold, the tail metadata in the third-level cache queue is eliminated until the cache amount of the third-level cache queue is less than its cache amount threshold.

[0024] Furthermore, the method further includes calculating the cache amount threshold of each level of cache queue according to the following formula:

[0025] k=k1+k2;

[0026] n=k1+k3;

[0027] m=k1*g1+(k1+k2)*g2;

[0028] Among them, k is the maximum file read and write access volume per unit time of the distributed file system node, n is the maximum number of files in a single directory in the scenario, m is the memory capacity of the distributed file system node, g1 is the size of the first-level cache queue, g2 is the size of the second-level cache queue, k1 is the cache capacity threshold of the first-level cache queue, k2 is the cache capacity threshold of the second-level cache queue, and k3 is the cache capacity threshold of the third-level cache queue.

[0029] In a second aspect, the technical solution of the present invention provides a distributed file system metadata cache device, comprising:

[0030] Cache configuration module: configure the three-level cache;

[0031] New metadata insertion module: when there is new metadata cache, the new metadata is inserted into the first-level cache queue;

[0032] Level 1 cache queue elimination module: When the cache volume in the level 1 cache queue is greater than its cache volume threshold, the metadata with the longest cache time in the level 1 cache queue is taken out until the cache volume in the level 1 cache queue is less than its cache volume threshold, and the taken out metadata is used as the metadata to be moved; the metadata to be read in the metadata to be moved is moved to the level 2 cache queue, and the other metadata is moved to the level 3 cache queue;

[0033] L2 cache queue elimination module: When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold;

[0034] Level 3 cache queue elimination module: When the cache amount in the level 3 cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the level 3 cache queue is eliminated until the cache amount in the level 3 cache queue is less than its cache amount threshold.

[0035] Further, the module includes a cache threshold configuration module;

[0036] The cache threshold configuration module calculates the cache thresholds of cache queues at each level according to the following formula:

[0037] k=k1+k2;

[0038] n=k1+k3;

[0039] m=k1*g1+(k1+k2)*g2;

[0040] Among them, k is the maximum file read and write access volume per unit time of the distributed file system node, n is the maximum number of files in a single directory in the scenario, m is the memory capacity of the distributed file system node, g1 is the size of the first-level cache queue, g2 is the size of the second-level cache queue, k1 is the cache capacity threshold of the first-level cache queue, k2 is the cache capacity threshold of the second-level cache queue, and k3 is the cache capacity threshold of the third-level cache queue.

[0041] In a third aspect, the technical solution of the present invention provides a terminal, including:

[0042] processor;

[0043] A memory for storing execution instructions of the processor;

[0044] The processor is configured to execute any of the above methods.

[0045] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0046] A distributed file system metadata caching method, device, terminal and storage medium provided by the present invention have the following beneficial effects compared with the prior art: a three-level cache mechanism is designed to select cache, metadata to be used is placed in the first and second level cache queues, metadata with low usage rate is placed in the third level cache queue, and metadata with the longest cache time in the third level cache queue is eliminated during elimination, thereby retaining the metadata to be used, ensuring the hit rate of metadata, reducing memory usage, shortening the metadata access IO path, reducing network communication pressure, disk access IO pressure, and setting a reasonable cache amount threshold, so as to effectively optimize the metadata cache performance on the basis of controllable memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of a distributed file system metadata caching method provided in Embodiment 1 of the present invention;

[0049] Figure 2 A schematic block diagram of the structure of a distributed file system metadata cache device provided in Embodiment 2 of the present invention;

[0050] Figure 3 A schematic diagram of the structure of a terminal provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] Embodiment 1

[0053] For small file scenarios in distributed file systems (object storage), improving the utilization of metadata cache can reduce the number of communications between the client and the metadata server and the number of IOs with the underlying disk, thereby reducing network pressure and disk pressure. In actual scenarios, due to user search operations, such as directory indexing, a large amount of useless metadata cache will be occupied, making the hit rate of the metadata cache very low; on the other hand, in small file high-performance read and write scenarios, due to the high requirements for business performance, it is often necessary to fully cache all metadata caches per unit time, which also includes useless metadata caches created by search operations, resulting in very high memory usage of the storage system.

[0054] Therefore, this embodiment 1 provides a distributed file system metadata caching method, designs a three-level cache mechanism, selects the cache, places the metadata to be used in the first and second level cache queues, places the metadata with low usage rate in the third level cache queue, and eliminates the metadata with the longest cache time from the third level cache queue, thereby ensuring the hit rate of the metadata and reducing memory usage.

[0055] like Figure 1 The figure is a schematic flow chart of a distributed file system metadata caching method provided in the first embodiment, which includes the following steps.

[0056] S1, configure the third-level cache.

[0057] That is, set up a three-level cache mechanism and configure the first-level cache queue, the second-level cache queue, and the third-level cache queue.

[0058] S2: When there is new metadata cache, the new metadata is inserted into the first-level cache queue.

[0059] That is, the latest metadata is cached in the first-level cache queue. In the three-level cache mechanism, the metadata stored in the first-level cache queue is the metadata that has been cached for a shorter time.

[0060] S3, when the cache amount in the first-level cache queue is greater than its cache amount threshold, the metadata cached the longest in the first-level cache queue is taken out until the cache amount in the first-level cache queue is less than its cache amount threshold, and the taken out metadata is used as the metadata to be moved.

[0061] That is, the metadata in the first-level cache queue is sorted from long to short according to the cache time (the time it has been cached since it was inserted into the first-level cache queue), and the metadata that exceeds the cache threshold is taken out starting from the longest cache time, so that the cache amount in the first-level cache queue is less than its cache threshold.

[0062] S4: The metadata to be read in the metadata to be moved is moved to the secondary cache queue, and the other metadata is moved to the tertiary cache queue.

[0063] When the metadata in the first-level cache queue exceeds the threshold, the excess part will be eliminated from the second-level and third-level cache queues. In order to improve the metadata usage hit rate and cache performance, in conjunction with the elimination mechanism of the third-level cache queue, this embodiment first detects whether the metadata is about to be read by the server when eliminating it from the second-level and third-level cache queues, and puts the metadata to be read into the second-level cache queue, and the others into the third-level cache queue.

[0064] S5, when the cache amount in the secondary cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the secondary cache queue is moved to the third-level cache queue until the cache amount in the secondary cache queue is less than its cache amount threshold.

[0065] S6, when the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold.

[0066] When eliminating data from the L2 cache queue to the L3 cache queue, and when eliminating metadata from the L3 cache queue, the elimination is done in descending order of cache time to avoid affecting the metadata usage hit rate.

[0067] This method selects caches and ensures that the cached items are to be used under the premise of certain cache items, thereby improving cache utilization and effectively reducing memory usage. For example, if 50,000 files are read and written based on 100,000 files, this method can achieve full metadata caching during the reading and writing process with 50,000 memory, while the existing process requires 100,000 memory to ensure the metadata hit rate.

[0068] The present embodiment provides a distributed file system metadata caching method, which designs a three-level cache mechanism, performs selective caching, places metadata to be used in the first and second level cache queues, places metadata with low usage rate in the third level cache queue, and eliminates metadata with the longest cache time from the third level cache queue during elimination, thereby retaining metadata to be used, ensuring the hit rate of metadata, reducing memory usage, shortening metadata access IO path, reducing network communication pressure, disk access IO pressure, and setting a reasonable cache amount threshold, thereby effectively optimizing metadata cache performance on the basis of controllable memory.

[0069] In some specific embodiments, determining whether the metadata is to be read can be achieved by determining whether the file corresponding to the metadata has been used for reading and writing or has been searched separately. If the corresponding file has been used for reading and writing or has been indexed separately, it means that the metadata is to be read. Specifically, in step S4, the metadata to be read is moved to the secondary cache queue in the metadata to be moved, and the other metadata is moved to the tertiary cache queue, which specifically includes:

[0070] Step 1: determine whether the file corresponding to the metadata to be moved has been read, written, or retrieved separately;

[0071] Step 2: If yes, move the metadata to be moved to the L2 cache queue; otherwise, move it to the L3 cache queue.

[0072] It should be noted that if the corresponding file has been used for reading and writing or has been searched separately, there will be a corresponding mark in the metadata, and the mark status is detected to determine whether the corresponding file has been used for reading and writing or has been searched separately.

[0073] In this embodiment, elimination is performed based on the cache time length. To facilitate metadata elimination, in some specific embodiments, data entering the cache queue is inserted into the head of the queue and then eliminated from the tail.

[0074] Specifically, in step S2, when there is new metadata cache, the new metadata is inserted into the first-level cache queue, specifically:

[0075] When there is new metadata cache, the new metadata is inserted into the head of the first-level cache queue.

[0076] Correspondingly, in step S4, the metadata to be read in the metadata to be moved is moved to the secondary cache queue, and the other metadata is moved to the tertiary cache queue, specifically:

[0077] The metadata to be read in the metadata to be moved is inserted into the head of the secondary cache queue, and the other metadata is inserted into the tertiary cache queue.

[0078] In step S5, when the cache amount in the second-level cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the second-level cache queue is moved to the third-level cache queue until the cache amount of the second-level cache queue is less than its cache amount threshold. Specifically, the metadata at the tail of the second-level cache queue is moved to the head of the third-level cache queue until the cache amount of the second-level cache queue is less than its cache amount threshold.

[0079] In step S6, when the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the third-level cache queue is eliminated until the cache amount of the third-level cache queue is less than its cache amount threshold. Specifically, the tail metadata in the third-level cache queue is eliminated until the cache amount of the third-level cache queue is less than its cache amount threshold.

[0080] In some specific embodiments, the metadata to be cached may have been stored in the cache queue, so when there is new metadata cache in step S2, it is first determined whether the corresponding metadata has been cached, and if so, it is extracted to the head of the first-level cache queue, otherwise it is directly inserted into the head of the first-level cache queue. Specifically, the following steps are included:

[0081] Step 1: when there is new metadata cache, determine whether the new metadata has been cached in the first-level cache queue, the second-level cache queue or the third-level cache queue;

[0082] Step 2: If yes, the cached metadata is taken out and inserted into the head of the first-level cache queue.

[0083] To improve the utilization rate of the metadata cache, this embodiment configures the cache threshold of each cache queue according to the node file read and write access volume and the node memory volume.

[0084] First, based on actual business analysis, the maximum file read and write access volume k (in units, that is, the number of entries accessed) per unit time of the distributed file system node is counted.

[0085] To achieve optimal performance, the first-level cache queue cache threshold k1 and the second-level cache queue cache threshold k2 (in units) are set to meet the following requirements:

[0086] k=k1+k2.

[0087] The third-level cache queue is an out-of-bounds buffer layer and is arranged reasonably according to the node memory situation. The utilization rate of the third-level cache queue is very low, so it does not need to be set too large normally, but it should meet the following requirements:

[0088] n=k1+k3;

[0089] Otherwise, the valid metadata retrieved from the directory may be lost immediately and need to be reloaded, resulting in IO waste. Where n is the maximum number of files in a single directory in the scene.

[0090] According to the memory limit m (in KB) of the distributed file system node, the cache threshold of each cache queue meets the following requirements:

[0091] m=k1*g1+(k1+k2)*g2;

[0092] Where g1 is the first-level cache queue size, g2 is the second-level cache queue size (in KB).

[0093] Rearrange the system of equations:

[0094] k=k1+k2;

[0095] n=k1+k3;

[0096] m=k1*g1+(k1+k2)*g2.

[0097] Among them, k, n, m, g1 and g2 are all quantitative, and k1, k2 and k3, that is, the cache amount threshold of each cache queue, can be calculated.

[0098] Embodiment 2

[0099] This embodiment provides a distributed file system metadata caching device, which is used to implement the aforementioned distributed file system metadata caching method.

[0100] like Figure 2 The figure is a schematic block diagram of the structure of a distributed file system metadata cache device provided in the second embodiment, which includes the following functional modules.

[0101] Cache configuration module 101: configure the three-level cache.

[0102] New metadata inserting module 102: when there is new metadata cache, the new metadata is inserted into the first-level cache queue.

[0103] First-level cache queue elimination module 103: When the cache amount in the first-level cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the first-level cache queue is taken out until the cache amount in the first-level cache queue is less than its cache amount threshold, and the taken out metadata is used as the metadata to be moved; the metadata to be read in the metadata to be moved is moved to the second-level cache queue, and the other metadata is moved to the third-level cache queue.

[0104] L2 cache queue elimination module 104: When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata cached the longest in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold.

[0105] The third-level cache queue elimination module 105: when the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached the longest in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold.

[0106] In some specific embodiments, in order to improve metadata cache utilization, this embodiment configures the cache threshold of each cache queue according to the node file read and write access volume and the node memory volume. Specifically, the device also includes a cache threshold configuration module 106.

[0107] The cache threshold configuration module 106 calculates the cache thresholds of cache queues at each level according to the following formula:

[0108] k=k1+k2;

[0109] n=k1+k3;

[0110] m=k1*g1+(k1+k2)*g2;

[0111] Among them, k is the maximum file read and write access volume per unit time of the distributed file system node, n is the maximum number of files in a single directory in the scenario, m is the memory capacity of the distributed file system node, g1 is the size of the first-level cache queue, g2 is the size of the second-level cache queue, k1 is the cache capacity threshold of the first-level cache queue, k2 is the cache capacity threshold of the second-level cache queue, and k3 is the cache capacity threshold of the third-level cache queue.

[0112] The distributed file system metadata caching device of this embodiment is used to implement the aforementioned distributed file system metadata caching method. Therefore, the specific implementation method of the device can be seen in the embodiment part of the distributed file system metadata caching method in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be introduced in detail here.

[0113] In addition, since the distributed file system metadata caching device of this embodiment is used to implement the aforementioned distributed file system metadata caching method, its function corresponds to that of the aforementioned method and will not be described in detail here.

[0114] Embodiment 3

[0115] Figure 3The present invention is a schematic diagram of the structure of a terminal device 300 provided in an embodiment of the present invention. The terminal device 300 can be used to execute the distributed file system metadata caching method provided in an embodiment of the present invention.

[0116] The terminal device 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention. It may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0117] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.

[0118] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0119] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0120] Embodiment 4

[0121] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0122] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.

[0123] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A distributed file system metadata caching method, characterized in that: The following steps are involved: Configure the third-level cache; When there is new metadata cache, insert the new metadata into the first-level cache queue, including when there is new metadata cache, determine whether the new metadata has been cached in the first-level cache queue, the second-level cache queue or the third-level cache queue, and if so, take out the cached metadata and insert it into the head of the first-level cache queue; When the cache amount in the first-level cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the first-level cache queue is taken out until the cache amount in the first-level cache queue is less than its cache amount threshold, and the taken out metadata is used as the metadata to be moved; The metadata to be read in the metadata to be moved is moved to the secondary cache queue, and the other metadata is moved to the tertiary cache queue, including determining whether the file corresponding to the metadata to be moved has been read and written or retrieved separately. If so, the metadata to be moved is moved to the secondary cache queue, otherwise it is moved to the tertiary cache queue; When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata cached the longest in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold; When the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached the longest in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold; This method calculates the cache threshold of each level of cache queue according to the following formula: k=k1+k2; n=k1+k3; m=k1*g1+(k1+k2)*g2; Among them, k is the maximum file read and write access volume per unit time of the distributed file system node, n is the maximum number of files in a single directory in the scenario, m is the memory capacity of the distributed file system node, g1 is the size of the first-level cache queue, g2 is the size of the second-level cache queue, k1 is the cache capacity threshold of the first-level cache queue, k2 is the cache capacity threshold of the second-level cache queue, and k3 is the cache capacity threshold of the third-level cache queue.

2. The distributed file system metadata caching method according to claim 1, characterized in that: When there is new metadata cache, the new metadata is inserted into the first-level cache queue, including: When there is new metadata cache, the new metadata is inserted into the head of the first-level cache queue.

3. The distributed file system metadata caching method according to claim 2, characterized in that: When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold. Specifically: When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata at the tail of the L2 cache queue is moved to the head of the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold; When the cache amount in the third-level cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the third-level cache queue is eliminated until the cache amount in the third-level cache queue is less than its cache amount threshold. Specifically: When the cache amount in the third-level cache queue is greater than its cache amount threshold, the tail metadata in the third-level cache queue is eliminated until the cache amount of the third-level cache queue is less than its cache amount threshold.

4. A distributed file system metadata cache device, characterized in that: include, Cache configuration module: configure the three-level cache; New metadata insertion module: when there is new metadata cache, the new metadata is inserted into the first-level cache queue, including when there is new metadata cache, determining whether the new metadata has been cached in the first-level cache queue, the second-level cache queue or the third-level cache queue, and if so, taking out the cached metadata and inserting it into the head of the first-level cache queue; Level 1 cache queue elimination module: When the cache amount in the level 1 cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the level 1 cache queue is taken out until the cache amount in the level 1 cache queue is less than its cache amount threshold, and the taken out metadata is used as the metadata to be moved; The metadata to be read in the metadata to be moved is moved to the secondary cache queue, and the other metadata is moved to the tertiary cache queue, including determining whether the file corresponding to the metadata to be moved has been read and written or retrieved separately. If so, the metadata to be moved is moved to the secondary cache queue, otherwise it is moved to the tertiary cache queue; L2 cache queue elimination module: When the cache amount in the L2 cache queue is greater than its cache amount threshold, the metadata with the longest cache time in the L2 cache queue is moved to the L3 cache queue until the cache amount in the L2 cache queue is less than its cache amount threshold; Level 3 cache queue elimination module: When the cache amount in the level 3 cache queue is greater than its cache amount threshold, the metadata cached for the longest time in the level 3 cache queue is eliminated until the cache amount in the level 3 cache queue is less than its cache amount threshold; Cache threshold configuration module: Calculates the cache thresholds of each cache queue according to the following formula: k=k1+k2; n=k1+k3; m=k1*g1+(k1+k2)*g2; Among them, k is the maximum file read and write access volume per unit time of the distributed file system node, n is the maximum number of files in a single directory in the scenario, m is the memory capacity of the distributed file system node, g1 is the size of the first-level cache queue, g2 is the size of the second-level cache queue, k1 is the cache capacity threshold of the first-level cache queue, k2 is the cache capacity threshold of the second-level cache queue, and k3 is the cache capacity threshold of the third-level cache queue.

5. A terminal, characterized in that: include: processor; A memory for storing execution instructions of the processor; The processor is configured to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images