Massive small file storage method, device, NFS service node and storage medium

Abstract

The embodiments of the present application propose a method, device, NFS service node and storage medium for storing massive small files, and relate to the field of file storage. The growth factor of the request queue in the current time period is calculated based on the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end of the time period before the current time period; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue; and it is determined whether to enable the cache in the NFS service node to store the files in the request queue based on the growth factor. This can avoid continuous writing in the cache, and can extend the service life of the cache while meeting the low latency requirements of file storage.

Description

Technical Field

[0001] The present application relates to the field of file storage, and more specifically, to a method and device for storing massive small files, an NFS service node, and a storage medium. Background Art

[0002] With the rapid development of the network environment, various data are growing exponentially, and billions of small files are often generated. Under this circumstance, the problem of massive small files is a recognized problem in industry and academia.

[0003] At present, in order to meet the low latency requirements for storing massive small files, it is usually necessary to set up a cache on the hard disk drive as a disk buffer, such as a solid-state drive, to reduce the access latency for storing massive small files. However, this method requires continuous writing in the cache, which will seriously shorten the service life of the cache. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, device, NFS service node and storage medium for storing massive small files to avoid continuous writing in the cache, thereby extending the service life of the cache while meeting the low latency requirements of file storage.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, the present application provides a method for storing a large number of small files, which is applied to an NFS service node in a multi-node NFS service cluster, and the method includes:

[0007] Calculate the growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end of the time period before the current time period; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue;

[0008] Determine whether to enable a cache in the NFS service node to store files in the request queue according to the growth factor.

[0009] In an optional implementation manner, determining whether to enable a cache in the NFS service node to store files in the request queue according to the growth factor includes:

[0010] If the growth factor is not less than a first threshold, enabling a cache in the NFS service node, and storing the files in the request queue through the cache and the data storage cluster;

[0011] If the growth factor is less than the first threshold, the files in the request queue are stored by the data storage cluster.

[0012] In an optional implementation, if the growth factor is less than the first threshold, storing the files in the request queue by using the data storage cluster includes:

[0013] If the growth factor is not greater than a second threshold, reading out the file stored in the buffer, and storing the read file in the data storage cluster;

[0014] If the growth factor is greater than the second threshold and less than the first threshold, the files in the request queue are stored by the data storage cluster.

[0015] In an optional embodiment, the method further comprises:

[0016] For the target file stored in the request queue, an initial file identifier corresponding to the target file is generated, and the file name of the target file and the initial file identifier corresponding to the target file are stored in the metadata management cluster.

[0017] In an optional embodiment, the method further comprises:

[0018] After storing the target file in the request queue through the cache and storing the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster, a backup NFS service node is selected from the multi-node NFS service cluster, and a target file backup request is sent to the backup NFS service node, so that the backup NFS service node stores the target file according to the target file backup request and generates a backup file identifier corresponding to the target file, generates a file identifier corresponding to the target file according to the initial file identifier and the backup file identifier corresponding to the target file in the metadata management cluster, and updates the initial file identifier corresponding to the target file in the metadata management cluster according to the file identifier.

[0019] In an optional embodiment, the method further comprises:

[0020] After reading out the file stored in the cache and storing the read file through the data storage cluster, a new file identifier of the file is generated, and the file identifier corresponding to the file in the metadata management cluster is updated according to the new file identifier.

[0021] In an optional implementation, the calculation of the growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period includes:

[0022] The growth factor of the request queue in the current time period is calculated according to the following formula:

[0023] H=(Y+ZX) / X

[0024] Among them, H represents the growth factor of the request queue in the current time period, Y represents the number of write requests corresponding to the request queue at the end time of the time period before the current time period, Z represents the number of write requests added to the request queue in the current time period, and X represents the number of files stored in the data storage cluster in the time period before the current time period.

[0025] In a second aspect, the present application provides a massive small file storage device, which is applied to an NFS service node in a multi-node NFS service cluster, and the device includes:

[0026] A calculation module, configured to calculate a growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end of the time period before the current time period; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue;

[0027] The storage module is used to determine whether to enable the cache in the NFS service node to store the files in the request queue according to the growth factor.

[0028] In a third aspect, the present application provides an NFS service node, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement any method described in any of the aforementioned embodiments.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a method as described in any one of the aforementioned embodiments.

[0030] The method, device, NFS service node and storage medium for storing a large number of small files provided by the embodiments of the present application calculate the growth factor of the request queue in the current time period, so as to obtain the growth of the write requests in the request queue, and then determine whether it is necessary to enable the cache to store the files in the request queue under the current situation according to the growth. Since the growth factor is calculated to determine whether to enable the cache, continuous writing in the cache can be avoided, so that the service life of the cache can be extended while meeting the low latency requirements of file storage.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of an SSD / HDD layered hybrid storage architecture is shown;

[0034] Figure 2 A statistical chart showing the number of massive small files stored at different times of the day;

[0035] Figure 3 A system architecture diagram of a file storage system is shown;

[0036] Figure 4 A block diagram of an NFS service node provided in an embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram of a process of storing a large number of small files provided by an embodiment of the present application is shown;

[0038] Figure 6 Another schematic diagram of a process for storing a large number of small files provided in an embodiment of the present application is shown;

[0039] Figure 7 Another schematic diagram of a process of storing a large number of small files provided in an embodiment of the present application is shown;

[0040] Figure 8 Another schematic diagram of a process of storing a large number of small files provided in an embodiment of the present application is shown;

[0041] Fig. 9A A schematic diagram of aggregate storage of files is shown;

[0042] Fig. 9B A schematic diagram of the global large file ID format is shown;

[0043] Fig. 10A An architecture diagram of a three-node NFS service cluster is shown;

[0044] Fig. 10B A write logic architecture diagram is shown;

[0045] Fig.11 A functional module diagram of a file storage device provided in an embodiment of the present application is shown.

[0046] Icon: 100-SSD / HDD layered hybrid storage architecture; 110-SSD; 120-HDD; 200-multi-node NFS service cluster; 210-data storage cluster; 220-metadata management cluster; 300-NFS service node; 310-memory; 320-processor; 330-communication module; 20-peak dividing line; 21-trough dividing line; 400-three-node NFS service cluster; 410-first NFS service node; 411-cache; 412-storage module; 420-second NFS service node; 430-third NFS service node; 500-computing prediction module; 510-cache module. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0049] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0050] At present, with the rapid development of the Internet, the Internet of Things, and cloud computing, various types of data are showing an explosive growth trend. Various data not only require huge storage capacity, but also have the characteristics of a wide variety of data, large size changes, and fast flow. Therefore, various small files with smaller file sizes are often generated. The size of these small files ranges from a few KB to a few MB, and the number of such small files is huge, often in the order of tens of billions. These small files in the order of tens of billions are massive small files (Lots of Small Files, LOSF).

[0051] There are two major characteristics when storing massive small files. One is the huge number of files, and the other is the small file size. Therefore, the latency requirement for storing massive small files is high, and the access latency of a single small file is usually required to be within seconds. However, due to its performance characteristics, the hard disk drive (HDD) cannot meet the low latency requirement for storing massive small files. Therefore, the existing technology often uses an all-flash architecture or a hybrid storage architecture to store files, thereby reducing the access latency of small files.

[0052] For the hybrid storage architecture, a cache can be set for the hard disk drive as a disk buffer. When storing, a large number of files are first stored in the cache. Since the hardware resource space of the cache is limited, only short-term storage can be achieved. Therefore, the files in the cache need to be stored in the hard disk drive afterwards to reduce the number of files falling into the hard disk drive. In this mode, the file can directly return to the write success status after being written to the cache, thereby greatly reducing the access delay of small files.

[0053] In the prior art, a solid state disk (SSD) is generally used as a disk buffer of a hard disk drive, that is, a SSD / HDD layered hybrid storage architecture is used to store a large number of small files. Figure 1, is a schematic diagram of an SSD / HDD hierarchical hybrid storage architecture 100 used in the prior art, and the SSD / HDD hierarchical hybrid storage architecture 100 includes: an SSD 110 and an HDD 120. The SSD can receive a file write request for a file to be written and store the file to be written, and at the same time return a write success response to the user end, and then the files stored therein can be successively stored in the HDD.

[0054] It can be seen that if a cache is used as a disk buffer of a hard disk drive, it is necessary to continuously write and erase in the cache. Since the number of erasable times in the cache is limited, this method will seriously shorten the service life of the cache.

[0055] In addition, for the AI ​​security field, the number of small files stored can be counted around the clock to obtain the changes in the number of massive small files stored over time. Specifically, by capturing the real business statistical distribution data of a whole day from the existing network environment, the fluctuation of the number of small files stored at different times within a day can be simulated. For details, please refer to Figure 2 , which is a statistical chart of the number of massive small files stored at different times of the day, where the horizontal axis represents time and the vertical axis represents the number of operations per second (Operation Per Second, OPS). On this basis, Figure 2 Indicates the changes in the number of massive small file storage operations at different times, which can reflect the changes in the business characteristics of massive small file storage over time. Figure 2 It can be seen that the business characteristics of massive small file storage can be manifested as a double peak pattern in the morning and evening peak periods at different times of the day, that is, the number of small file storage operations will increase exponentially during the morning and evening rush hours, and it will be manifested as a peak state of small file storage. The small file OPS in the peak state is several times that of the rest of the day and dozens of times that of the rest of the night. In addition, the peak period does not last long, only about an hour. Obviously, in the massive small file storage scenario in the field of AI security, the peak state of small file storage lasts for a short time, but the OPS of small files is larger in the peak state.

[0056] On this basis, in order to realize the storage of massive small files in the field of AI security, the existing technology needs to determine the required cache hardware resources according to the OPS of the small file storage business during the peak period. Since the OPS of small files during the peak period is very large, more caches are often required to meet the low latency requirements of massive small file storage during the peak period. However, since the peak period of small file storage lasts for a short time, the OPS of small files is low most of the time. Therefore, most of the time in a day (about 91.67% of the time) there will be a problem of excess cache hardware resources.

[0057] Taking the above-mentioned SSD / HDD tiered hybrid storage architecture as an example, if the OPS is 6000 when the file storage volume is the largest in a day, and the storage capacity of the HDD is 3000 and the storage capacity of the SSD is 1000, then 6 SSDs are needed to meet the low latency requirements for storing massive small files. According to the above analysis, since the peak period is only about one hour, the OPS of small file storage is relatively small during other times. When the OPS is small, the hardware resources of the 6 SSDs cannot be fully utilized, resulting in a waste of resources.

[0058] Therefore, based on the above defects, an embodiment of the present application provides a method for storing massive small files. By calculating the growth factor, it is determined whether to enable the cache, thereby avoiding continuous writing in the cache and maximizing the use of cache hardware resources, thereby extending the service life of the cache while meeting the low latency requirements of file storage.

[0059] The following is a detailed description of some implementation methods of the present application in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Figure 3 , is a system architecture diagram of a file storage system, which includes a multi-node NFS (Network File System) service cluster 200, a data storage cluster 210, and a metadata management cluster 220.

[0060] Among them, the multi-node NFS service cluster 200 is used for NFS protocol access processing, storing the file content data of the file to be stored in the data storage cluster 210, and storing the metadata of the file to be stored in the metadata management cluster 220; the data storage cluster 210 is used to allocate and manage the storage medium space, and provide reliable and distributed data management capabilities through cross-node multi-copy or correction and deletion redundancy strategies, focusing on the realization of storage underlying capabilities such as reliability, high performance and scalability, and providing storage space access capabilities based on file semantics to the outside; the metadata management cluster 220 is used to manage and maintain basic metadata information such as directory tree hierarchical relationships and file permissions in the namespace, as well as value-added service information such as NFS user quotas, permissions and quality of service (Quality of Service, QoS), providing reliable and distributed metadata management capabilities through a multi-master and multi-standby mode, focusing on the realization of database underlying capabilities such as the four basic elements for the correct execution of database transactions (atomicity, consistency, isolation, durability, ACID), and providing metadata access capabilities to the outside.

[0061] Optionally, the multi-node NFS service cluster 200 may be a cluster consisting of multiple NFS service nodes, and the data storage cluster 210 may be a cluster consisting of multiple hard disk drives.

[0062] In a possible implementation, each NFS service node may be a different NFS server, the data storage cluster 210 may be mounted on a data storage server, and the metadata management cluster 220 may be mounted on a metadata management server.

[0063] Please refer to Figure 4 , is a block diagram of an NFS service node 300 provided in an embodiment of the present application. The NFS service node includes a memory 310, a processor 320, and a communication module 330. The memory 310, the processor 320, and the communication module 330 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0064] The memory 310 is used to store programs or data. The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0065] The processor 320 is used to read / write data or programs stored in the memory and execute corresponding functions.

[0066] The communication module 330 is used to establish a communication connection between the NFS service node and other communication terminals through the network, and to send and receive data through the network.

[0067] It should be understood that Figure 4 The structure shown is only a schematic diagram of the structure of the NFS service node. The NFS service node may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0068] Next, a method for storing a large number of small files provided by an embodiment of the present application is introduced by taking an NFS service node in the multi-node NFS service cluster as the execution subject. For details, see Figure 5 , is a flow chart of a method for storing a large number of small files provided in an embodiment of the present application, the method comprising:

[0069] Step S30, calculating the growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period;

[0070] The request queue includes write requests for files, and the growth factor represents the growth of the write requests in the request queue.

[0071] Optionally, an NFS protocol stack and a storage module may be set in the NFS service node. The NFS protocol stack is used to encapsulate the user request for the file to be written into a write request for the file when receiving a user request, and add it to the request queue to wait for storage; the storage module is used to store the request queue consisting of write requests for multiple files.

[0072] Optionally, after the write request for the file is added to the request queue, the first NFS service node may return a "write successful" response to the user.

[0073] In one possible implementation, after receiving a user request for a file to be written, the NFS service node may convert the content of the file into a data format, thereby obtaining the content data of the file to be written, and then encapsulate the content data of the file to be written and the attribute data of the file to be written into a write request for the file, and add the encapsulated write request to the request queue to wait for storage.

[0074] It can be understood that the request queue is a queue composed of write requests for each file to be written, and the number of write requests per second is the above-mentioned OPS. On this basis, the number of write requests in the request queue will fluctuate to a certain extent at different times. Therefore, the growth of write requests in the request queue can be known by calculating the growth factor of the request queue. In this embodiment, each write request added to the request queue can be understood as a business, and the growth of write requests in the request queue at each moment can feedback the growth of the file storage OPS, and then feedback the business characteristics of small file storage in the current time period.

[0075] In this embodiment, the growth factor of the request queue in the current time period can be calculated by the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period.

[0076] Optionally, the growth factor of the request queue can be calculated according to a preset time interval, and the growth factor calculation is performed once every preset time interval. If the current moment is the moment when the growth factor needs to be calculated, the above-mentioned current time period is a time period with the current moment as the end moment and the moment of the preset time interval from the current moment as the start moment; the time period before the current time period is a time period with the start moment of the current time period as the end moment and the moment of the preset time interval from the start moment of the current time period as the start moment. Optionally, the preset time interval can be set in advance and saved.

[0077] In an example, if the preset time interval is 1 second and the current time is 10:00, the current time period is 09:59 to 10:00; the time period before the current time period is 09:58 to 09:59.

[0078] Optionally, the number of write requests corresponding to the end time of the time period before the current time period of the request queue is the number of write requests accumulated in the request queue at the end of the time period before the current time period. It can be understood that in the above example, the number of write requests is the number of write requests in the request queue at 09:59.

[0079] In this embodiment, the number of write requests added to the request queue, the number of files stored in the data storage cluster, and the number of write requests in the request queue can be monitored in real time, so that the values ​​can be directly used for calculation when calculating the growth factor. Optionally, the growth factor of the request queue in the current time period can be calculated according to the following formula:

[0080] H=(Y+ZX) / X

[0081] Among them, H represents the growth factor of the request queue in the current time period, Y represents the number of write requests corresponding to the end time of the request queue before the current time period, Z represents the number of write requests added to the request queue in the current time period, and X represents the number of files stored in the data storage cluster in the time period before the current time period.

[0082] Step S31, determining whether to enable a cache in the NFS service node to store files in the request queue according to the growth factor.

[0083] Optionally, the NFS service node further includes a cache. If it is determined that the cache should be enabled at present according to the growth factor, the cache can store the files in the request queue. Optionally, the cache can be enabled by opening a special channel between a storage module storing write requests for multiple files and the cache, or by sending a storage instruction to the cache, which is not limited in the present application.

[0084] Optionally, since the calculated growth factor is the growth factor for the current time period, and the growth factor can feedback the business characteristics of small file storage in the current time period, after calculating the growth factor for the current time period, the business characteristics of small file storage in the next time period of the current time period can be predicted based on the growth factor for the current time period, thereby determining whether it is necessary to enable the cache in the NFS service node to store files in the request queue.

[0085] The method for storing a large number of small files provided in the embodiment of the present application calculates the growth factor of the request queue in the current time period according to the number of write requests added corresponding to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period, so as to determine whether to enable the cache in the NFS service node to store the files in the request queue according to the growth factor. Since the growth factor is calculated to determine whether to enable the cache, continuous writing in the cache can be avoided, so that the service life of the cache can be extended while meeting the low latency requirements of file storage.

[0086] Optionally, a threshold value may be set to determine whether to enable the cache. Specifically, Figure 5 On the basis of Figure 6 For another flowchart of the method for storing a large number of small files provided in the embodiment of the present application, see Figure 6 , the above step S31 can also be implemented by the following steps:

[0087] Step S31-1, if the growth factor is not less than the first threshold, the cache in the NFS service node is enabled, and the files in the request queue are stored through the cache and the data storage cluster;

[0088] Optionally, since the write requests in the request queue include file content data and file attribute data of the files to be written, the files in the request queue are stored through the cache and the data storage cluster, that is, the file content data in the write requests are stored in the cache and the data storage cluster.

[0089] It can be understood that the cache and data storage cluster can obtain the file content data of the file to be written from the write request in the request queue, and then store it.

[0090] In this embodiment, the business characteristics of small file storage can be divided, so as to determine how to store files in the request queue according to different business characteristics. Specifically, the first threshold can be set in advance according to the needs and saved. After the growth factor is calculated, the growth factor is compared with the first threshold. If the growth factor is not less than the first threshold, it means that the growth of write requests in the request queue indicates that the request queue is in a peak state at this time, that is, the business characteristics of small file storage in the current time period are reflected as a peak state. On this basis, it can be predicted that the business state of small file storage in the next time period of the current time period will also be a peak state. Please continue to refer to Figure 2 If the growth factor is not less than the first threshold, it means that the business characteristics of small file storage are reflected in its Figure 2 The portion above the mid-peak dividing line 20 is the peak state. At this time, there are many write requests in the request queue, so it is necessary to enable the cache to store the files in the request queue.

[0091] In this embodiment, if the growth factor is not less than the first threshold, the files in the request queue can be stored simultaneously in the cache and the data storage cluster, that is, the cache is used to divert the storage of files that should be stored in the data storage cluster.

[0092] Obviously, since in this case the cache and the data storage cluster can be used simultaneously to implement the storage of the write requests in the request queue, the number of caches to be set can be reduced.

[0093] In an example, if the OPS is 6000 when the file storage volume is the largest in a day, and the storage capacity of the data storage cluster is 3000, and the storage capacity of the cache is 1000, then only 3 caches are needed to meet the low latency requirements for storing massive small files. Compared with the solution of using 6 SSDs in the prior art, the solution provided in the embodiment of the present application can reduce the number of caches required under the same conditions.

[0094] In one possible implementation, the first threshold can be taken as 2. After being enabled, the cache can obtain and store the file content data in the write request from the request queue based on its own performance and storage capacity; the data storage cluster can also obtain and store the file content data in the write request from the request queue based on its own performance and storage capacity.

[0095] Optionally, when the cache or data storage cluster stores the content data in the write request, an aggregate storage method may be adopted, that is, each file is aggregated into a large file in a head-to-tail manner for storage.

[0096] In a possible implementation, the size of the large file in the data storage cluster is 1 GB, and the size of the large file in the buffer is 64 MB. It is understandable that if multiple files are stored in the data storage cluster as a large file through aggregate storage, the maximum size of the large file is 1 GB; if multiple files are stored in the buffer as a large file through aggregate storage, the maximum size of the large file is 64 MB.

[0097] Step S31 - 2 : if the growth factor is less than the first threshold, the files in the request queue are stored in the data storage cluster.

[0098] Optionally, since the write requests in the request queue include file content data and file attribute data of the files to be written, storing the files in the request queue through the data storage cluster means storing the file content data in the write requests in the data storage cluster.

[0099] Optionally, if the growth factor is less than the first threshold, it means that the request queue is in a non-peak state, that is, the business characteristics of small file storage are in a non-peak state. Therefore, the cache may not be enabled and the files in the request queue may be stored directly through the data storage cluster.

[0100] In this embodiment, the data storage cluster can obtain the write request from the request queue and store it according to its own performance and storage capacity.

[0101] The massive small file storage method provided in the embodiment of the present application sets a first threshold and determines whether the growth factor is greater than the first threshold, thereby determining whether the growth of write requests in the request queue indicates that the request queue is in a peak state, so that the cache is enabled only in the peak state, thereby storing the files in the request queue through the cache and the data storage cluster at the same time, and the cache is not enabled when the request queue is not in a peak period, and the files in the request queue are stored only through the data storage cluster. Therefore, it is possible to use the cache to divert and store files only during peak periods, thereby achieving maximum utilization of cache resources and meeting the low latency requirements for file storage.

[0102] Optionally, according to Figure 2It can be seen that when the business characteristics of small file storage are in the off-peak state, it shows a relatively slow growth trend in a certain period of time, and shows a continuous decline in the valley state at other times. On this basis, the off-peak state of small file storage can be refined into a stable state and a valley state. For details, please continue to refer to Figure 2 , the trough dividing line 21 can divide the non-peak state of small file storage into a stable state and a trough state. Among them, the part between the peak dividing line 20 and the trough dividing line 21 represents that the business state of small file storage is a stable state, and the part below the trough dividing line 21 represents that the business state of small file storage is a trough state. Obviously, when the business characteristics of small file storage are in a stable state, the growth of write requests in the request queue is relatively slow, and the OPS at different times is relatively consistent; when the business characteristics of the small file storage are in a trough state, the write requests in the request queue maintain a downward trend, and the OPS at certain times approaches 0.

[0103] In this embodiment, it can be understood that the service characteristics of small file storage can be specifically manifested in the following three states: peak state, stable state, and valley state. Considering the growth of write requests in the request queue, the state of the request queue in the stable state and valley state is different. Therefore, a second threshold can be set to judge the state of the request queue, so as to determine how to store files. Specifically, in Figure 6 On the basis of Figure 7 For another flowchart of the method for storing a large number of small files provided in the embodiment of the present application, please refer to Figure 7 , the above step S31-2 can also be implemented by the following steps:

[0104] Step S31-2-1, if the growth factor is not greater than the second threshold, reading out the file stored in the cache, and storing the read file in the data storage cluster;

[0105] Optionally, a second threshold can be set in advance and saved. If the growth factor is not greater than the second threshold, it means that the growth of write requests in the request queue indicates that the request queue is in a low period, that is, the business characteristics of small file storage in the current time period are in a low period.

[0106] In this embodiment, if the request queue is in a low-peak state, the file OPS at this time is relatively small, so the file content data previously stored in the buffer can be read out and saved in the data storage cluster.

[0107] Optionally, during peak hours, multiple files are aggregated and stored in the cache as large files. Therefore, if you want to read them out, you need to decompose the large files in the cache, read out the content data of each file, repackage the file content data into a write request corresponding to the file, and add it to the request queue.

[0108] Optionally, when the large files in multiple buffers are completely parsed, the buffer file deletion interface can be called to delete the large files in the buffer.

[0109] Since the data storage cluster can obtain the write request from the request queue and store the file content data in the write request according to its own performance and storage capacity, the files stored in the cache during the peak period can be stored back in the data storage cluster. It can be understood that storing multiple files in the cache is to reduce the storage pressure of the data storage cluster during the peak period. Storing multiple files in the cache during the peak period is not lifelong storage, but temporary storage. The files in the cache need to be stored back in the data storage cluster.

[0110] Step S31-2-2: if the growth factor is greater than the second threshold and less than the first threshold, the files in the request queue are stored through the data storage cluster.

[0111] Optionally, if the growth factor is greater than the second threshold and less than the first threshold, it means that the growth of write requests in the request queue indicates that the request queue is in a stable state. At this time, the growth of write requests in the request queue is relatively slow, and the file OPS is between peak and trough periods. Therefore, the files in the request queue can be directly stored through the data storage cluster.

[0112] It can be understood that the first threshold is greater than the second threshold. In a possible implementation, the second threshold may be 1.

[0113] The method for storing a large number of small files provided by the embodiment of the present application sets a second threshold value to determine whether the growth factor is not greater than the second threshold value, thereby determining whether the request queue is in a trough state or a stable state. Therefore, when the request queue is in a stable state, the file in the write request of the request queue can be stored through the data storage cluster, and when the request queue is in a trough state, the file in the buffer can be read out and the read file can be stored through the data storage cluster, thereby meeting the low latency requirement of file storage while re-saving the file in the buffer to the data storage cluster. In addition, the business state of small file storage can be predicted by calculation, so that the peak state, stable state and trough state of small file storage can be realized, and different storage methods can be intelligently selected for file storage, which not only meets the low latency requirement of file storage, but also makes the most of the hardware resources of the buffer, and at the same time greatly prolongs the service life of the buffer. Optionally, the content data of the file is stored in the buffer and the data storage cluster. Since for a certain file to be written, when the content data of the file is saved in the corresponding buffer and data storage cluster, the metadata of the file should also be saved. The metadata is data used to describe file attributes, including the file name, file size, and storage location of the file.

[0114] It can be understood that for a certain file to be written, the write request includes the file content data and file attribute data of the file. Then, after the file is saved, the metadata of the file can be generated according to the attribute data and storage location of the file, and the metadata of the file can be saved to the metadata management cluster. Specifically, Figure 5 On the basis of Figure 8 For another flowchart of the method for storing a large number of small files provided in the embodiment of the present application, please refer to Figure 8 , the method further comprises:

[0115] Step S32: for the target file stored in the request queue, generate an initial file identifier corresponding to the target file, and store the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster.

[0116] Optionally, the target file is a file whose file content data has been saved in a data storage cluster or in a cache, but whose metadata has not been saved; the file identifier represents properties such as the storage location of the file; the initial file identifier corresponding to the target file is the file identifier generated when the target file is first stored in the data storage cluster or in the cache.

[0117] Optionally, an initial file identifier of the target file may be generated while the target file is stored in the data storage cluster or in the cache, and the file name of the target file and the initial file identifier corresponding to the target file may be stored in the metadata management cluster.

[0118] It can be understood that, in this embodiment, the file name of the target file and the initial file identifier corresponding to the target file can together constitute the metadata of the target file.

[0119] In a possible implementation, the file name of the target file and the initial file identifier corresponding to the target file can be saved in the metadata management cluster in KV format. In an example, if the file name of the target file is " / nfs / 1.jpg" and the initial file identifier corresponding to the target file is "1152922605193266177-0-5000", the file name and the initial file identifier can form a KV key-value pair "< / nfs / 1.jpg,1152922605193266177-0-5000> " is saved to the metadata management cluster.

[0120] In another possible implementation, the file name of the target file and the initial file identifier corresponding to the target file may be saved respectively. Optionally, considering that the file temporarily stored in the cache of the NFS service node is likely to be lost in an emergency, the file stored in the cache may be backed up to prevent the file from being lost or damaged. Specifically, the method for storing a large number of small files provided in the embodiment of the present application also includes:

[0121] After storing the target file in the request queue through the cache and storing the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster, a backup NFS service node is selected from the multi-node NFS service cluster, and a target file backup request is sent to the backup NFS service node, so that the backup NFS service node stores the target file according to the target file backup request and generates a backup file identifier corresponding to the target file, generates a file identifier corresponding to the target file according to the initial file identifier and the backup file identifier corresponding to the target file in the metadata management cluster, and updates the initial file identifier corresponding to the target file in the metadata management cluster according to the file identifier.

[0122] Optionally, the target file is a file whose file content data has been stored in a cache, whose metadata has been stored in a metadata management cluster, but which has not been backed up.

[0123] In this embodiment, when the target file is stored in the cache and the metadata corresponding to the target file is stored in the metadata management cluster, the cache can select a backup NFS service node from the multi-node NFS service cluster and establish a master-slave relationship with it.

[0124] If the NFS service node storing the target file is used as the primary NFS service node storing the target file, it can be understood that the backup NFS service node is any other NFS service node in the multi-node NFS service cluster except the primary NFS service node.

[0125] In this embodiment, the cache can send a target file backup request to the backup NFS service node. When the cache in the backup NFS service node receives the target file backup request, it can store the file content data of the target file according to the target file backup request, and at the same time generate a backup file identifier corresponding to the target file stored in the cache in the backup NFS service node.

[0126] Optionally, the cache in the backup NFS service node can also obtain the initial file identifier corresponding to the target file from the metadata management cluster according to the file name of the target file, and generate a file identifier corresponding to the target file based on the initial file identifier and the backup file identifier, and then update the initial file identifier corresponding to the target file stored in the metadata management cluster according to the file identifier corresponding to the target file, thereby saving the file identifier corresponding to the target file in the metadata management cluster.

[0127] It can be understood that the file identifier corresponding to the target file can represent the storage path of the target file in the primary NFS service node and the storage path of the target file in the backup NFS service node.

[0128] Optionally, the file identifier corresponding to the target file can be obtained by bitwise ORing the first 8 bits of the initial file identifier and the backup file identifier corresponding to the target file. In one example, if the initial file identifier of the target file " / nfs / 2.jpg" is "1125900981633024-0-C800", and the backup file identifier of the target file is "2199023257602-0-C800", then the file identifier corresponding to the target file obtained by bitwise ORing the first 8 bits of the two is "1128100004890626-0-C800". At this time, the KV key-value pair consisting of the file name of the target file and the initial file identifier stored in the metadata management cluster is "< / nfs / 2.jpg,1125900981633024-0-C800> ", which can be updated to a KV key-value pair consisting of the file name of the target file and the file identifier corresponding to the target file"< / nfs / 2.jpg,1128100004890626-0-C800> ”.

[0129] Optionally, after the two NFS service nodes establish a master-slave relationship, all files stored in the cache of the master NFS service node can be backed up to the cache of the backup NFS service node and stored in the backup NFS service node in the form of large files through aggregate storage.

[0130] Obviously, the caches in the two NFS service nodes with a primary and backup relationship store large files with the same data content. Therefore, if the large file in the primary NFS service node is deleted after the analysis is complete, the corresponding large file in the backup node also needs to be deleted.

[0131] Optionally, after the cache in the primary NFS service node deletes a large file, it may send a file deletion request to the cache in the backup NFS service node having a primary-backup relationship with it. After receiving the file deletion request, the cache in the backup NFS service node may delete the corresponding large file stored therein.

[0132] Optionally, since the files in the cache need to be read out and stored in the data storage cluster when the request queue is in a low-peak state, after the read files are stored again in the data storage cluster, the file identifier in the metadata management cluster needs to be updated. Specifically, the method further includes:

[0133] After reading out the file stored in the cache and storing the read file in the data storage cluster, a new file identifier of the file is generated, and the file identifier corresponding to the file in the metadata management cluster is updated according to the new file identifier.

[0134] Optionally, the new file identifier is an identifier generated for a file that has been previously stored in the cache and for which an initial file identifier has been generated, when the file is read from the cache and stored again in the data storage cluster.

[0135] In this embodiment, after reading the file stored in the cache, the read file content data and the attribute data of the file will be converted into a write request for the file and added to the request queue, and the data storage cluster will store the file content data in the write request in the request queue. If the data storage cluster stores the file read from the cache, a new file identifier corresponding to the file will be generated.

[0136] Since the file identifier corresponding to the file has been saved in the metadata when the file is stored in the cache, after a new file identifier corresponding to the file is generated, the file identifier corresponding to the file stored in the metadata management cluster can be updated according to the new file identifier corresponding to the file.

[0137] Optionally, after the file is stored in the cache or data storage cluster, the file identifier of the generated file occupies 16 bytes, the first 8 bytes represent the global large file ID, the middle 4 bytes represent the offset in the large file, and the last 4 bytes represent the size of the small file. In the 64-bit global large file ID, the upper 4 bits represent the storage location of the large file. If the value is 0, it means that the large file is stored in the cache, and if the value is 1, it means that the large file has been stored in the data storage cluster.

[0138] For details, please refer to Fig. 9A , is a schematic diagram of aggregate storage in which the files to be written are connected end to end and stored in the form of a large file. Taking four files as an example, namely the first file " / pic / 1.jpg", the second file " / pic / 2.jpg", the third file " / pic / 3.jpg", and the fourth file " / pic / 4.jpg", the large file formed by connecting the four files end to end and the file identifiers corresponding to the four files are as follows: Fig. 9A As shown; please refer to Fig. 9B , which is a schematic diagram of the global large file ID format.

[0139] Taking a three-node NFS service cluster as an example, the following describes a method for storing a large number of small files provided by an embodiment of the present application when the request queue is in different states.

[0140] Please refer to Fig. 10A, is an architecture diagram of a three-node NFS service cluster 400 , including a first NFS service node 410 , a second NFS service node 420 , a third NFS service node 430 , a data storage cluster 210 , and a metadata management cluster 220 .

[0141] Take the first NFS service node as an example, please refer to Fig. 10B , is a write logic architecture diagram of a file to be written in a first NFS service node, and the first NFS service node includes a storage module 412 and a cache 411.

[0142] Among them, the storage module is used to receive file write requests and add them to the storage request queue, and return a "write successful" response to the client; the cache 411 is used to be enabled when the request queue is at a peak period, so as to store files together with the data storage cluster.

[0143] Optionally, the storage module 412 may be Figure 4 The memory in the buffer, such as a random access memory (RAM), may also be other devices capable of storing write requests for files, and the buffer 411 may be a solid state drive (Nvem SSD).

[0144] Optionally, the data storage cluster can store files in the request queue through a normal channel. When it is determined to enable the cache, one of the normal channels between the storage module and the data storage cluster can be upgraded to a special channel between the storage module and the cache, so that the cache can store files in the request queue through the special channel.

[0145] In this embodiment, the data storage cluster may be a data storage cluster composed of HDDs.

[0146] If a user initiates a user request to write a file " / nfs / 1.jpg" through a user terminal, the first NFS service node can convert the file content of the file " / nfs / 1.jpg" to be written into a data format after receiving the user request, and encapsulate the file content data and the file attribute data, such as the file name and file size, into a write request for the file, and add it to the file request queue in the storage module, and return a "write successful" response to the client.

[0147] The NFS service node calculates a growth factor of the request queue at preset intervals, and calculates a magnitude relationship between the growth factor and a first threshold and a second threshold.

[0148] In one possible implementation, the growth factor is less than the first threshold and greater than the second threshold (ie, the business characteristics of small file storage are in a stable state). In this case, the data storage cluster can store the files in the request queue through a normal channel.

[0149] If the file to be written " / nfs / 1.jpg" is stored through the data storage cluster at this time, the initial file identifier "1152922605193266177-0-5000" of the file can be generated when it is aggregated and stored in / 1 / 1 / 1 / 1 / 1 in the data storage cluster.

[0150] The first NFS service node can generate a KV key-value pair " / nfs / 1.jpg" consisting of the file name and the initial file identifier "1152922605193266177-0-5000" of the file.< / nfs / 1.jpg,1152922605193266177-0-5000> " is stored in the metadata management cluster, completing the storage of the file " / nfs / 1.jpg" to be written.

[0151] In another possible implementation, the growth factor is not less than the first threshold. In this case, the ordinary channel can be upgraded to a special channel to enable the cache, and the files in the request queue are stored through the cache and the data storage cluster.

[0152] Since both the cache and the data storage cluster are used to store the files in the request queue at this time, if the file " / nfs / 1.jpg" to be written is stored through the data storage cluster at this time, its storage process can refer to the storage process when the request queue is in the stable state.

[0153] If the file to be written " / nfs / 1.jpg" is stored in the large file / 1 / 1 in the cache in an aggregated form, the initial file identifier "1125900981633024-0-C800" of the file is generated, and the first NFS service node generates a KV key-value pair "1125900981633024-0-C800" consisting of the file name " / nfs / 1.jpg" and the initial file identifier "1125900981633024-0-C800" of the file.< / nfs / 1.jpg,1125900981633024-0-C800> "Save to the metadata management cluster.

[0154] At the same time, the cache in the first NFS service node can select any NFS service node except the first node in the three-node NFS service cluster as a backup NFS service node.

[0155] Taking the backup NFS service node selected by the cache in the first NFS service node as the second NFS service node as an example, the first NFS service node can establish a master-backup relationship with the second NFS service node, and the cache in the first NFS service node can send a file backup request for the file " / nfs / 1.jpg" to the cache in the second NFS service node.

[0156] When the cache in the second NFS service node receives the file backup request, the file content data of the file " / nfs / 1.jpg" can be stored in the cache in the second NFS service node, and the backup file identifier "2199023257602-0-C800" of the file can be generated. At the same time, the cache in the second NFS service node queries the metadata management cluster for the initial file identifier corresponding to the file according to the file name " / nfs / 1.jpg", thereby performing bitwise OR processing on the first 8 bits of the initial file identifier and the backup file identifier of the file to obtain the file identifier "1128100004890626-0-C800" of the file.

[0157] Afterwards, the KV key-value pair "< / nfs / 1.jpg,1128100004890626-0-C800> "Stored in the metadata management cluster"< / nfs / 1.jpg,1125900981633024-0-C800> ” to update.

[0158] In yet another possible implementation, the growth factor is not greater than the second threshold. In this case, the files in the cache in the first NFS service node may be read from the large file and stored again in the data storage cluster.

[0159] The large files stored in the cache are parsed to obtain the file content data of multiple files. If the read file is the file content data of the above-mentioned file " / nfs / 1.jpg", the file content data of the file and the file name, file size and other attribute data of the file are encapsulated and converted into a write request for the file " / nfs / 1.jpg", and the write request is added to the request queue in the storage module.

[0160] Determine whether all large files in the cache have been parsed. If the large file has been parsed, call the cache deletion interface to delete the parsed large file. At the same time, the cache can send a file deletion request to the cache of the backup NFS node. In this embodiment, since the second NFS service node is the backup NFS service node of the first NFS service node, the cache in the first NFS service node can send a file deletion request to the cache in the second NFS service node. After receiving the file deletion request, the cache in the second NFS service node can delete the backup large file corresponding to the large file deleted by the cache in the first NFS service node.

[0161] Since the data storage cluster can store the file content data in the write request in the request queue through the common channel between the storage module, after the data storage cluster stores the file content data of the file " / nfs / 1.jpg", a new file identifier "1154922605193266177-0-5000" can be generated for the file " / nfs / 1.jpg".

[0162] When the file " / nfs / 1.jpg" is stored in the cache, the metadata management cluster stores a KV key-value pair consisting of the file name and the file identifier.< / nfs / 1.jpg,1128100004890626-0-C800> Therefore, the KV key-value pair " can be formed based on the file name " / nfs / 1.jpg" of the file and the new file identifier "1154922605193266177-0-5000" corresponding to the file.< / nfs / 1.jpg,1154922605193266177-0-5000> "The KV key-value pair stored in the metadata management cluster is updated to complete the storage of the file " / nfs / 1.jpg".

[0163] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a massive small file storage device is given below. Fig.11 , Fig.11 A functional module diagram of a massive small file storage device provided in an embodiment of the present application. It should be noted that the basic principle and technical effects of the massive small file storage device provided in this embodiment are the same as those of the above embodiments. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiments. The massive small file storage device includes: a calculation prediction module 500 and a cache module 510.

[0164] The calculation prediction module 500 is used to calculate the growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue;

[0165] It can be understood that the calculation prediction module 500 can execute the above step S30.

[0166] The cache module 510 is used to determine whether to enable the cache in the NFS service node to store the files in the request queue according to the growth factor.

[0167] It can be understood that the cache module 510 can execute the above step S31.

[0168] Optionally, the cache module 510 is also used to enable the cache in the NFS service node if the growth factor is not less than the first threshold, and store the files in the request queue through the cache and the data storage cluster; if the growth factor is less than the first threshold, store the files in the request queue through the data storage cluster.

[0169] It is understandable that the cache module 510 can also execute the above steps S31 - 1 to S31 - 2.

[0170] Optionally, the cache module 510 is also used to read out the files stored in the cache and store the read files through the data storage cluster if the growth factor is not greater than the second threshold; if the growth factor is greater than the second threshold and less than the first threshold, the files in the request queue are stored through the data storage cluster.

[0171] It is understandable that the cache module 510 can also execute the above steps S31-2-1 to S31-2-2.

[0172] Optionally, the cache module 510 is further configured to generate an initial file identifier corresponding to the target file stored in the request queue, and store the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster.

[0173] It is understandable that the cache module 510 can also execute the above step S32.

[0174] Optionally, the cache module 510 is also used to store the target file in the request queue through the cache, and after storing the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster, select a backup NFS service node from the multi-node NFS service cluster, and send a target file backup request to the backup NFS service node, so that the backup NFS service node stores the target file according to the target file backup request, and generates a backup file identifier corresponding to the target file, generates a file identifier corresponding to the target file according to the initial file identifier and the backup file identifier corresponding to the target file in the metadata management cluster, and updates the initial file identifier corresponding to the target file in the metadata management cluster according to the file identifier.

[0175] Optionally, the cache module 510 is also used to generate a new file identifier for the file after reading the file stored in the cache and storing the read file through the data storage cluster, and update the file identifier corresponding to the file in the metadata management cluster according to the new file identifier.

[0176] Optionally, the calculation prediction module 500 is further configured to calculate a queue growth factor within a preset time interval from the current moment according to the following formula:

[0177] H=(Y+ZX) / X

[0178] Among them, H represents the growth factor of the request queue in the current time period, Y represents the number of write requests corresponding to the end time of the request queue before the current time period, Z represents the number of write requests added to the request queue in the current time period, and X represents the number of files stored in the data storage cluster in the time period before the current time period.

[0179] The massive small file storage device provided by the embodiment of the present application calculates the growth factor of the request queue in the current time period according to the number of write requests added corresponding to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period through a calculation prediction module; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue; the cache module determines whether to enable the cache in the NFS service node to store the files in the request queue according to the growth factor. Therefore, by calculating the growth factor to determine whether to enable the cache, it is possible to avoid continuous writing in the cache, thereby extending the service life of the cache while meeting the low latency requirements for massive small file storage.

[0180] Optionally, the above modules can be stored in the form of software or firmware. Figure 4The memory shown in FIG. 1 is stored in the operating system (OS) of the NFS service node and can be used by Figure 4 Meanwhile, the data and program codes required for executing the above modules can be stored in the memory.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0183] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for storing a large number of small files, characterized in that: Applied to an NFS service node in a multi-node NFS service cluster, the method comprises: Calculate the growth factor of the request queue in the current time period according to the number of write requests added to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period; The calculating the growth factor of the request queue in the current time period according to the number of write requests added in the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period includes: The growth factor of the request queue in the current time period is calculated according to the following formula: H=(Y+ZX) / X Wherein, H represents the growth factor of the request queue in the current time period, Y represents the number of write requests corresponding to the request queue at the end time of the time period before the current time period, Z represents the number of write requests added to the request queue in the current time period, and X represents the number of files stored in the data storage cluster in the time period before the current time period; Wherein, the request queue includes write requests for files, and the growth factor represents the growth of the write requests in the request queue; Determine whether to enable a cache in the NFS service node to store files in the request queue according to the growth factor.

2. The method according to claim 1, characterized in that The determining, according to the growth factor, whether to enable the cache in the NFS service node to store the files in the request queue includes: If the growth factor is not less than a first threshold, enabling a cache in the NFS service node, and storing the files in the request queue through the cache and the data storage cluster; If the growth factor is less than the first threshold, the files in the request queue are stored by the data storage cluster.

3. The method according to claim 2, characterized in that If the growth factor is less than the first threshold, storing the files in the request queue by using the data storage cluster includes: If the growth factor is not greater than a second threshold, reading out the file stored in the buffer, and storing the read file in the data storage cluster; If the growth factor is greater than the second threshold and less than the first threshold, the files in the request queue are stored by the data storage cluster.

4. The method according to claim 2, characterized in that: The method further comprises: For the target file stored in the request queue, an initial file identifier corresponding to the target file is generated, and the file name of the target file and the initial file identifier corresponding to the target file are stored in the metadata management cluster.

5. The method according to claim 4, characterized in that The method further comprises: After storing the target file in the request queue through the cache and storing the file name of the target file and the initial file identifier corresponding to the target file in the metadata management cluster, a backup NFS service node is selected from the multi-node NFS service cluster, and a target file backup request is sent to the backup NFS service node, so that the backup NFS service node stores the target file according to the target file backup request and generates a backup file identifier corresponding to the target file, generates a file identifier corresponding to the target file according to the initial file identifier and the backup file identifier corresponding to the target file in the metadata management cluster, and updates the initial file identifier corresponding to the target file in the metadata management cluster according to the file identifier.

6. The method according to claim 5, characterized in that The method further comprises: After reading out the file stored in the cache and storing the read file through the data storage cluster, a new file identifier of the file is generated, and the file identifier corresponding to the file in the metadata management cluster is updated according to the new file identifier.

7. A device for storing a large number of small files, characterized in that: Applied to an NFS service node in a multi-node NFS service cluster, the device comprises: A calculation prediction module, used to calculate the growth factor of the request queue in the current time period according to the number of write requests added corresponding to the request queue in the current time period, the number of files stored in the data storage cluster in the time period before the current time period, and the number of write requests corresponding to the request queue at the end time of the time period before the current time period; the request queue includes write requests for files, and the growth factor represents the growth of write requests in the request queue; The calculation prediction module is further used to calculate the growth factor of the request queue in the current time period according to the following formula: H=(Y+ZX) / X Wherein, H represents the growth factor of the request queue in the current time period, Y represents the number of write requests corresponding to the request queue at the end time of the time period before the current time period, Z represents the number of write requests added to the request queue in the current time period, and X represents the number of files stored in the data storage cluster in the time period before the current time period; The cache module is used to determine whether to enable the cache in the NFS service node to store the files in the request queue according to the growth factor.

8. An NFS service node, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement any one of the methods described in claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images