File caching method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a file caching method and device, a storage medium and electronic equipment, and relates to the field of servers, a file storage system is used for reading and writing a file and caching the read file to a built-in caching space, and the storage medium is used for storing the read file. The method comprises the following steps: determining a cache quota configured for the cache space by the file storage system based on the data volume of files needing to be read and written by the file storage system; the received file read-write request is analyzed, the file attribute of a target file corresponding to the file read-write request is obtained, and the target file is a constant code stream file; and determining a caching mode of the target file according to the file attribute and the caching quota. The problem that in the related technology, in the read-write process of the constant code stream file, due to the fact that the performance of a storage system is insufficient, read-write is unstable is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the server field, and specifically, to a file caching method, device, storage medium and electronic device. Background Art

[0002] In the field of video surveillance and broadcasting media, files are required to be read and written in the form of a constant bitrate. Any lag caused by insufficient storage performance will cause frame loss, which will seriously affect the quality of video and media. Traditional distributed storage systems mainly rely on general cache management strategies when facing intensive reading and writing of constant bitrate files. However, when processing multiple constant bitrate files at the same time, these strategies often cannot effectively guarantee the reading and writing stability of each file and the fair allocation of cache resources. As a result, during the reading and writing of constant bitrate files, the problem of unstable reading and writing occurs due to insufficient storage system performance.

[0003] In the related art, during the reading and writing process of constant bitstream files, due to insufficient storage system performance, the reading and writing is unstable, and no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a file caching method, device, storage medium and electronic device to at least solve the problem in the related art that, during the reading and writing process of a constant bitstream file, the reading and writing is unstable due to insufficient storage system performance.

[0005] According to one embodiment of the present application, a file caching method is provided, which is applied to a file storage system, wherein the file storage system is used to read and write files, and cache the read files into a built-in cache space, the method comprising: determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; parsing a received file read and write request to obtain file attributes of a target file corresponding to the file read and write request, wherein the target file is a constant bitstream file; and determining a caching method for the target file based on the file attributes and the cache quota.

[0006] In an exemplary embodiment, the file attribute includes a file identifier, the cache quota includes a first cache quota and a second cache quota, the first cache quota is less than the second cache quota, and the cache method for the target file is determined according to the file attribute and the cache quota, including: determining whether the target file has been cached in the cache space according to the file identifier; if it is determined that the target file is not cached in the cache space, determining a first occupied capacity of the cache space; comparing the first occupied capacity with the first cache quota to obtain a first comparison result; if it is determined that the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file to the cache space; if it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read and write request corresponding to the target file to a waiting queue to wait for the target file to be cached to the cache space; if it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, rejecting the file read and write request corresponding to the target file.

[0007] In an exemplary embodiment, the file read and write request corresponding to the target file is added to a waiting queue to wait for the target file to be cached to the cache space, including: detecting the cache space according to a preset period, and determining the second occupied capacity of the cache space when it is determined that the cache space has released files in the previous period; comparing the second occupied capacity with the first cache quota to obtain a second comparison result; and caching the target file to the cache space when it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota.

[0008] In an exemplary embodiment, when it is determined that a preset condition is met, the cache file in the cache space is released, and the preset condition includes at least one of the following: the cache time of the cache file exceeds a first preset time; the cache file has not been accessed for more than a second preset time; the cache file has the lowest access frequency within a preset time period.

[0009] In an exemplary embodiment, the file attributes also include a file size, and the file read and write request corresponding to the target file is added to a waiting queue to wait for the target file to be cached to the cache space, including: when it is determined that the file read and write request is a write request, obtaining the stripe size of the file storage system; comparing the file size of the target file with the stripe size to obtain a third comparison result; when it is determined that the third comparison result indicates that the file size of the target file is smaller than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, adding the file read and write request to a priority queue, and the priority of the priority queue is higher than the priority of the waiting queue.

[0010] In an exemplary embodiment, the method further includes: when determining that the file read and write request is a read request, reading the target file from the file storage system, reading data of a preset address length starting from the end storage address of the target file, and obtaining pre-read data; caching the target file and the pre-read data into the cache space.

[0011] In an exemplary embodiment, after caching the target file and the pre-read data into the cache space, the method further includes: responding to a release instruction for releasing the cache space, detecting whether the cache time of the pre-read data exceeds a third preset time; and releasing the pre-read data if it is determined that the cache time of the pre-read data exceeds the third preset time.

[0012] According to another embodiment of the present application, a file caching device is provided, including: a first determination module, used to determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; an attribute acquisition module, used to parse the received file read and write requests, and obtain the file attributes of the target file corresponding to the file read and write requests, wherein the target file is a constant code stream file; and a second determination module, used to determine the caching method for the target file based on the file attributes and the cache quota.

[0013] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0014] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0015] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0016] Through this application, the cache quota configured by the file storage system for the cache space is determined according to the amount of file data that the file storage system needs to read and write; the received file read and write request is parsed to obtain the file attributes of the target file corresponding to the file read and write request, wherein the target file is a constant bitstream file; and the cache method for the target file is determined according to the file attributes and the cache quota. This can solve the problem of unstable reading and writing due to insufficient storage system performance in the reading and writing process of constant bitstream files in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a hardware structure block diagram of a server device of a file caching method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of a file caching method according to an embodiment of the present application;

[0019] Figure 3 is a control flow chart of a file caching method according to an embodiment of the present application;

[0020] Figure 4 is a quota update flow chart of a file caching method according to an embodiment of the present application;

[0021] Figure 5 is a flowchart of a write request submission method according to an embodiment of the present application;

[0022] Figure 6 is a pre-read data protection flow chart of a file caching method according to an embodiment of the present application;

[0023] Figure 7 It is a structural block diagram of a file caching device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 11 is a hardware structure block diagram of a server device of a file caching method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the file caching method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method for implementing the above-mentioned file caching. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a file caching method is provided, which is applied to a file storage system. The file storage system is used to read and write files, and cache the read files into a built-in cache space. Figure 2 is a flowchart of a file caching method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0030] Step S202, determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0031] Step S204, parsing the received file read / write request to obtain the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0032] Step S206: determining a cache mode for the target file according to the file attribute and the cache quota.

[0033] Through the above steps, the cache quota configured by the file storage system for the cache space is determined based on the amount of file data that the file storage system needs to read and write; the received file read and write request is parsed to obtain the file attributes of the target file corresponding to the file read and write request, and the target file is a constant code stream file; the cache method for the target file is determined according to the file attributes and the cache quota, thereby solving the problem of unstable reading and writing due to insufficient storage system performance during the reading and writing process of the constant code stream file in the related technology.

[0034] In an exemplary embodiment, the file attribute includes a file identifier, the cache quota includes a first cache quota and a second cache quota, the first cache quota is less than the second cache quota, and the cache method for the target file is determined according to the file attribute and the cache quota, including: determining whether the target file has been cached in the cache space according to the file identifier; if it is determined that the target file is not cached in the cache space, determining a first occupied capacity of the cache space; comparing the first occupied capacity with the first cache quota to obtain a first comparison result; if it is determined that the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file to the cache space; if it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read and write request corresponding to the target file to a waiting queue to wait for the target file to be cached to the cache space; if it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, rejecting the file read and write request corresponding to the target file.

[0035] Optionally, assuming that the total size of the cache space in the distributed storage system is 10GB, the first cache quota is set to 5GB, and the second cache quota is set to 8GB. The system currently occupies 4GB of cache space. When a read or write request for a new file (for example, a surveillance video file) arrives, the system first checks whether the file already exists in the cache. If not, it continues to check the current occupied capacity of 4GB, which is less than the first cache quota of 5GB. The system will directly load part or all of the data of the file into the cache to speed up its read and write operations.

[0036] Assume that the cache space usage increases to 6GB, and there are new file read and write requests. After checking, the distributed storage system finds that the current occupied capacity is 6GB, which is equal to or greater than the first cache quota of 5GB, but less than the second cache quota of 8GB, so the new file read and write requests are placed in the waiting queue. These requests will be added to the cache in priority order (such as the most recent or most frequently used files first) after the cache files in the cache space are released, instead of being blocked or rejected immediately. If the cache space usage reaches 9GB, and there are new file read and write requests, the system will check and confirm that the occupied capacity is 9GB, which is equal to or greater than the second cache quota of 8GB. At this time, the system will directly reject the file's read and write requests to avoid cache overload and ensure that the read and write performance of cached files is not affected.

[0037] In this embodiment, the file cache mode is dynamically adjusted by setting the file identifier, the first cache quota and the second cache quota. When processing constant bitstream files in a distributed storage system, cache resources can be dynamically managed, which not only ensures efficient use of the cache, but also avoids waste and overload of cache resources, thereby ensuring the stability and response speed of the constant bitstream file reading and writing service.

[0038] In an exemplary embodiment, the file read and write request corresponding to the target file is added to a waiting queue to wait for the target file to be cached to the cache space, including: detecting the cache space according to a preset period, and determining the second occupied capacity of the cache space when it is determined that the cache space has released files in the previous period; comparing the second occupied capacity with the first cache quota to obtain a second comparison result; and caching the target file to the cache space when it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota.

[0039] Optionally, assume that the total cache space size of a distributed storage system is 10GB, where the first cache quota is set to 5GB. Currently, 5.8GB of cache space is occupied. At this time, the distributed storage system receives a read / write request for a new file (i.e., a target file), but because the distributed storage system detects that the currently occupied capacity is greater than the first cache quota, the request is placed in a waiting queue.

[0040] In the subsequent preset cycle detection, the distributed storage system found that some file data in the previous cycle was eliminated because it had not been accessed for a long time, and released 1.2GB of cache space, reducing the occupied capacity to 4.6GB. At this time, the distributed storage system will recalculate the second occupied capacity and compare it with the first cache quota. The comparison result shows that the second occupied capacity is less than the first cache quota (4.6GB<5GB), so the distributed storage system will take out the target file read and write requests that were previously placed in the waiting queue, and cache the target file data in the cache space for subsequent read and write operations.

[0041] In this embodiment, when the cache space approaches the first cache quota limit, the usage of the cache space is periodically detected, and the cache strategy is dynamically adjusted based on the detection results to ensure that the cache can be reasonably utilized even when cache resources are tight, thereby improving the system's read and write performance and response speed.

[0042] In an exemplary embodiment, when it is determined that a preset condition is met, the cache file in the cache space is released, and the preset condition includes at least one of the following: the cache time of the cache file exceeds a first preset time; the cache file has not been accessed for more than a second preset time; the cache file has the lowest access frequency within a preset time period.

[0043] Optionally, assume that there is a cache space of a distributed storage system, and the management strategy of the cache space includes the above-mentioned cache file release conditions. The cache space currently stores data of three files A, B, and C, which represent different video stream caches respectively. The first preset time set by the distributed storage system is 30 minutes, the second preset time is 10 minutes, and the access frequency is counted once an hour. Assume that file A has been continuously accessed since it was loaded into the cache space, and the access frequency is high. The last access to file B occurred 9 minutes ago. Although it did not reach the second preset time, the access frequency was lower than that of file A. File C has been in the cache space for 31 minutes since it was loaded into the cache space, and has not been accessed during this period.

[0044] In the next evaluation cycle of cache management (assuming it is 30 minutes later), when the distributed storage system checks whether the residence time of all cached files exceeds the first preset time (30 minutes), since the cache time of file C has exceeded 30 minutes and has not been accessed, file C will be released according to the first condition. When the distributed storage system checks the access status of files, although file B has not reached the second preset time, the system still needs to count the access frequency in the past hour and finds that file B has the lowest access frequency. According to the third condition, file B may also be released in the next hourly access frequency statistics to make room for more frequently accessed files. File A will not be released because it is continuously accessed at a high frequency and has not exceeded any preset time. It will continue to remain in the cache space to provide a fast response for subsequent read and write requests.

[0045] In this embodiment, a cache file release mechanism is described, which aims to dynamically manage cache space by setting specific conditions to ensure efficient use of cache resources while avoiding resource waste or cache overload. This cache file release mechanism is an important part of the cache management strategy. It monitors the life cycle and access mode of cache files and automatically releases files that are no longer active or have exceeded a specific time limit to make room for the cache of new files.

[0046] In an exemplary embodiment, the file attributes also include a file size, and the file read and write request corresponding to the target file is added to a waiting queue to wait for the target file to be cached to the cache space, including: when it is determined that the file read and write request is a write request, obtaining the stripe size of the file storage system; comparing the file size of the target file with the stripe size to obtain a third comparison result; when it is determined that the third comparison result indicates that the file size of the target file is smaller than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, adding the file read and write request to a priority queue, and the priority of the priority queue is higher than the priority of the waiting queue.

[0047] Optionally, assume that the stripe size of a distributed storage system is set to 1MB, the preset ratio is 0.8 (i.e., the file size is at least 80% of the stripe size), the current cache space is close to the first cache quota, and all write requests are temporarily placed in the waiting queue. At this time, the distributed storage system receives three write requests: the size of file X is 500KB, which is obviously smaller than the stripe size, but the ratio of its size to the stripe size is 0.5, which is less than the preset ratio. The size of file Y is 900KB, which is smaller than the stripe size, and the ratio of its size to the stripe size is 0.9, which exceeds the preset ratio. The size of file Z is 1.2MB, which is larger than the stripe size. At this time, according to the above strategy, file Y will be added to the priority queue first, because file Y is closer to the complete stripe size, and the distributed storage system is more efficient when writing files close to the complete stripe size, so it can improve data processing efficiency and accelerate the request processing process.

[0048] It should be noted that stripe size is an important concept in distributed storage systems. It means that when writing data to multiple storage devices, the data will be divided into blocks of equal size, and each block will be written to a different device. The choice of stripe size has a significant impact on the performance of distributed storage systems. A larger stripe size can reduce metadata overhead and increase writing speed, but increase the uneven distribution of data; a smaller stripe size can increase the dispersion of data, but will lead to higher metadata management costs.

[0049] In this embodiment, the distributed storage system can intelligently distinguish different write requests and adopt a priority processing strategy for the write of small files, especially files that are close to the stripe size and have high utilization, so as to improve the system's write efficiency and response speed, while avoiding delays or denial of service caused by insufficient cache space.

[0050] In an exemplary embodiment, the method further includes: when determining that the file read and write request is a read request, reading the target file from the file storage system, reading data of a preset address length starting from the end storage address of the target file, and obtaining pre-read data; caching the target file and the pre-read data into the cache space.

[0051] Optionally, assume that there is a distributed storage system with sufficient cache space, file A is a video file with a size of 20MB, and the current read request is to read the last 5MB of file A. The pre-read address length preset by the distributed storage system is 1MB, and the storage of file A is divided and distributed based on a stripe size of 1MB. According to the pre-read strategy, while reading the last 5MB of data of file A, the distributed storage system will also read the subsequent 1MB of data from the termination storage address of file A as pre-read data. The pre-read data is loaded into the cache together with the data in the read request to provide a quick response for subsequent possible read operations. If the subsequent read request happens to point to the address range covered by the pre-read data, that is, the last 3MB of file A (2MB read request + 1MB pre-read data), then the read operation can be completed directly from the cache without accessing the storage system again, thereby greatly improving the reading efficiency and user experience.

[0052] In this embodiment, through this pre-reading strategy, the distributed storage system can effectively predict and optimize read operations, especially when processing constant bitstream files in the field of video surveillance or broadcasting and television media resources, which can significantly reduce read delays and improve cache hit rates, thereby improving the read performance and response speed of the entire distributed storage system.

[0053] In an exemplary embodiment, after caching the target file and the pre-read data into the cache space, the method further includes: responding to a release instruction for releasing the cache space, detecting whether the cache time of the pre-read data exceeds a third preset time; and releasing the pre-read data if it is determined that the cache time of the pre-read data exceeds the third preset time.

[0054] Optionally, assuming that the third preset time set in the distributed storage system is 5 seconds, this means that any pre-read data that stays in the cache for more than 5 seconds and is not actually used by a subsequent read request will be regarded as expired data and needs to be released from the cache. During the operation of the distributed storage system, there are two files F1 and F2 whose pre-read data are cached. The pre-read data of file F1 has stayed in the cache for 4 seconds, while the pre-read data of file F2 has stayed for 6 seconds. At this time, the distributed storage system responds to the release instruction of the cache space and starts to detect the pre-read data in the cache.

[0055] For the pre-read data of file F1, detection finds that its cache time is less than the third preset time (4 seconds < 5 seconds), so the distributed storage system determines that this part of data still has pre-read value and does not need to be released immediately, and continues to remain in the cache. For the pre-read data of file F2, detection finds that its cache time exceeds the third preset time (6 seconds > 5 seconds), and the distributed storage system determines that this part of data has expired. In order to avoid continuing to occupy the cache space, a release operation is performed to remove the pre-read data of file F2 from the cache and release the cache space occupied by it.

[0056] In this embodiment, through such a mechanism, the distributed storage system can intelligently manage the pre-read data in the cache, ensure that the cache space is always occupied by the most needed data, improve the utilization efficiency of the cache, reduce the waste of cache space, and thus improve the overall performance and response speed of the distributed storage system, especially when processing constant bitstream files, it can effectively avoid performance bottlenecks caused by excessive pre-read data and long-term occupation of cache space.

[0057] Obviously, the above-described embodiments are only a part of the embodiments of the present application, not all of the embodiments. In order to better understand the above-mentioned file caching method, the above-mentioned process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application, specifically:

[0058] In an alternative embodiment, Figure 3 is a control flow chart of the file caching method according to an embodiment of the present application, such as Figure 3 The specific contents are as follows:

[0059] Step S301: When the distributed storage system is initialized, a global cache quota is set according to the number of files and objects read and written by the current distributed storage system, where the global cache quota includes a soft quota (equivalent to the first cache quota) and a hard quota (equivalent to the second cache quota). The soft quota is used as a warning line. When the cache usage approaches this limit, the distributed storage system will take preventive measures; while the hard quota is an absolute upper limit. When this limit is reached or exceeded, no new objects will be allowed to be added to the cache.

[0060] Step S302: During the operation of the distributed storage system, the client performs read and write operations on constant bitstream files. These constant bitstream files generate data at a uniform rate and require stable storage performance to avoid degradation of video or media quality.

[0061] Step S303: Each time the client attempts to add a new object (such as file data) to the cache, the distributed storage system first checks the usage of the current cache.

[0062] Step S304: Evaluate whether the current cache space has reached the soft quota. If yes, execute step S306; otherwise, execute step S305.

[0063] Step S305: If the cache occupancy does not exceed the soft quota, file caching and constant bitrate reading and writing are performed normally.

[0064] Step S306: If the cache occupancy exceeds the soft quota, the distributed storage system will start a waiting mechanism and put the new cache request into a waiting queue.

[0065] Step S307, periodically checking whether there is enough cache space available in the cache space by polling. When a cache object is released in the cache space, the request will be read from the waiting queue in order, and then the new file will be added to the cache space.

[0066] Step S309: evaluating whether the current cache space has reached the hard quota.

[0067] Step S310: When the current cache space reaches the hard quota, no new objects will be allowed to be added to the cache to ensure the stability and robustness of the distributed storage system. This mechanism is the key to quota control, which can prevent cache overload and ensure the normal reading and writing of all existing file data.

[0068] Through the above-mentioned embodiments, lock contention in the distributed storage system is reduced, data congestion is avoided, and the response speed and efficiency of the distributed storage system are improved. The efficient use of cache space is ensured, while the breakthrough of the hard quota is avoided, and the read and write access of the existing number of paths is ensured not to be affected. Through the above-mentioned waiting mechanism and priority queue management, the distributed storage system can dynamically adjust the use of the cache to ensure that the actual use of the cache is always lower than the hard quota, but as close to the soft quota as possible, thereby maximizing the utilization efficiency of the cache while preventing request blocking.

[0069] In the above embodiment, efficient management and utilization of cache resources are achieved through intelligent cache allocation and release strategies, while cache overload is avoided, ensuring high responsiveness and continuous service capabilities of the distributed storage system. Figure 4 is a quota update flow chart of the file caching method according to an embodiment of the present application, such as Figure 4 The specific contents are as follows:

[0070] Step S401: When a client (such as an application, an operating system or other service) initiates a read or write operation on a file, the distributed storage system receives the read or write request.

[0071] Step S402: The distributed storage system first parses the file path in the request, and searches and obtains the inode number (index number) of the corresponding file through the inode mechanism of the file distributed storage system. The distributed storage system checks whether there is a data structure related to the inode number of the requested file in the cache. If the file type and size have changed, the distributed storage system will identify that this is a new file based on the inode number and execute step S403, otherwise execute S404.

[0072] Step S403: Increase the variable count of the number of files in the cache by one.

[0073] Step S404: According to the type and size of the new file, the system memory allocates cache and updates the cache quota base.

[0074] Step S405: After the global cache quota is updated, the distributed storage system intelligently determines whether the request needs to wait according to the current cache status and the type of the read or write request.

[0075] In the above embodiment, a quota update flow chart of the file caching method is described. In order to better understand the above file caching method, the above process is described below in conjunction with the embodiment, but it is not intended to limit the technical solution of the embodiment of the present application. Figure 5 is a flowchart of a write request submission method according to an embodiment of the present application, such as Figure 5 As shown, the specific steps are as follows:

[0076] Step S501: When a client (such as an application, an operating system or other service) initiates a write operation, the write request is received by the distributed storage system.

[0077] Step S502: When a write request enters the distributed storage system, the distributed storage system maps the write data to the internal bufferhead structure. The bufferhead structure is a key data structure in the Linux kernel for managing the cache of disk blocks in memory. It is responsible for storing and maintaining disk block status information, such as dirty bits, block device offsets, etc. By mapping the write data to the bufferhead structure, the distributed storage system can effectively manage the write data and ensure the correctness and consistency of the data.

[0078] Step S503: The distributed storage system further analyzes the write request to determine whether it meets the pre-set flushing (i.e., data is flushed from the cache to the disk) condition. This judgment is usually based on whether the amount of data covered by the write request reaches or approaches the stripe size of the storage device. If the write request covers the entire stripe or is nearly complete (for example, more than 80% of the stripe size), it is considered to meet the flushing condition. In addition, the distributed storage system may also decide whether the request needs to be flushed immediately based on the current usage status of the cache, the cache policy (such as LRU, LFU) or the distributed storage system configuration (such as cache quota).

[0079] Step S504: When the distributed storage system determines that the write request meets the flushing condition, the corresponding bufferhead structure is placed in a high-priority flushing queue. This flushing queue is regularly checked by the background refresh thread, which is responsible for flushing the data in the cache to the disk. By placing the write requests that meet the conditions in the flushing queue first, the distributed storage system can respond to write operations quickly, avoid data backlogs, and reduce the delay of sync synchronization operations, thereby improving the write performance of the distributed storage system and the persistence of data.

[0080] Step S505: If the write request fails to meet the flushing condition, the distributed storage system temporarily retains it in the cache and waits for subsequent write requests. These subsequent write requests may belong to the same file or object as the current request, so the distributed storage system can try to merge them into a larger request to meet the flushing condition. The merge strategy aims to take advantage of the opportunity of continuous write operations to improve the flushing efficiency and reduce the number of disk I / Os, thereby improving the overall performance of the distributed storage system. At the same time, this step will also take into account the cache usage efficiency and cache quota to avoid excessive cache occupancy.

[0081] In the above embodiments, the distributed storage system can adopt differentiated processing strategies for different types of write requests, optimize write performance, increase data persistence speed, and ensure effective management and utilization of cache resources. Figure 6 is a pre-read data protection flow chart of the file caching method according to an embodiment of the present application, such as Figure 6 As shown, the specific steps are as follows:

[0082] Step S601: When the distributed storage system detects that the cache usage is too high or resources need to be reclaimed, a cache trim (cleaning and releasing cached data) process is started.

[0083] Step S602: The distributed storage system selects a bufferhead element from the Least Recently Used (LRU) linked list for inspection. The LRU linked list is a commonly used data structure in cache management, which is used to track the usage frequency of elements in the cache and help the distributed storage system identify which elements are least used, so as to give priority to releasing the cache space occupied by these elements when needed.

[0084] Step S603: The distributed storage system checks whether the selected bufferhead is in a clean state. If the bufferhead is in a clean state, jump to step S604, and the distributed storage system further determines whether the bufferhead is pre-read data. If the bufferhead is not in a clean state, it means that the data may be undergoing a read or write operation, or needs to be synchronized to the disk. At this time, the distributed storage system will not perform cache recovery, and the distributed storage system returns to step S602 and selects the next bufferhead element in the LRU linked list for inspection to avoid recovering the data in use. Among them, the clean state means that the data in the buffer is consistent with the data on the disk, and the data has not been modified.

[0085] Step S604: When the bufferhead is in the clean state, the distributed storage system further determines whether the bufferhead is pre-read data. If the data in the bufferhead is identified as pre-read data, jump to step S605 to determine whether the cache time of the pre-read data exceeds the preset time threshold; if the data in the bufferhead is not pre-read data, jump to step S606, and the distributed storage system immediately releases and reclaims the bufferhead cache space to make room for subsequent cache operations. Pre-reading means that the distributed storage system loads part of the data into the cache in advance to improve the efficiency of subsequent read operations. Pre-read data is usually loaded into the cache before the read operation to improve the hit rate of the read cache.

[0086] Step S605: For the pre-read data, the distributed storage system checks whether the time it stays in the cache exceeds a preset time threshold (such as 3 seconds). If the pre-read data has stayed in the cache for more than 3 seconds, it means that the data may no longer be needed, and jumps to step S606. The distributed storage system will release and recycle the corresponding bufferhead cache space. If the cache time does not reach 3 seconds, it means that the data may still be used by subsequent read operations. The distributed storage system will retain the data and return to step S602 to continue checking the next bufferhead element in the LRU linked list.

[0087] Step S606: the distributed storage system immediately releases and reclaims the bufferhead cache space to free up space for subsequent cache operations.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0089] In the present embodiment, a file caching device is also provided, and the system is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions that have been made are not repeated here. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0090] Figure 7 is a structural block diagram of a file caching device according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0091] A first determination module 70, configured to determine a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0092] The attribute acquisition module 72 is used to parse the received file read / write request and acquire the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0093] The second determination module 74 is used to determine a cache method for the target file according to the file attribute and the cache quota.

[0094] Through the above-mentioned device, the cache quota configured by the file storage system for the cache space is determined based on the amount of file data that the file storage system needs to read and write; the received file read and write request is parsed to obtain the file attributes of the target file corresponding to the file read and write request, and the target file is a constant code stream file; the caching method for the target file is determined according to the file attributes and the cache quota, thereby solving the problem of unstable reading and writing due to insufficient storage system performance during the reading and writing process of the constant code stream file in the related technology.

[0095] In an exemplary embodiment, the attribute acquisition module 72 is also used to determine whether the target file has been cached in the cache space based on the file identifier; when it is determined that the target file is not cached in the cache space, determine the first occupied capacity of the cache space; compare the first occupied capacity with the first cache quota to obtain a first comparison result; when it is determined that the first comparison result indicates that the first occupied capacity is less than the first cache quota, cache the target file to the cache space; when it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, add the file read and write request corresponding to the target file to a waiting queue to wait for the target file to be cached to the cache space; when it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, reject the file read and write request corresponding to the target file.

[0096] In an exemplary embodiment, the attribute acquisition module 72 is also used to detect the cache space according to a preset period, and when it is determined that the cache space has released files in the previous period, determine the second occupied capacity of the cache space; compare the second occupied capacity with the first cache quota to obtain a second comparison result; and when it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota, cache the target file to the cache space.

[0097] In an exemplary embodiment, the attribute acquisition module 72 is also used for the cache time of the cache file to exceed a first preset time; the cache file has not been accessed for more than a second preset time; the cache file has the lowest access frequency within a preset time period.

[0098] In an exemplary embodiment, the attribute acquisition module 72 is also used to obtain the stripe size of the file storage system when it is determined that the file read and write request is a write request; compare the file size of the target file with the stripe size to obtain a third comparison result; and when it is determined that the third comparison result indicates that the file size of the target file is smaller than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, add the file read and write request to a priority queue, and the priority of the priority queue is higher than the priority of the waiting queue.

[0099] In an exemplary embodiment, the second determination module 74 is also used to, when determining that the file read and write request is a read request, read the target file from the file storage system, read data of a preset address length starting from the end storage address of the target file, and obtain pre-read data; and cache the target file and the pre-read data into the cache space.

[0100] In an exemplary embodiment, the second determination module 74 is also used to respond to a release instruction for releasing the cache space, detect whether the cache time of the pre-read data exceeds a third preset time; and release the pre-read data if it is determined that the cache time of the pre-read data exceeds the third preset time.

[0101] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0102] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0103] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0104] S1, determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0105] S2, parsing the received file read / write request to obtain the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0106] S3: Determine a cache method for the target file according to the file attribute and the cache quota.

[0107] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0108] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0109] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0110] S1, determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0111] S2, parsing the received file read / write request to obtain the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0112] S3: Determine a cache method for the target file according to the file attribute and the cache quota.

[0113] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0114] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:

[0115] S1, determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0116] S2, parsing the received file read / write request to obtain the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0117] S3: Determine a cache method for the target file according to the file attribute and the cache quota.

[0118] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0119] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0120] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A file caching method, characterized in that: Applied to a file storage system, the file storage system is used to read and write files, and cache the read files into a built-in cache space, the method includes: Determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; Parsing the received file read / write request to obtain the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; A cache mode for the target file is determined according to the file attribute and the cache quota.

2. The method according to claim 1, characterized in that The file attribute includes a file identifier, the cache quota includes a first cache quota and a second cache quota, the first cache quota is smaller than the second cache quota, and determining a cache mode for the target file according to the file attribute and the cache quota includes: Determining whether the target file has been cached in the cache space according to the file identifier; In the case where it is determined that the target file is not cached in the cache space, determining a first occupied capacity of the cache space; Comparing the first occupied capacity with the first cache quota to obtain a first comparison result; If it is determined that the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file into the cache space; When it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read and write request corresponding to the target file to a waiting queue to wait for caching the target file into the cache space; When it is determined that the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, the file read and write request corresponding to the target file is rejected.

3. The method according to claim 2, characterized in that Adding the file read / write request corresponding to the target file to a waiting queue to wait for caching the target file into the cache space includes: Detecting the cache space according to a preset cycle, and determining a second occupied capacity of the cache space when it is determined that the cache space has released a file in a previous cycle; comparing the second occupied capacity with the first cache quota to obtain a second comparison result; When it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota, the target file is cached in the cache space.

4. The method according to claim 3, characterized in that When it is determined that a preset condition is met, the cached file in the cache space is released, and the preset condition includes at least one of the following: The cache time of the cache file exceeds a first preset time; The cache file has not been accessed for more than a second preset time; The cache file has the lowest access frequency within a preset time period.

5. The method according to claim 2, characterized in that: The file attribute also includes the file size, and adding the file read / write request corresponding to the target file to a waiting queue to wait for caching the target file into the cache space includes: When determining that the file read / write request is a write request, obtaining a stripe size of the file storage system; Comparing the file size of the target file with the stripe size to obtain a third comparison result; When it is determined that the third comparison result indicates that the file size of the target file is smaller than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, the file read and write request is added to a priority queue, and the priority of the priority queue is higher than the priority of the waiting queue.

6. The method according to claim 1, characterized in that The method further comprises: In the case where it is determined that the file read / write request is a read request, the target file is read from the file storage system, and data of a preset address length is read from the end storage address of the target file to obtain pre-read data; The target file and the pre-read data are cached in the cache space.

7. The method according to claim 6, characterized in that After caching the target file and the pre-read data in the cache space, the method further includes: In response to a release instruction for releasing the cache space, detecting whether a cache time of the pre-read data exceeds a third preset time; When it is determined that the cache time of the pre-read data exceeds the third preset time, the pre-read data is released.

8. A file cache device, characterized in that: The device comprises: A first determination module, configured to determine a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; The attribute acquisition module is used to parse the received file read / write request and acquire the file attribute of the target file corresponding to the file read / write request, wherein the target file is a constant bit stream file; The second determination module is used to determine the cache mode of the target file according to the file attribute and the cache quota.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images