File caching methods, devices, storage media, and electronic devices

Abstract

This application provides a file caching method, apparatus, storage medium, and electronic device, relating to the server field. The file storage system is used for reading and writing files, and caching read files to a built-in cache space. The method includes: determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; parsing a received file read / write request to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; and determining a caching method for the target file based on the file attributes and the cache quota. This solves the problem in related technologies where, during the reading and writing of constant bitstream files, insufficient storage system performance leads to unstable read / write operations.

Description

Technical Field

[0001] This application relates to the field of servers, and more specifically, to a file caching method, apparatus, storage medium, and electronic device. Background Technology

[0002] In video surveillance and broadcast media asset management, files must be read and written at a constant bitrate. Any stuttering due to insufficient storage performance will cause frame drops, severely impacting video and media quality. Traditional distributed storage systems rely primarily on general caching management strategies when dealing with intensive read / write operations of constant bitrate files. However, when processing multiple constant bitrate files simultaneously, these strategies often fail to effectively guarantee the stability of read / write operations for each file and the fair allocation of cache resources. Consequently, during the read / write process of constant bitrate files, insufficient storage system performance leads to instability.

[0003] In the context of related technologies, there is no effective solution to the problem of unstable read / write operations caused by insufficient storage system performance during the reading and writing of constant bitstream files. Summary of the Invention

[0004] This application provides a file caching method, apparatus, storage medium, and electronic device to at least solve the problem of unstable read / write operations caused by insufficient storage system performance during the reading and writing of constant bitstream files in related technologies.

[0005] According to one embodiment of this application, a file caching method is provided, applied to a file storage system for reading and writing files, and caching read files to a built-in cache space. The method includes: determining a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; parsing a received file read / write request to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; and determining a caching method for the target file based on the file attributes and the cache quota.

[0006] In an exemplary embodiment, the file attributes include a file identifier, and the cache quota includes a first cache quota and a second cache quota, wherein the first cache quota is less than the second cache quota. Determining the caching method for the target file based on the file attributes and the cache quota includes: determining whether the target file is already cached in the cache space based on the file identifier; if the target file is not cached in the cache space, determining a first occupied capacity of the cache space; comparing the first occupied capacity with the first cache quota to obtain a first comparison result; if the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space; and if the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, rejecting the file read / write request corresponding to the target file.

[0007] In an exemplary embodiment, adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space includes: detecting the cache space according to a preset period; if it is determined that the cache space has released a file in the previous period, determining a second occupied capacity of the cache space; comparing the second occupied capacity with the first cache quota to obtain a second comparison result; if it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota, caching the target file in the cache space.

[0008] In one exemplary embodiment, when it is determined that preset conditions are met, the cached files in the cache space are released. The preset conditions include at least one of the following: the cache time of the cached file exceeds a first preset time; the cached file has not been accessed for more than a second preset time; the access frequency of the cached file is the lowest within a preset time period.

[0009] In one exemplary embodiment, the file attribute further includes file size. Adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space includes: if it is determined that the file read / write request is a write request, obtaining the stripe size of the file storage system; comparing the file size of the target file with the stripe size to obtain a third comparison result; if it is determined that the third comparison result indicates that the file size of the target file is less than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, adding the file read / write request to a priority queue, wherein the priority queue has a higher priority than the waiting queue.

[0010] In one exemplary embodiment, the method further includes: when it is determined that the file read / write request is a read request, reading the target file from the file storage system, reading data of a preset address length starting from the termination storage address of the target file to obtain pre-read data; and caching the target file and the pre-read data into the cache space.

[0011] In an exemplary embodiment, after caching the target file and the pre-read data into the cache space, the method further includes: responding to a release instruction for releasing the cache space, detecting whether the caching time of the pre-read data exceeds a third preset time; and if it is determined that the caching time of the pre-read data exceeds the third preset time, releasing the pre-read data.

[0012] According to another embodiment of this application, a file caching device is provided, comprising: a first determining module, configured to determine a cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write; an attribute acquisition module, configured to parse a received file read / write request and acquire file attributes of a target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; and a second determining module, configured to determine a caching method for the target file based on the file attributes and the cache quota.

[0013] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0014] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0016] This application determines the cache quota configured for the cache space of the file storage system based on the amount of file data that the file storage system needs to read and write; it parses the received file read / write requests to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; and it determines the caching method for the target file based on the file attributes and the cache quota. This solves the problem of unstable read / write operations caused by insufficient storage system performance during the read / write process of constant bitstream files in related technologies. Attached Figure Description

[0017] Figure 1 This is a hardware structure block diagram of a server device for a file caching method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart of a file caching method according to an embodiment of this application;

[0019] Figure 3 This is a control flowchart of a file caching method according to an embodiment of this application;

[0020] Figure 4 This is a flowchart illustrating the quota update process of the file caching method according to an embodiment of this application;

[0021] Figure 5 This is a flowchart illustrating the write request submission process of the file caching method according to an embodiment of this application;

[0022] Figure 6 This is a flowchart illustrating the pre-read data protection process of a file caching method according to an embodiment of this application;

[0023] Figure 7 This is a structural block diagram of a file caching device according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1This is a hardware structure block diagram of a server device for a file caching method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the file caching method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned file caching method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] This embodiment provides a file caching method applied to a file storage system, which is used to read and write files and cache the read files to a built-in cache space. Figure 2 This is a flowchart of a file caching method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0031] Step S204: Parse the received file read / write request to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0032] Step S206: Determine the caching method for the target file based on the file attributes and the cache quota.

[0033] Through the above steps, the cache quota configured by the file storage system for the cache space is determined based on the amount of file data that the file storage system needs to read and write; the received file read and write requests are parsed to obtain the file attributes of the target file corresponding to the file read and write request, wherein the target file is a constant bitstream file; the caching method for the target file is determined according to the file attributes and the cache quota, thereby solving the problem of unstable read and write operations caused by insufficient storage system performance during the read and write process of constant bitstream files in related technologies.

[0034] In an exemplary embodiment, the file attributes include a file identifier, and the cache quota includes a first cache quota and a second cache quota, wherein the first cache quota is less than the second cache quota. Determining the caching method for the target file based on the file attributes and the cache quota includes: determining whether the target file is already cached in the cache space based on the file identifier; if the target file is not cached in the cache space, determining a first occupied capacity of the cache space; comparing the first occupied capacity with the first cache quota to obtain a first comparison result; if the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space; and if the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, rejecting the file read / write request corresponding to the target file.

[0035] Optionally, assuming the total cache space in the distributed storage system is 10GB, the first cache quota is set to 5GB, and the second cache quota is set to 8GB. The system currently occupies 4GB of cache space. When a read / write request for a new file (e.g., a surveillance video file) arrives, the system first checks whether the file already exists in the cache. If it does not exist, it continues to check the currently occupied capacity of 4GB, which is less than the first cache quota of 5GB. The system will then directly load some or all of the file's data into the cache to accelerate its read / write operations.

[0036] Suppose the cache usage subsequently increases to 6GB, and then a new file read / write request arrives. The distributed storage system checks and finds that the current occupied capacity is 6GB, equal to or greater than the first cache quota of 5GB, but less than the second cache quota of 8GB. Therefore, the new file read / write request is placed in a waiting queue. These requests will be added to the cache according to priority (e.g., most recently or most frequently used files are prioritized) after the cached files in the cache space are released, instead of being immediately blocked or rejected. If the cache usage reaches 9GB, and a new file read / write request arrives, the system will check and confirm that the occupied capacity is 9GB, equal to or greater than the second cache quota of 8GB. In this case, the system will directly reject the read / write request for that file to avoid cache overload and ensure that the read / write performance of cached files is not affected.

[0037] In this embodiment, by setting file identifier, first cache quota and second cache quota, the caching method of the file is dynamically adjusted. When processing constant bitstream files in a distributed storage system, cache resources can be dynamically managed, which not only ensures the efficient use of cache, but also avoids the waste and overload of cache resources, thus ensuring the stability and response speed of constant bitstream file read and write services.

[0038] In an exemplary embodiment, adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space includes: detecting the cache space according to a preset period; if it is determined that the cache space has released a file in the previous period, determining a second occupied capacity of the cache space; comparing the second occupied capacity with the first cache quota to obtain a second comparison result; if it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota, caching the target file in the cache space.

[0039] Alternatively, suppose a distributed storage system has a total cache size of 10GB, with a first cache quota of 5GB. Currently, 5.8GB of cache space is occupied. At this point, the distributed storage system receives a read / write request for a new file (i.e., the target file). However, because the distributed storage system detects that the currently occupied capacity exceeds the first cache quota, it places the request in a waiting queue.

[0040] In subsequent pre-defined periodic checks, the distributed storage system discovered that some file data from the previous period had been evicted due to prolonged inactivity, releasing 1.2GB of cache space and reducing the occupied capacity to 4.6GB. At this point, the distributed storage system recalculates the second occupied capacity and compares it with the first cache quota. The comparison shows that the second occupied capacity is less than the first cache quota (4.6GB < 5GB). Therefore, the distributed storage system retrieves the target file read / write requests previously placed in the waiting queue, caches the target file data in the cache space, and performs subsequent read / write operations.

[0041] In this embodiment, when the cache space is close to the first cache quota limit, the cache space usage is periodically detected, and the caching strategy is dynamically adjusted based on the detection results to ensure that the cache can be used reasonably even when cache resources are scarce, thereby improving the system's read and write performance and response speed.

[0042] In one exemplary embodiment, when it is determined that preset conditions are met, the cached files in the cache space are released. The preset conditions include at least one of the following: the cache time of the cached file exceeds a first preset time; the cached file has not been accessed for more than a second preset time; the access frequency of the cached file is the lowest within a preset time period.

[0043] Optionally, assume a distributed storage system with a cache space whose management strategy includes the aforementioned cache file release conditions. The cache space currently stores data for three files, A, B, and C, representing different video stream caches. The distributed storage system has a first preset timeout of 30 minutes and a second preset timeout of 10 minutes, and access frequency is counted hourly. Assume that file A has been continuously accessed since being loaded into the cache space, and the access frequency is high. File B was last accessed 9 minutes ago; although the second preset timeout has not yet been reached, its access frequency is lower than that of file A. File C has been in the cache space for 31 minutes since being loaded, and has not been accessed during this time.

[0044] In the next evaluation cycle of cache management (let's say 30 minutes later), when the distributed storage system checks whether the dwell time of all cached files exceeds the first preset time (30 minutes), since file C's cache time has exceeded 30 minutes and it hasn't been accessed, file C will be released according to the first condition. When the distributed storage system checks the file access status, although file B hasn't reached the second preset time, the system also needs to calculate the access frequency over the past hour. It finds that file B has the lowest access frequency. According to the third condition, file B may also be released in the next hourly access frequency calculation to free up space for more frequently accessed files. File A, due to its consistently high access frequency and not exceeding any preset time, will not be released and will continue to remain in the cache space to provide a fast response for subsequent read and write requests.

[0045] This embodiment describes a cache file release mechanism designed to dynamically manage cache space by setting specific conditions, ensuring efficient utilization of cache resources while avoiding resource waste or cache overload. This cache file release mechanism is a crucial component of cache management strategies. It automatically releases files that are no longer active or have exceeded a specific time limit by monitoring the lifecycle and access patterns of cache files, freeing up space for new files.

[0046] In one exemplary embodiment, the file attribute further includes file size. Adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space includes: if it is determined that the file read / write request is a write request, obtaining the stripe size of the file storage system; comparing the file size of the target file with the stripe size to obtain a third comparison result; if it is determined that the third comparison result indicates that the file size of the target file is less than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, adding the file read / write request to a priority queue, wherein the priority queue has a higher priority than the waiting queue.

[0047] Optionally, suppose a distributed storage system has a stripe size of 1MB and a preset ratio of 0.8 (i.e., the file size is at least 80% of the stripe size). The current cache space is close to the first cache quota, and all write requests are temporarily placed in a waiting queue. At this time, the distributed storage system receives three write requests: File X is 500KB, significantly smaller than the stripe size, but its size-to-stripe ratio is 0.5, less than the preset ratio. File Y is 900KB, smaller than the stripe size, and its size-to-stripe ratio is 0.9, exceeding the preset ratio. File Z is 1.2MB, larger than the stripe size. According to the above strategy, file Y will be added to the priority queue first because it is closer to the complete stripe size. The distributed storage system is more efficient when writing files close to the complete stripe size, thus improving data processing efficiency and accelerating the request processing.

[0048] It's important to note that stripe size is a crucial concept in distributed storage systems. It refers to the way data is divided into equal-sized blocks when written to multiple storage devices, with each block written to a different device. The choice of stripe size significantly impacts the performance of a distributed storage system. Larger stripe sizes reduce metadata overhead and increase write speed, but increase data unevenness; smaller stripe sizes increase data dispersion but lead to higher metadata management costs.

[0049] In this embodiment, the distributed storage system can intelligently distinguish different write requests. For the writing of small files, especially those close to the stripe size and with high utilization, a priority processing strategy is adopted to improve the system's write efficiency and response speed, while avoiding delays or denial of service due to insufficient cache space.

[0050] In one exemplary embodiment, the method further includes: when it is determined that the file read / write request is a read request, reading the target file from the file storage system, reading data of a preset address length starting from the termination storage address of the target file to obtain pre-read data; and caching the target file and the pre-read data into the cache space.

[0051] Alternatively, suppose there is a distributed storage system with ample cache space. File A is a video file, 20MB in size, and the current read request is to read the last 5MB of file A. The distributed storage system has a preset prefetch address length of 1MB, and the storage of file A is divided and distributed based on 1MB stripes. According to the prefetch strategy, while reading the last 5MB of data in file A, the distributed storage system also reads the next 1MB of data starting from the end storage address of file A as prefetch data. The prefetch data and the data in the read request are loaded into the cache together, providing a fast response for subsequent possible read operations. If a subsequent read request happens to point to the address range covered by the prefetch data, i.e., the last 3MB of file A (2MB read request + 1MB prefetch data), then the read operation can be completed directly from the cache without accessing the storage system again, thus greatly improving read efficiency and user experience.

[0052] In this embodiment, through this pre-read strategy, the distributed storage system can effectively predict and optimize read operations. Especially when processing constant bitstream files in the fields of video surveillance or broadcast media assets, it can significantly reduce read latency, improve cache hit rate, and thus improve the read performance and response speed of the entire distributed storage system.

[0053] In an exemplary embodiment, after caching the target file and the pre-read data into the cache space, the method further includes: responding to a release instruction for releasing the cache space, detecting whether the caching time of the pre-read data exceeds a third preset time; and if it is determined that the caching time of the pre-read data exceeds the third preset time, releasing the pre-read data.

[0054] Optionally, assuming the third preset timeout in the distributed storage system is 5 seconds, this means that any pre-fetched data that remains in the cache for more than 5 seconds without being actually used by subsequent read requests will be considered expired data and needs to be released from the cache. During the operation of the distributed storage system, there are two files, F1 and F2, with pre-fetched data cached. The pre-fetched data for file F1 has remained in the cache for 4 seconds, while the pre-fetched data for file F2 has remained for 6 seconds. At this point, the distributed storage system responds to a cache space release command and begins checking the pre-fetched data in the cache.

[0055] For the pre-read data of file F1, the detection found that its cache time was less than the third preset time (4 seconds < 5 seconds). Therefore, the distributed storage system determined that this data still had pre-read value and did not need to be released immediately, so it remained in the cache. For the pre-read data of file F2, the detection found that its cache time exceeded the third preset time (6 seconds > 5 seconds). The distributed storage system determined that this data had expired. To avoid continuously occupying cache space, a release operation was performed, removing the pre-read data of file F2 from the cache and releasing the cache space it occupied.

[0056] In this embodiment, through such a mechanism, the distributed storage system can intelligently manage the pre-fetched data in the cache, ensuring that the cache space is always occupied by the most needed data, improving the utilization efficiency of the cache, reducing the waste of cache space, and thus improving the overall performance and response speed of the distributed storage system. Especially when processing constant bitstream files, it can effectively avoid performance bottlenecks caused by excessive pre-fetched data and long-term occupation of cache space.

[0057] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the above file caching method, the process is described below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0058] In an optional embodiment, Figure 3 This is a control flowchart of a file caching method according to an embodiment of this application, such as... Figure 3 As shown, the specific content is as follows:

[0059] Step S301: During the initialization of the distributed storage system, a global cache quota is set based on the current number of files and objects read and written by the distributed storage system. The global cache quota includes a soft quota (equivalent to the first cache quota) and a hard quota (equivalent to the second cache quota). The soft quota serves as a warning line; when cache usage approaches this limit, the distributed storage system will take preventative measures. The hard quota, on the other hand, is an absolute upper limit; when this limit is reached or exceeded, no new objects will be allowed to be added to the cache.

[0060] Step S302: During the operation of the distributed storage system, the client performs read and write operations on constant bitstream files. These constant bitstream files generate data at a uniform rate and require stable storage performance to avoid degradation of video or media quality.

[0061] Step S303: Each time a client attempts to add a new object (such as file data) to the cache, the distributed storage system first checks the current cache usage.

[0062] Step S304: Evaluate whether the current cache space usage has reached the soft quota. If yes, proceed to step S306; otherwise, proceed to step S305.

[0063] Step S305: If the cache usage does not exceed the soft quota, file caching and constant bitstream read / write are performed normally.

[0064] Step S306: If the cache usage exceeds the soft quota, the distributed storage system will initiate a waiting mechanism, placing new cache requests into a waiting queue.

[0065] Step S307: Periodically check whether there is enough cache space available by polling. When a cached object is released in the cache space, requests will be read from the waiting queue in order, and then the new file will be added to the cache space.

[0066] Step S309: Evaluate whether the current cache space usage has reached the hard quota.

[0067] In step S310, when the current cache space reaches the hard quota, no new objects will be allowed to be added to the cache to ensure the stability and robustness of the distributed storage system. This mechanism is key to quota control; it prevents cache overload and ensures the normal operation of all existing file data read and write operations.

[0068] The above embodiments reduce lock contention in the distributed storage system, avoid data congestion, and improve the response speed and efficiency of the distributed storage system. They ensure efficient utilization of cache space while preventing hard quota violations, guaranteeing that read and write access to existing paths is not affected. Through the aforementioned waiting mechanism and priority queue management, the distributed storage system can dynamically adjust cache usage, ensuring that the actual cache usage is always lower than the hard quota, but as close as possible to the soft quota, thereby maximizing cache utilization efficiency while preventing request blocking.

[0069] In the above embodiments, an intelligent cache allocation and release strategy is used to achieve efficient management and utilization of cache resources, while avoiding cache overload and ensuring the high responsiveness and continuous service capability of the distributed storage system. Figure 4 This is a flowchart illustrating the quota update process of the file caching method according to an embodiment of this application, as follows: Figure 4 As shown, the specific content is as follows:

[0070] Step S401: When a client (such as an application, operating system or other service) initiates a read or write operation on a file, the distributed storage system receives this read or write request.

[0071] Step S402: The distributed storage system first parses the file path in the request, and then uses the inode mechanism of the distributed storage system to find and obtain the inode number (index number) of the corresponding file. The distributed storage system checks whether there is a data structure related to the inode number of the requested file in the cache. If the file type and size have changed, the distributed storage system will identify that this is a new file based on the inode number and execute step S403; otherwise, it will execute step S404.

[0072] Step S403: Increment the counter of the variable representing the number of files in the cache.

[0073] Step S404: Based on the type and size of the new file, allocate system memory for cache and update the cache quota base.

[0074] Step S405: After updating the global cache quota, the distributed storage system intelligently determines whether a request needs to wait based on the current cache status and the type of read / write request.

[0075] The above embodiments describe a flowchart of the quota update process for the file caching method. To better understand the above file caching method, the following description, in conjunction with embodiments, illustrates the process but is not intended to limit the technical solutions of the embodiments of this application. Figure 5 This is a flowchart of the write request submission process for the file caching method according to an embodiment of this application, as follows: Figure 5 As shown, the specific steps are as follows:

[0076] Step S501: When a client (such as an application, operating system or other service) initiates a write operation, the write request is received by the distributed storage system.

[0077] Step S502: When a write request enters the distributed storage system, the system maps the write data to its internal bufferhead structure. The bufferhead structure is a key data structure in the Linux kernel used to manage the caching of disk blocks in memory. It is responsible for storing and maintaining the state information of disk blocks, such as dirty bits and block device offsets. By mapping write data to the bufferhead structure, the distributed storage system can effectively manage written data, ensuring data correctness and consistency.

[0078] Step S503: The distributed storage system further analyzes the write request to determine whether it meets the pre-defined flushing (i.e., data being flushed from the cache to the disk) conditions. This determination is typically based on whether the amount of data covered by the write request reaches or approaches the stripe size of the storage device. If the write request covers the entire stripe or nearly the entire stripe (e.g., more than 80% of the stripe size), it is considered to meet the flushing conditions. Additionally, the distributed storage system may also determine whether a request needs to be flushed immediately based on the current cache usage state, cache policies (such as LRU, LFU), or distributed storage system configurations (such as cache quotas).

[0079] Step S504: When the distributed storage system determines that a write request meets the flushing conditions, it places the corresponding bufferhead structure into a high-priority flushing queue. This flushing queue is periodically checked by a background flushing thread, which is responsible for flushing the data in the cache to disk. By prioritizing write requests that meet the conditions and placing them into the flushing queue, the distributed storage system can respond quickly to write operations, avoid data backlog, reduce the latency of sync operations, and thus improve the write performance and data persistence of the distributed storage system.

[0080] Step S505: If a write request fails to meet the flush condition, the distributed storage system temporarily holds it in the cache, awaiting subsequent write requests. These subsequent write requests may belong to the same file or object as the current request, allowing the distributed storage system to attempt to merge them into a larger request to meet the flush condition. This merging strategy aims to leverage the opportunity of consecutive write operations to improve flush efficiency, reduce disk I / O operations, and thus improve the overall performance of the distributed storage system. Simultaneously, this step also considers cache utilization efficiency and cache quotas to avoid excessive cache usage.

[0081] In the above embodiments, the distributed storage system can adopt differentiated processing strategies for different types of write requests, optimize write performance, improve data persistence speed, and ensure effective management and utilization of cache resources. Figure 6 This is a flowchart illustrating the pre-read data protection process of a file caching method according to an embodiment of this application, as follows: Figure 6 As shown, the specific steps are as follows:

[0082] Step S601: When the distributed storage system detects that the cache usage is too high or that resources need to be reclaimed, the cache trim (cleaning and releasing cached data) process is initiated.

[0083] Step S602: The distributed storage system selects a bufferhead element from the Least Recently Used (LRU) linked list for inspection. The LRU linked list is a commonly used data structure in cache management, used to track the usage frequency of elements in the cache, helping the distributed storage system identify which elements are used the least, and thus prioritize releasing the cache space occupied by these elements when needed.

[0084] Step S603: The distributed storage system checks whether the selected bufferhead is in a clean state. If the bufferhead is in a clean state, proceed to step S604, where the distributed storage system further determines whether the bufferhead contains data to be read. If the bufferhead is not in a clean state, it indicates that the data may be undergoing read / write operations or needs to be synchronized to disk. In this case, the distributed storage system will not perform cache reclamation and will return to step S602 to select the next bufferhead element in the LRU list for inspection to avoid reclaiming data currently in use. The clean state means that the data in the buffer is consistent with the data on disk and has not been modified.

[0085] Step S604: When the bufferhead is in the clean state, the distributed storage system further determines whether the bufferhead contains pre-read data. If the data in the bufferhead is identified as pre-read data, the process jumps to step S605 to determine whether the cache time of the pre-read data exceeds a preset time threshold. If the data in the bufferhead is not pre-read data, the process jumps to step S606, where the distributed storage system immediately releases and reclaims the bufferhead cache space to free up space for subsequent cache operations. Pre-reading refers to the distributed storage system loading some data into the cache in advance to improve the efficiency of subsequent read operations. Pre-read data is typically loaded into the cache before the read operation to improve the cache hit rate.

[0086] Step S605: For pre-read data, the distributed storage system checks whether its residence time in the cache exceeds a preset time threshold (e.g., 3 seconds). If the pre-read data has been in the cache for more than 3 seconds, it indicates that the data may no longer be needed, and the process jumps to step S606, where the distributed storage system releases and reclaims the corresponding bufferhead cache space. If the cache time has not reached 3 seconds, it indicates that the data may still be used by subsequent read operations, and the distributed storage system retains the data and returns to step S602 to continue checking the next bufferhead element in the LRU list.

[0087] Step S606: The distributed storage system immediately releases and reclaims the bufferhead cache space to free up space for subsequent caching operations.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0089] This embodiment also provides a file caching device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0090] Figure 7 This is a structural block diagram of a file caching device according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0091] The first determining module 70 is used to determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0092] The attribute acquisition module 72 is used to parse the received file read / write request and obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0093] The second determining module 74 is used to determine the caching method for the target file based on the file attributes and the cache quota.

[0094] Using the above-described apparatus, the cache quota configured by the file storage system for the cache space is determined based on the amount of file data that the file storage system needs to read and write; the received file read / write request is parsed to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; the caching method for the target file is determined according to the file attributes and the cache quota, thereby solving the problem of unstable read / write due to insufficient storage system performance during the read / write process of constant bitstream files in related technologies.

[0095] In an exemplary embodiment, the attribute acquisition module 72 is further configured to: determine whether the target file has been cached in the cache space based on the file identifier; if the target file is not cached in the cache space, determine a first occupied capacity of the cache space; compare the first occupied capacity with the first cache quota to obtain a first comparison result; if the first comparison result indicates that the first occupied capacity is less than the first cache quota, cache the target file in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, add the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space; and if the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, reject the file read / write request corresponding to the target file.

[0096] In an exemplary embodiment, the attribute acquisition module 72 is further configured to detect the cache space according to a preset period, and if it is determined that the cache space has released a file in the previous period, determine the second occupied capacity of the cache space; compare the second occupied capacity with the first cache quota to obtain a second comparison result; and if it is determined that the second comparison result indicates that the second occupied capacity is less than the first cache quota, cache the target file to the cache space.

[0097] In one exemplary embodiment, the attribute acquisition module 72 is further configured to: 1) the cache time of the cached file exceeds a first preset time; 2) the cached file is not accessed for more than a second preset time; and 3) the access frequency of the cached file is lowest within a preset time period.

[0098] In an exemplary embodiment, the attribute acquisition module 72 is further configured to, when determining that the file read / write request is a write request, acquire the stripe size of the file storage system; compare the file size of the target file with the stripe size to obtain a third comparison result; and, when determining that the third comparison result indicates that the file size of the target file is less than the stripe size and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, add the file read / write request to a priority queue, wherein the priority of the priority queue is higher than the priority of the waiting queue.

[0099] In an exemplary embodiment, the second determining module 74 is further configured to, when determining that the file read / write request is a read request, read the target file from the file storage system, read data of a preset address length starting from the termination storage address of the target file to obtain pre-read data; and cache the target file and the pre-read data in the cache space.

[0100] In an exemplary embodiment, the second determining module 74 is further configured to, in response to a release instruction for releasing the cache space, detect whether the cache time of the pre-read data exceeds a third preset time; and if it is determined that the cache time of the pre-read data exceeds the third preset time, release the pre-read data.

[0101] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0102] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0103] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0104] S1, determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0105] S2, parse the received file read / write request, and obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0106] S3, determine the caching method for the target file based on the file attributes and the cache quota.

[0107] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0108] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0109] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0110] S1, determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0111] S2, parse the received file read / write request, and obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0112] S3, determine the caching method for the target file based on the file attributes and the cache quota.

[0113] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0114] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps:

[0115] S1, determine the cache quota configured by the file storage system for the cache space based on the amount of file data that the file storage system needs to read and write;

[0116] S2, parse the received file read / write request, and obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file;

[0117] S3, determine the caching method for the target file based on the file attributes and the cache quota.

[0118] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0119] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be integrated on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0120] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A file caching method, characterized in that, Applied to a file storage system for reading and writing files, and caching read files to a built-in cache space, the method includes: The cache quota configured by the file storage system for the cache space is determined based on the amount of file data that the file storage system needs to read and write; The received file read / write request is parsed to obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; The caching method for the target file is determined based on the file attributes and the cache quota. If the file read / write request is determined to be a read request, the target file is read from the file storage system, and data of a preset address length is read starting from the terminating storage address of the target file to obtain pre-read data; the target file and the pre-read data are cached in the cache space. In response to a release command for releasing the cache space, the system detects whether the cache time of the pre-fetched data exceeds a third preset time; if it is determined that the cache time of the pre-fetched data exceeds the third preset time, the system releases the pre-fetched data. The file attributes include a file identifier, and the cache quota includes a first cache quota and a second cache quota, wherein the first cache quota is less than the second cache quota. Determining the caching method for the target file based on the file attributes and the cache quota includes: determining whether the target file is already cached in the cache space based on the file identifier; if the target file is not cached in the cache space, determining a first occupied capacity of the cache space; comparing the first occupied capacity with the first cache quota to obtain a first comparison result; if the first comparison result indicates that the first occupied capacity is less than the first cache quota, caching the target file in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space; and if the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, rejecting the file read / write request corresponding to the target file.

2. The method according to claim 1, characterized in that, Adding the file read / write request corresponding to the target file to the waiting queue to wait for the target file to be cached in the cache space includes: The cache space is checked according to a preset period. If it is determined that the cache space has released files in the previous period, the second occupied capacity of the cache space is determined. The second occupied capacity is compared with the first cache quota to obtain a second comparison result; If the second comparison result indicates that the second occupied capacity is less than the first cache quota, the target file is cached in the cache space.

3. The method according to claim 2, characterized in that, If a preset condition is met, the cached files in the cache space are released. The preset condition includes at least one of the following: The cache time of the cached file exceeds the first preset time; The cached file has not been accessed for more than a second preset time; The cached file is accessed at the lowest frequency within a preset time period.

4. The method according to claim 1, characterized in that, The file attributes also include file size. Adding the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space includes: If the file read / write request is determined to be a write request, the stripe size of the file storage system is obtained; The file size of the target file is compared with the strip size to obtain a third comparison result; If the third comparison result indicates that the file size of the target file is smaller than the stripe size, and the ratio of the file size of the target file to the stripe size is greater than a preset ratio, the file read / write request is added to a priority queue, where the priority queue has a higher priority than the waiting queue.

5. A file caching device, characterized in that, The device includes: The first determining module is used to determine the cache quota configured for the cache space of the file storage system based on the amount of file data that the file storage system needs to read and write; The attribute acquisition module is used to parse the received file read / write request and obtain the file attributes of the target file corresponding to the file read / write request, wherein the target file is a constant bitstream file; The second determining module is used to determine the caching method for the target file based on the file attributes and the cache quota. The device is further configured to, when determining that the file read / write request is a read request, read the target file from the file storage system, read data of a preset address length starting from the terminating storage address of the target file to obtain pre-read data; cache the target file and the pre-read data in the cache space; in response to a release instruction for releasing the cache space, detect whether the cache time of the pre-read data exceeds a third preset time; and, if it is determined that the cache time of the pre-read data exceeds the third preset time, release the pre-read data. The file attributes include a file identifier, and the cache quota includes a first cache quota and a second cache quota, wherein the first cache quota is less than the second cache quota; The second determining module is further configured to: determine whether the target file is already cached in the cache space based on the file identifier; if the target file is not cached in the cache space, determine the first occupied capacity of the cache space; compare the first occupied capacity with the first cache quota to obtain a first comparison result; if the first comparison result indicates that the first occupied capacity is less than the first cache quota, cache the target file in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the first cache quota and less than the second cache quota, add the file read / write request corresponding to the target file to a waiting queue to wait for the target file to be cached in the cache space; if the first comparison result indicates that the first occupied capacity is equal to or greater than the second cache quota, reject the file read / write request corresponding to the target file.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 4.

Images