Memory cache framework system for distributed file system client

Abstract

The invention relates to a memory cache framework system for a distributed file system client, which comprises a memory pool layer used for allocating a first memory space in a memory pool in response to a first request; the memory pool is divided into an idle list management memory space and a standby memory space; the data segment layer is used for linking a memory address corresponding to the file data cached in the memory pool through a double linked list to obtain a logic address corresponding to the file data; the file layer is used for responding to a file reading request, realizing a reading operation on a specified reading data segment by adding a reading lock to the specified reading data segment, and responding to a file writing request, realizing a writing operation on a specified writing data segment by adding a writing lock to the specified writing data segment; the monitoring layer is used for managing the memory use condition of the memory pool and monitoring the state of the data segment; according to the memory cache framework system, the data cache performance can be remarkably improved, the memory use pressure is reduced, and the data consistency and security are ensured.

Description

Technical Field

[0001] The present application relates to the field of distributed storage, and in particular to a memory cache framework system for a distributed file system client. Background Art

[0002] In a distributed file system, by caching the file data stored on the disk in the client's memory, the data access speed and file read and write efficiency can be improved. This caching method is called the memory cache of the distributed file system client. The existing open source distributed file system (such as CEPH) client memory cache framework has the following disadvantages: 1. The architecture design takes into account both block storage and file storage. Although this architecture design has better compatibility, there is a redundant hierarchical design for the cache of file data, which makes the data cache performance unable to achieve the best effect. For example, when responding to a small file read and write request of a few bytes, the data segment will be repeatedly split and merged, resulting in multiple invalid memory applications and memory releases, affecting the data cache performance. 2. Using a general memory management module for memory management, the general memory management module uses 4KB alignment to allocate memory (that is, memory can only be allocated in integer multiples of 4KB). When responding to a small file read and write request of a few bytes, memory amplification will occur (that is, the allocated memory is greater than the requested memory). When the data segment is large and split, it will also cause memory amplification. It is necessary to add memory release logic to avoid this phenomenon. However, adding memory release logic will cause cache performance to degrade, and it is impossible to accurately count the actual size of the used memory space. 3. Use a large range of read-write locks for protection. Only one file can be read or written at a time. It is impossible to read and write multiple files concurrently. The same file cannot be read concurrently. The performance is very poor for scenarios where multiple clients need to access multiple different files at the same time. Summary of the invention

[0003] In view of this, the present application proposes a memory cache framework system and a memory caching method for a distributed file system client, which can effectively control the total data cache size, realize rapid memory allocation, reduce memory usage pressure, improve data cache performance, and ensure data consistency and security.

[0004] According to one aspect of the present application, a memory cache framework system for a distributed file system client is provided, comprising: a memory pool layer, for allocating a first memory space in a memory pool in response to a first request; the memory pool is a memory space pre-applied to the distributed file system client; the first request is used to request allocation of a requested memory space required for caching first file data; the first memory space is greater than or equal to the requested memory space; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to a maximum fixed memory space, the first memory space is allocated in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, the first memory space is allocated in the spare memory space; A data segment layer, used to obtain the logical address corresponding to the file data by linking the memory address corresponding to the file data cached in the memory pool through a bidirectional linked list; the logical address is used to access the file data; a file layer, used to respond to a file read request, by adding a read lock for the specified read data segment, to implement a read operation on the specified read data segment, and respond to a file write request, by adding a write lock for the specified write data segment, to implement a write operation on the specified write data segment; the specified read data segment is the file data interval specified to be read in the file read request; the specified write data segment is the file data interval specified to be written in the file write request; a monitoring layer, used to manage the memory usage of the memory pool, and monitor the status of data segments; the data segment is the file data interval in the file; the status of the data segment includes a dirty data segment status and a clean data segment status.

[0005] In a possible implementation, the memory pool layer is further used to record unused fixed memory spaces and their memory addresses through a free list.

[0006] In a possible implementation, the memory pool layer is also used to: determine the size relationship between the requested memory space and the maximum fixed memory space in the free list; if the requested memory space is smaller than the maximum fixed memory space in the free list, determine whether there is a fixed memory space in the free list that is equal to the requested memory space; wherein, if there is a fixed memory space in the free list that is equal to the requested memory space, use the first fixed memory space in the free list as the first memory space, and delete the first fixed memory space and its memory address from the free list; the first fixed memory space is the fixed memory space in the free list that is equal to the requested memory space any fixed memory space in the free list; if there is no fixed memory space equal to the requested memory space in the free list, split the second fixed memory space in the free list into a first memory subspace and a second memory subspace, use the first memory subspace as the first memory space, delete the second fixed memory space and its memory address from the free list, and add the second memory subspace and its memory address to the free list; the second fixed memory space is any fixed memory space in the fixed memory space in the free list that is larger than the requested memory space; the first memory subspace is larger than or equal to the requested memory space; in the case where the first memory subspace is larger than the requested memory space, add the first memory subspace and its memory address to the free list; or, if there is no fixed memory space equal to the requested memory space in the free list, use the second fixed memory space as the first memory space; if the requested memory space is equal to the maximum fixed memory space in the free list, use the third fixed memory space in the free list as the first memory space, and delete the third fixed memory space and its memory address from the free list; the third fixed memory space is any maximum fixed memory space in the free list; if the requested memory space is larger than the maximum fixed memory space in the free list, determine whether the spare memory space is larger than the spare memory space. Whether there is an unallocated continuous memory space equal to the requested memory space; wherein, if there is an unallocated continuous memory space equal to the requested memory space in the spare memory space, the unallocated continuous memory space is used as the first memory space; if there is no unallocated continuous memory space equal to the requested memory space in the spare memory space, the requested memory space is split into two equal requested memory subspaces, each of the requested memory subspaces is regarded as the requested memory space, and the size relationship between the requested memory space and the maximum fixed memory space in the free list and subsequent steps are executed until the first memory space is allocated in the memory pool.

[0007] In a possible implementation, if the first memory space is larger than the requested memory space, a cursor is used to mark a starting position for caching the first file data in the first memory space.

[0008] In a possible implementation, the memory pool is divided into a plurality of minimum memory allocation units of equal size; and the memory pool layer is further used to mark the usage status of each of the minimum memory allocation units through a bitmap.

[0009] In a possible implementation, the memory pool layer is also used to: release the second memory space in the memory pool in response to the second request; determine whether there is an unallocated continuous memory space adjacent to the second memory space; if there is an unallocated continuous memory space adjacent to the second memory space, merge the second memory space and the unallocated continuous memory space adjacent to the second memory space to obtain a third memory space; determine whether the third memory space contains the intersection of the free list management memory space and the backup memory space; if the third memory space contains the intersection, release the third memory space to the backup memory space; if the third memory space does not contain the intersection, determine whether there is a fixed memory space in the free list that is equal to the third memory space. a fixed memory space; wherein, if there is a fixed memory space equal to the third memory space in the free list, the third memory space is released as a fixed memory space to the free list management memory space, and the third memory space and its memory address are added to the free list; if there is no fixed memory space equal to the third memory space in the free list, the third memory space is split into multiple third memory sub-spaces according to the size of the fixed memory space in the free list; wherein there is a fixed memory space equal to at least one third memory sub-space in the free list; each of the third memory sub-spaces is released as a fixed memory space to the free list management memory space, and each of the third memory sub-spaces and its memory address are added to the free list.

[0010] In a possible implementation, the file read request includes the specified read data segment and the specified incoming space; the file layer is further used to: add a read lock to the specified read data segment when receiving the file read request; determine whether all file data in the specified read data segment is cached in the memory pool; if all file data in the specified read data segment is not cached in the memory pool, add the missing data segment to a read-back queue; the missing data segment is a data segment to which file data in the specified read data segment that is not cached in the memory pool belongs; request the memory pool layer to allocate a memory space corresponding to the missing data segment; read the file data in the missing data segment from the disk into the memory space corresponding to the missing data segment allocated by the memory pool layer, so that all file data in the specified read data segment is cached in the memory pool, and delete the missing data segment from the read-back queue; if all file data in the specified read data segment is cached in the memory pool, fill all file data in the specified read data segment in the memory pool into the specified incoming space; release the read lock of the specified read data segment.

[0011] In a possible implementation, the file write request includes the specified write data segment, the write start position, the write length and the write data; the file layer is also used to: add a write lock for the specified write data segment when receiving the file write request; determine the write interval according to the specified write data segment, the write start position and the write length; judge whether the memory space corresponding to the write interval in the memory pool matches the memory space required by the write interval; if the memory space corresponding to the write interval does not match the memory space required by the write interval, request the memory pool layer to allocate memory space to make the memory space corresponding to the write interval match the memory space required by the write interval; if the memory space corresponding to the write interval matches the memory space required by the write interval, fill the write data into the memory space corresponding to the write interval, mark the status of the specified write data segment as a dirty data segment status, and add the specified write data segment to a dirty data queue; the dirty data queue is used to store data segments marked as dirty data segment status; release the write lock of the specified write data segment.

[0012] In a possible implementation, the file layer is further used to: upon receiving a dirty flush request, add a write lock to a specified dirty data segment in the dirty flush request; the specified dirty data segment is a data segment in the dirty data queue; the dirty flush request is used to implement flushing the file data in the specified dirty data segment to the disk for storage; determine whether there is a gap in the file data in the specified dirty data segment; if there is a gap in the file data in the specified dirty data segment, read the file data corresponding to the gap in the file data in the specified dirty data segment from the disk, and fill the gap in the file data in the specified dirty data segment; if there is no gap in the file data in the specified dirty data segment, determine whether there is a data segment in the flushing queue that has an overlapping interval with the specified dirty data segment; the flushing queue is used to store the data segment of the file data being flushed to the disk for storage; if there is a gap in the flushing queue that overlaps with the specified dirty data segment, If the file data in the data segment with overlapping intervals with the designated dirty data segment in the flushing queue is flushed, after the file data in the data segment with overlapping intervals with the designated dirty data segment in the flushing queue is flushed, the designated dirty data segment is added to the flushing queue, and the file data in the designated dirty data segment is flushed to the disk for storage; if there is no data segment with overlapping intervals with the designated dirty data segment in the flushing queue, the designated dirty data segment is directly added to the flushing queue, and the file data in the designated dirty data segment is flushed to the disk for storage; after the file data in the designated dirty data segment is flushed, the designated dirty data segment is deleted from the flushing queue and the dirty data queue, and the state of the designated dirty data segment is marked as the clean data segment state, and the designated dirty data segment is added to the clean data queue; the clean data queue is used to store the data segment marked as the clean data segment state; the write lock of the designated dirty data segment is released.

[0013] According to another aspect of the present application, a memory caching method for a distributed file system client is provided, which is applied to the above-mentioned memory caching framework system; the method comprises: in response to a first request, allocating a first memory space in a memory pool using a memory pool layer; the memory pool is a memory space applied for in advance to the distributed file system client; the first request is used to request allocation of a requested memory space required for caching first file data; the first memory space is greater than or equal to the requested memory space; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to the maximum fixed memory space, allocating the first memory space in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, allocating the first memory space in the spare memory space. A memory space; using a data segment layer to link the memory addresses corresponding to the file data cached in the memory pool through a bidirectional linked list to obtain a logical address corresponding to the file data; the logical address is used to access the file data; in response to a file read request, using the file layer to add a read lock to the specified read data segment to implement a read operation on the specified read data segment, and in response to a file write request, using the file layer to add a write lock to the specified write data segment to implement a write operation on the specified write data segment; the specified read data segment is the file data interval specified to be read in the file read request; the specified write data segment is the file data interval specified to be written in the file write request; using a monitoring layer to manage the memory usage of the memory pool and monitor the status of data segments; the data segment is the file data interval in the file; the status of the data segment includes a dirty data segment status and a clean data segment status.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned memory caching method for a distributed file system client or implement the functions of the above-mentioned memory caching framework system for a distributed file system client when executing the instructions stored in the memory.

[0015] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the above-mentioned memory caching method for a distributed file system client or implement the functions of the above-mentioned memory caching framework system for a distributed file system client.

[0016] According to another aspect of the present application, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned memory caching method for a distributed file system client or implements the functions of the above-mentioned memory caching framework system for a distributed file system client.

[0017] The memory cache framework system for distributed file system clients proposed in this application, the memory pool layer uses the memory pool to cache file data, divides the memory pool into two major areas: free list management memory space and spare memory space, and determines in which area to allocate memory space according to the size of the memory space requested to be allocated, which can effectively control the total data cache size and realize fast memory allocation; the data segment layer uses a bidirectional linked list to link the discrete memory addresses of file data, and can obtain the continuous logical address of file data to achieve efficient data access; the file layer adds a read lock to the data segment that needs to be read when reading the file, and adds a write lock to the data segment that needs to be written when writing the file, which can ensure the concurrent reading and serial writing of a single file, and is suitable for various complex file reading and writing scenarios; the monitoring layer monitors the memory usage status and data segment status of the memory pool, which can ensure that the cached data space does not exceed the maximum limit of the memory pool, and ensure that the use of the memory pool is within a reasonable and efficient range. The memory cache framework system for distributed file system clients of the embodiment of the present application can significantly improve data cache performance, reduce memory usage pressure, and ensure data consistency and security.

[0018] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 A schematic diagram of a memory cache framework system for a distributed file system client according to an embodiment of the present application is shown.

[0021] Figure 2 A schematic diagram showing a free list management memory space and a spare memory space according to an embodiment of the present application.

[0022] Figure 3 A schematic diagram of a free list according to an embodiment of the present application is shown.

[0023] Figure 4A schematic diagram showing file data access using a cursor according to an embodiment of the present application is shown.

[0024] Figure 5 A schematic diagram of a memory cache framework system for a distributed file system client according to an embodiment of the present application is shown.

[0025] Figure 6 A flowchart of a memory caching method for a distributed file system client according to an embodiment of the present application is shown.

[0026] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0029] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0030] As the scale and complexity of data processing continue to increase, efficient data caching systems play an increasingly important role in modern computing environments. However, existing data caching systems have problems such as low memory management efficiency, difficulty in ensuring data consistency, and obvious performance bottlenecks. These problems seriously restrict the improvement of the overall performance of the data caching system and increase the complexity of development and maintenance. For example, for distributed file system clients, their existing memory caching frameworks have problems such as low data caching performance, high memory usage pressure, difficulty in accurately counting the size of used memory space, and inability to read and write multiple files concurrently.

[0031] In view of this, the present application proposes a memory cache framework system for distributed file system clients, which can significantly improve data cache performance, reduce memory usage pressure, realize concurrent reading and serial writing of the same file, and ensure data consistency and security. Compared with the existing memory cache framework, the memory cache framework system of the present application can realize multiple small file read and write requests (such as a few bytes) sharing a minimum memory allocation unit through a cursor mechanism and bitmap management, which can solve the problem of memory amplification and cache performance degradation when the existing memory cache framework processes small file read and write requests; and the memory cache framework system of the present application adds read-write lock protection to different data segments, which is smaller than the existing memory cache framework that adds read-write lock protection to the entire file, and can improve the performance of file reading and writing; in addition, the size of files written and read by high-performance file systems is usually regular, and the memory cache framework system of the present application can achieve the ideal performance of memory pool allocation through pre-configuration or learning during operation.

[0032] Figure 1 A schematic diagram of a memory cache framework system for a distributed file system client according to an embodiment of the present application is shown as follows: Figure 1 As shown, the framework system is divided into four layers from bottom to top, namely, a memory pool layer 101, a data segment layer 102, a file layer 103 and a monitoring layer 104.

[0033] The memory pool layer 101 is used to allocate a first memory space in the memory pool in response to a first request; the memory pool is a memory space applied for in advance to the distributed file system client; the first request is used to request allocation of a requested memory space required for caching the first file data; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to the maximum fixed memory space, the first memory space is allocated in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, the first memory space is allocated in the spare memory space.

[0034] For example, a piece of memory space can be applied to the distributed file system client in advance as a memory pool. For example, if the memory of the distributed file system client is 4 GB, 4 GB of memory space can be applied at one time as a memory pool. The memory pool can be used to cache file data.

[0035] Exemplarily, the first request can be used to request the memory pool layer to allocate memory space to cache the first file data. The memory space required for caching the first file data is the requested memory space. For example, if the first file data to be cached is 4KB, the requested memory space is 4KB. The first memory space is the memory space actually allocated in the memory pool for caching the first file data. The first memory space can be equal to the requested memory space, or it can be larger than the requested memory space. If the first memory space is larger than the requested memory space, a cursor can be used to mark the starting position of caching the first file data in the first memory space.

[0036] In a possible implementation, the memory pool is divided into a plurality of minimum memory allocation units of equal size; and the memory pool layer is further used to mark the usage status of each of the minimum memory allocation units through a bitmap.

[0037] For example, since the operation of the file system is usually processed in pages, the default size of a page is 4KB, and 4KB can be used as the minimum memory allocation unit. The use status of each minimum memory allocation unit in the memory pool can be managed by a bitmap. The bitmap uses a bit array to represent the element status in a set, and the minimum memory allocation unit in the memory pool can be regarded as an element in the set, and the use status of each minimum memory allocation unit is marked by a bitmap. For example, each minimum memory allocation unit may correspond to a plurality of bits, wherein each bit is used to indicate the usage status of a fixed length of memory space (e.g., 16B) in the minimum memory allocation unit. If [0, 16B) in a minimum memory allocation unit has been allocated for use, the bit corresponding to [0, 16B) in the minimum memory allocation unit in the bitmap may be set to 1, indicating that [0, 16B) in the minimum memory allocation unit is in a used state; if [0, 16B) in a minimum memory allocation unit has not been allocated for use, the bit corresponding to [0, 16B) in the minimum memory allocation unit in the bitmap may be set to 0, indicating that [0, 16B) in the minimum memory allocation unit is in an unused state. In this way, by flattening the memory pool through the bitmap, the usage status of each minimum memory allocation unit in the memory pool can be intuitively obtained, and which memory spaces in each minimum memory allocation unit are used and which memory spaces are not used can be conveniently viewed, thereby conveniently and accurately counting the memory usage of the memory pool.

[0038] Exemplarily, the boundary point between the free list management memory space and the spare memory space may be dynamically moved, that is, the sizes of the free list management memory space and the spare memory space may be dynamically changed.

[0039] A person skilled in the art can pre-divide some fixed memory spaces (also referred to as fixed memory blocks) based on experience or actual needs. The fixed memory spaces can be divided from the lowest memory address of the memory pool to the high memory address. The size of the pre-divided fixed memory space can be an integer multiple of 4KB (i.e., the minimum memory allocation unit). These pre-divided fixed memory spaces constitute the initial free list management memory space. The remaining memory space outside the initial free list management memory space in the memory pool is the initial standby memory space. Since memory allocation is highly correlated with the business, the memory size allocated by the file system corresponding to different businesses when reading and writing files is usually regular. A person skilled in the art can pre-configure multiple fixed memory spaces according to the memory allocation characteristics of different businesses. For example, if a certain business needs to allocate more 8KB and 16KB memory space and less 32KB memory space, more 8KB and 16KB fixed memory spaces can be pre-configured, and less 32KB fixed memory space can be pre-configured. In this way, pre-configuring fixed memory space according to the memory allocation characteristics of different businesses can improve the memory allocation performance and memory usage efficiency of the memory pool.

[0040] The size of the divisible fixed memory space can be pre-specified, for example, the divisible fixed memory space is specified to be 4KB, 16KB, 64KB, 128KB, 256KB, 512KB, or 1024KB. When dividing the fixed memory space, it needs to be divided according to the specified size. The maximum divisible fixed memory space is 1024KB.

[0041] If the requested memory space is less than or equal to the maximum fixed memory space, you can find free (i.e., unallocated) fixed memory space in the initial free list management memory space for allocation; when the requested memory space cannot be allocated in the initial free list management memory space, you can continue to divide the fixed memory space and expand the free list management memory space.

[0042] If the requested memory space is larger than the maximum fixed memory space, or the free fixed memory space in the free list management memory space is insufficient and cannot be expanded, the required memory space can be allocated in the spare memory space. When allocating memory in the spare memory space, it can be allocated from the highest memory address of the memory pool to the lower memory address. The cursor can be used to mark the lowest memory address of the used memory space in the spare memory space, so that the memory usage of the spare memory space can be easily observed.

[0043] Figure 2 A schematic diagram showing a free list management memory space and a spare memory space according to an embodiment of the present application is shown as follows: Figure 2As shown in the figure, the memory space of the memory pool is 4GB, and the fixed memory space that can be divided can be specified as 4KB, 16KB, and 32KB. Starting from the lowest memory address of the memory pool, a 4KB fixed memory space, a 16KB fixed memory space, and a 32KB fixed memory space can be pre-divided as the initial free list management memory space, and the remaining memory space is the initial standby memory space. If the memory space requested to be allocated is 32KB, and the 32KB fixed memory space in the initial free list management memory space has been used, and the initial free list management memory space cannot be allocated to 32KB, then another 32KB fixed memory space can be divided to allocate the required 32KB. At this time, the free list management memory space is expanded, the standby memory space is reduced, and the intersection point between the free list management memory space and the standby memory space moves from the low memory address to the high memory address. When the intersection point moves to the highest memory address point it can reach, it means that the memory space in the memory pool is almost exhausted, and the memory allocation efficiency will be reduced. If a 1GB memory space needs to be allocated, and 1GB is larger than the maximum fixed memory space of 32KB, 1GB of memory space can be allocated from the highest memory address to the lower memory address in the spare memory space, and the lowest memory address of the used memory space in the spare memory space is marked with a spare memory cursor. If a 1GB memory space needs to be allocated next, it can continue to be allocated in the spare memory space and the spare memory cursor can be moved at the same time.

[0044] During the operation of the memory pool, the memory usage characteristics such as the usage frequency and memory usage size of memory spaces of different sizes in the memory pool can be learned and counted. The future operation and memory usage of the memory pool can be estimated based on the memory usage characteristics, so that the allocation of memory space can be dynamically adjusted to enable the memory pool to achieve a more ideal memory allocation performance.

[0045] In a possible implementation, the memory pool layer 101 is further used to record the unused fixed memory space and its memory address through a free list. The unused fixed memory space means that part of the fixed memory space is used, part of the fixed memory space is not used, or the fixed memory space is not used at all.

[0046] Figure 3 A schematic diagram of an idle list according to an embodiment of the present application is shown as follows: Figure 3As shown, assuming that the size of the fixed memory space includes 4KB, 8KB, 16KB, 32KB, 64KB, 128KB, 256KB, 384KB, 512KB, 640KB, 768KB, 896KB and 1024KB, each size of the fixed memory space can correspond to a list, and the list stores the memory addresses of the free fixed memory spaces of the corresponding size. For example, the list corresponding to the 4KB fixed memory space stores the memory addresses of all the free 4KB fixed memory spaces. The lists corresponding to the fixed memory spaces of different sizes constitute the entire free list. In the memory allocation process, the free fixed memory space can be preferentially searched in the free list to determine whether the free fixed memory space can be allocated with the required memory space. If the required memory space exceeds the largest fixed memory space in the free list, it is allocated in the spare memory space. In this way, the free fixed memory space is managed by the free list, and the memory address of the available fixed memory space can be conveniently and efficiently found in the memory allocation process, so that the rapid allocation of memory can be realized.

[0047] Exemplarily, when allocating memory in the free list management memory space, if the allocated fixed memory space is larger than the requested memory space, the unused memory space in the fixed memory space may continue to be allocated.

[0048] As an example, for a 16B file data that is smaller than the minimum fixed memory space (i.e., 4KB fixed memory space), a 4KB fixed memory space can be allocated for it, and the starting position of the 16B file data cached in the 4KB fixed memory space is marked by the first cursor, and the bit corresponding to the memory space used to cache the 16B file data in the 4KB fixed memory space in the bitmap is marked as used, and the memory address of the 4KB fixed memory space still exists in the free list. If it is necessary to continue to allocate memory space for a 32B file data, the memory address of the 4KB fixed memory space can be found through the free list, and the memory space that has been used in the 4KB fixed memory space can be known according to the bitmap and the first cursor. The 32B file data can be cached in the unused memory space in the 4KB fixed memory space, and the starting position of the 32B file data cached in the 4KB fixed memory space is marked by the second cursor, and at the same time, the bit corresponding to the memory space used to cache the 32B file data in the 4KB fixed memory space is marked as used in the bitmap. Similarly, if it is necessary to cache other smaller file data, you can continue to search for unused memory space in the 4KB fixed memory space. For each file data cached in the 4KB fixed memory space, use a cursor to mark the starting position of the file data cache, and mark the memory space used by the file data as used in the bitmap. If the memory space for caching the 16B file data and the 32B file data in the 4KB fixed memory space is released, the bit corresponding to the memory space for caching the 16B file data and the 32B file data is marked as unused. In this way, through the cursor mechanism and bitmap management, you can easily and intuitively view the used memory space and unused memory space in the fixed memory space, improve memory management efficiency, and realize caching multiple smaller file data in a minimum fixed memory space, improving data caching performance.

[0049] The following describes the process of memory allocation by the memory pool layer 101, which may include the following steps:

[0050] (1) Determine the size relationship between the requested memory space and the maximum fixed memory space in the free list.

[0051] (2) If the requested memory space is smaller than the maximum fixed memory space in the free list, determine whether there is a fixed memory space in the free list that is equal to the requested memory space.

[0052] (3) If there is a fixed memory space in the free list that is equal to the requested memory space, use the first fixed memory space in the free list as the first memory space, and delete the first fixed memory space and its memory address from the free list; the first fixed memory space is any fixed memory space in the free list that is equal to the requested memory space.

[0053] As an example, if the memory space requested for allocation is 16KB, the largest fixed memory space in the free list is 32KB, and there are multiple 16KB fixed memory spaces in the free list, one of the multiple 16KB fixed memory spaces in the free list is selected as the first memory space for allocation, the memory address of the allocated 16KB fixed memory space is the memory address of the allocated first memory space, and the memory address of the allocated 16KB fixed memory space is deleted from the free list. If there is only one 16KB fixed memory space in the free list, the 16KB fixed memory space is used as the first memory space and the memory address of the 16KB fixed memory space is deleted from the free list.

[0054] (4) If there is no fixed memory space in the free list that is equal to the requested memory space, split the second fixed memory space in the free list into a first memory subspace and a second memory subspace, use the first memory subspace as the first memory space, delete the second fixed memory space and its memory address from the free list, and add the second memory subspace and its memory address to the free list; the second fixed memory space is any fixed memory space in the free list that is larger than the requested memory space; the first memory subspace is larger than or equal to the requested memory space; when the first memory subspace is larger than the requested memory space, add the first memory subspace and its memory address to the free list; or, if there is no fixed memory space in the free list that is equal to the requested memory space, use the second fixed memory space as the first memory space.

[0055] Exemplarily, if the second fixed memory space can be split into a first memory subspace greater than or equal to the requested memory space and a second memory subspace not less than 4KB, the second fixed memory space is split and the split first memory subspace is used as the first memory space.

[0056] As an example, if the memory space requested for allocation is 12KB, which is not larger than the largest fixed memory space in the free list, but there is no fixed memory space in the free list that is exactly equal to 12KB, and there is a 16KB fixed memory space, then a 16KB fixed memory space can be selected and split into a 12KB memory subspace that is exactly equal to the requested memory space and a 4KB memory subspace. The split 12KB memory subspace is used as the first memory space, and the split 4KB memory subspace can be used as a new fixed memory space. At the same time, the memory address of the split 16KB fixed memory space is deleted from the free list, and the memory address of the split 4KB memory subspace is added to the free list.

[0057] As another example, if the memory space requested for allocation is 11KB, which is not larger than the largest fixed memory space in the free list, there is no fixed memory space exactly equal to 11KB in the free list, and there is a 16KB fixed memory space. The 16KB fixed memory space has the possibility of being split into a 12KB memory subspace and a 4KB memory subspace that are larger than the requested memory space, and the split 4KB memory subspace can be used for other memory allocation requests, then the 16KB fixed memory space can be split into a 12KB memory subspace and a 4KB memory subspace, and the split 12KB memory subspace is used as the first memory space, and the split 4KB memory subspace can be used as a new fixed memory space, and at the same time, the memory address of the split 16KB fixed memory space is deleted from the free list, and the memory address of the split 4KB memory subspace is added to the free list. Since only 11KB of the 12KB memory subspace is used and the remaining 1KB is not used, that is, the 12KB memory subspace is not fully used, the memory address of the 12KB memory subspace is added to the free list, a cursor is used to mark the starting position of the cache of the 11KB file data in the 12KB memory subspace, and the bit corresponding to the memory space used by the 11KB file data in the bitmap is marked as used, and the unused 1KB of the 12KB memory subspace can still be allocated.

[0058] Exemplarily, if the second fixed memory space cannot be split into a first memory subspace that is greater than or equal to the requested memory space and a second memory subspace that is not less than 4 KB, the second fixed memory space is used as the first memory space.

[0059] As an example, if the memory space requested for allocation is 2KB, and the smallest fixed memory space in the free list is 4KB, and the 4KB fixed memory space cannot be split any further, the 4KB fixed memory space is used as the first memory space. Since only 2KB of the 4KB fixed memory space is used, and the remaining 2KB is not used, the memory address of the 4KB fixed memory space is still in the free list. At this time, a cursor is used to mark the starting position of the 2KB file data cached in the 4KB fixed memory space, and the bit corresponding to the memory space used by the 2KB file data in the bitmap is marked as used, and the unused memory space in the 4KB fixed memory space can continue to be allocated.

[0060] (5) If the requested memory space is equal to the maximum fixed memory space in the free list, use the third fixed memory space in the free list as the first memory space, and delete the third fixed memory space and its memory address from the free list; the third fixed memory space is any maximum fixed memory space in the free list.

[0061] As an example, if the memory space requested for allocation is 32KB, the largest fixed memory space in the free list is also 32KB, and there are multiple 32KB fixed memory spaces in the free list, one of the multiple 32KB fixed memory spaces in the free list is selected as the first memory space for allocation, the memory address of the allocated 32KB fixed memory space is the memory address of the allocated first memory space, and the memory address of the allocated 32KB fixed memory space is deleted from the free list. If there is only one 32KB fixed memory space in the free list, the 32KB fixed memory space is used as the first memory space and the memory address of the 32KB fixed memory space is deleted from the free list.

[0062] (6) If the requested memory space is larger than the maximum fixed memory space in the free list, determine whether there is unallocated continuous memory space in the spare memory space that is equal to the requested memory space.

[0063] (7) If there is unallocated continuous memory space in the spare memory space that is equal to the requested memory space, the unallocated continuous memory space is used as the first memory space.

[0064] As an example, if the requested memory space is 1 GB, which is larger than the maximum fixed memory space in the free list, and there is unallocated 1 GB of continuous memory space in the spare memory space, 1 GB of continuous memory space is allocated from the spare memory space as the first memory space.

[0065] (8) If there is no unallocated continuous memory space equal to the requested memory space in the spare memory space, split the requested memory space into two equal requested memory sub-spaces, regard each of the requested memory sub-spaces as the requested memory space, and execute the steps of determining the size relationship between the requested memory space and the maximum fixed memory space in the free list and subsequent steps until the first memory space is allocated in the memory pool.

[0066] For example, if the free list management memory space and the spare memory space cannot allocate continuous requested memory space, the requested memory space can be halved, and then the above memory allocation method is executed on the halved requested memory subspaces respectively, until the required memory space is allocated in the memory pool. This memory allocation strategy can be called a recursive halving strategy. In this way, when continuous requested memory space cannot be allocated, by adopting a recursive halving strategy, discrete memory space can be allocated in the memory pool as the first memory space, thereby improving the memory usage efficiency of the memory pool and reducing the memory usage pressure.

[0067] In a possible implementation, the memory pool layer 101 is further configured to: release a second memory space in the memory pool in response to a second request.

[0068] When the allocated memory space is no longer needed, the memory pool layer can be requested to release the memory space, and the released memory space can be reallocated to the free list management space or the spare memory space. The following describes the process of memory release by the memory pool layer 101, which may include the following steps:

[0069] (1) Determine whether there is an unallocated continuous memory space adjacent to the second memory space.

[0070] (2) If there is an unallocated continuous memory space adjacent to the second memory space, merge the second memory space and the unallocated continuous memory space adjacent to the second memory space to obtain a third memory space.

[0071] (3) Determine whether the third memory space includes a boundary point between the free list management memory space and the backup memory space.

[0072] (4) If the third memory space includes the intersection point, release the third memory space to the spare memory space.

[0073] (5) If the third memory space does not include the intersection point, determine whether there is a fixed memory space in the free list that is equal to the third memory space.

[0074] (6) If there is a fixed memory space in the free list that is equal to the third memory space, the third memory space is released as a fixed memory space to the free list management memory space, and the third memory space and its memory address are added to the free list.

[0075] (7) If there is no fixed memory space equal to the third memory space in the free list, split the third memory space into multiple third memory sub-spaces according to the size of the fixed memory space in the free list; wherein there is a fixed memory space equal to at least one of the third memory sub-spaces in the free list; each of the third memory sub-spaces is released as a fixed memory space to the free list management memory space, and each of the third memory sub-spaces and its memory address are added to the free list.

[0076] As an example, if the memory space requested to be released is 128KB, check whether there is adjacent free continuous memory space on the left and right sides of the space requested to be released. Assuming that there is a total of 64KB of adjacent free continuous memory space on the left and right sides, the 128KB requested to be released and the adjacent free 64KB are merged to obtain a 192KB continuous memory space. If the 192KB continuous memory space contains the intersection of the free list management memory space and the spare memory space, the 192KB continuous memory space is released to the spare memory space. If the 192KB continuous memory space does not contain the intersection point of the free list management memory space and the spare memory space, and there is a 192KB fixed memory space in the free list, then the 192KB continuous memory space is released as a new fixed memory space to the free list management memory space, and the memory address of the 192KB continuous memory space is added to the free list; if there is no 192KB fixed memory space in the free list, search for a fixed memory space smaller than 192KB in the free list, assuming that the size of the largest fixed memory space smaller than 192KB in the free list is xKB, then split the 192KB continuous memory space into a memory subspace of xKB and a memory subspace of (192-x)KB, release the split xKB memory subspace as a new fixed memory space to the free list management memory space, and add the memory address of the 192KB continuous memory space to the free list management memory space The memory address of the memory subspace is added to the free list, and then it is determined whether there is a fixed memory space of size (192-x)KB in the free list. If so, the split memory subspace of (192-x)KB is released as a new fixed memory space to the free list management memory space and its memory address is added to the free list. If there is no fixed memory space of size (192-x)KB in the free list, the memory subspace of (192-x)KB is further split into memory subspaces of the same size as the fixed memory space in the free list and the remaining memory subspaces. The above determination and release operations are continued until no further splitting is possible. The last remaining split memory subspace is released to the free list management memory space and its memory address is added to the free list. In this way, the 192KB continuous memory space can be split into multiple fixed memory spaces and released to the free list management memory space. For example, if there are 128KB fixed memory space and 32KB fixed memory space in the free list, the 192KB continuous memory space can be split into one 128KB memory subspace and two 32KB memory subspaces, and the split 128KB memory subspace and 32KB memory subspace are released as fixed memory space to the free list management memory space and the memory address is added to the free list.

[0077] The memory pool layer of the memory cache framework system of the embodiment of the present application uses the memory pool to cache file data, divides the memory pool into two major areas: free list management memory space and backup memory space, and adopts a customized proprietary algorithm to manage the memory pool. This can effectively control the total data cache size, avoid uncontrollable total memory size, and achieve rapid allocation and release of memory. It can effectively balance memory usage efficiency and allocation speed, is suitable for memory management needs of various scales, and can improve data cache performance.

[0078] The data segment layer 102 is used to link the memory addresses corresponding to the file data cached in the memory pool through a bidirectional linked list to obtain the logical addresses corresponding to the file data; the logical addresses are used to access the file data.

[0079] When file data is cached in memory, the memory address assigned to it (i.e., the memory address of the memory space that caches the file data) is a physical address, which corresponds to the actual storage location of the file data in memory. When users access file data in memory, they need to access it through logical addresses. Logical addresses refer to the addresses of memory units seen from the perspective of applications. The logical addresses of file data can be mapped to the physical addresses of file data, that is, they can be mapped to the actual storage location of file data in memory, thereby enabling access to file data.

[0080] The data segment layer 102 can link the physical addresses of the file data through a bidirectional linked list to obtain the continuous logical addresses of the file data. When performing memory allocation, if the memory space of the memory pool is close to the usage limit, the memory pool cannot allocate the required continuous memory space at one time, and the file data may be allocated to a discrete memory space for caching. At this time, the physical address of the file data is discrete. The data segment layer 102 can link the discrete physical addresses through a bidirectional linked list to obtain a continuous logical address, so that the difference caused by the discrete physical address can be shielded from the upper layer, and the actually discrete physical address can be normally accessed through the continuous logical address of the file data.

[0081] Exemplarily, a cursor may be used to access and locate file data. Figure 4 A schematic diagram of using a cursor to access file data according to an embodiment of the present application is shown. Assuming that a request is made to access data segment 1 in file 1 (i.e., a file data interval in file 1), the logical address of data segment 1 can be mapped to the physical address of data segment 1 in the memory, such as Figure 4As shown, the file data corresponding to data segment 1 is cached in four memory spaces, and the memory addresses are addr1, addr2, addr3, and addr4 respectively. The four memory addresses are linked through a bidirectional linked list. According to the specific data to be accessed in the access request, the physical address of the memory space where the data is located can be found through the cursor and the offset address corresponding to the cache position of the data in the memory space can be located. According to the access needs, the cursor can be moved left and right to the specified position in the memory space corresponding to data segment 1, thereby realizing the access to the file data in data segment 1. By using the cursor mechanism, efficient data access and positioning can be achieved.

[0082] The file layer 103 is used to respond to a file read request by adding a read lock to the specified read data segment to implement a read operation on the specified read data segment, and to respond to a file write request by adding a write lock to the specified write data segment to implement a write operation on the specified write data segment; the specified read data segment is the file data interval specified to be read in the file read request; the specified write data segment is the file data interval specified to be written in the file write request.

[0083] The file layer 103 is mainly used to control the read and write requests of the file, and realizes the concurrent control of the file read and write requests by introducing the read-write lock mechanism. Since there are few actual application scenarios in which a file is written by multiple users at the same time, the concurrent reading and serial writing of the file can be realized by adding a read-write lock to the data segment specified for reading and writing in the file read and write request (that is, a single file can be read by multiple users at the same time, but can only be written by one user at a time), so that when the file read and write requests overlap, sorting processing can be performed to ensure data consistency. The file read and write request actually reads and writes different data segments in the file (that is, different file data intervals in the file), and different data segments have different states. The file layer 103 can also manage different data segments.

[0084] The file layer 103 can implement operations such as reading, writing, flushing, and eliminating clean data of file data. The file reading operation requires reading the specified data segment cached in the memory and returning the read data to the input space specified by the upper layer application. The file reading request may include the specified read data segment and the specified input space. The file layer 103 may perform the file reading operation by the following steps:

[0085] (1) When receiving the file read request, adding a read lock for the specified read data segment.

[0086] (2) Determine whether all file data in the specified read data segment is cached in the memory pool.

[0087] (3) If all the file data in the designated read data segment is not cached in the memory pool, the missing data segment is added to the read-back queue; the missing data segment is a data segment in the designated read data segment to which the file data that is not cached in the memory pool belongs.

[0088] If all file data in the specified read data segment is not cached in the memory pool, it is called a cache miss; if some file data in the specified read data segment is cached in the memory pool, and some file data is not cached in the memory pool, it is called a cache partial hit; if all file data in the specified read data segment is cached in the memory pool, it is called a cache hit. In the case of a cache miss or a cache partial hit, the data segment to which the file data in the specified read data segment that is not cached in the memory pool (i.e., the missing data segment) belongs is added to the readback queue.

[0089] (4) Requesting the memory pool layer to allocate memory space corresponding to the missing data segment.

[0090] If there is a vacant data segment, a request is made to the memory pool layer 101 to allocate a memory space corresponding to the vacant data segment in the memory pool for caching the file data in the vacant data segment.

[0091] (5) Reading the file data in the missing data segment from the disk into the memory space corresponding to the missing data segment allocated by the memory pool layer, so that all the file data in the specified read data segment are cached in the memory pool, and deleting the missing data segment from the read-back queue.

[0092] (6) When all the file data in the designated read data segment is cached in the memory pool, all the file data in the designated read data segment in the memory pool is filled into the designated incoming space.

[0093] As an example, if the specified read data segments are data segments 1 and data segments 2 in file 1, and only the data of data segment 1 is cached in the memory pool, and the data of data segment 2 is not cached, then data segment 2 is added to the read-back queue, and a memory space corresponding to the data length of data segment 2 is allocated in the memory pool. The data of data segment 2 is read back from the disk to the memory space corresponding to data segment 2 allocated in the memory pool, and data segment 2 is deleted from the read-back queue. At this time, all file data in the specified read data segment is cached in the memory pool, and all file data in the specified read data segment cached in the memory pool is filled into the incoming space specified by the upper-layer application.

[0094] As another example, if the specified read data segments are data segments 1 and data segments 2 in file 1, and all the data of data segments 1 and data segments 2 are cached in the memory pool (i.e., a cache hit), then all the file data in the specified read data segments cached in the memory pool are directly filled into the incoming space specified by the upper-layer application.

[0095] (7) Release the read lock of the specified read data segment.

[0096] This completes the operation of reading the file data in the specified read data segment into the specified input space.

[0097] The file write operation needs to write the data passed in by the upper layer application to the specified location in the specified data segment of the memory cache. When the write operation is completed in the memory and the data segment written in the memory has not been flushed to the disk for storage, the data segment cached in the memory is inconsistent with the data in the data segment stored on the disk. At this time, the data segment to be written needs to be marked as a dirty data segment state. The file write request may include the specified write data segment, the write start position, the write length and the write data. The file layer 103 may perform the file write operation by the following steps:

[0098] (1) When receiving the file write request, adding a write lock for the designated write data segment.

[0099] (2) Determine a write interval according to the designated write data segment, the write start position and the write length.

[0100] The write interval is an interval in the specified write data segment starting from the write start position and corresponding to the write length.

[0101] (3) Determine whether the memory space corresponding to the write interval in the memory pool matches the memory space required by the write interval.

[0102] (4) If the memory space corresponding to the write interval does not match the memory space required by the write interval, request the memory pool layer to allocate memory space so that the memory space corresponding to the write interval matches the memory space required by the write interval.

[0103] Since the address of the write interval in the file write request is a logical address, it may not match the memory space required for the write interval after being mapped to the actual memory space in the memory pool. For example, if a request is made to write 4KB of file data at the logical address [0,4KB], but the actual memory space mapped to the logical address in the memory pool is only 2KB, then it is necessary to request to allocate another 2KB of memory space in the memory pool so that the memory space corresponding to the write interval matches the 4KB memory space required for the write interval.

[0104] (5) When the memory space corresponding to the write interval matches the memory space required for the write interval, the write data is filled into the memory space corresponding to the write interval, and the status of the specified write data segment is marked as a dirty data segment status, and the specified write data segment is added to a dirty data queue; the dirty data queue is used to store data segments marked as dirty data segment status.

[0105] When the memory space required for the write interval is allocated in the memory pool, the data to be written is filled into these memory spaces. If the memory spaces corresponding to different data segments are connected end to end after the write operation, these data segments can be merged according to actual needs. The written data segment is marked as a dirty data segment before it is flushed to the disk, indicating that the data in the data segment is dirty data and inconsistent with the data stored on the disk.

[0106] (6) Release the write lock of the specified write data segment.

[0107] This completes the operation of writing the write data to the specified location of the specified write data segment.

[0108] The process of flushing the data in the data segment written in the memory to the disk for storage is called dirty flushing. Dirty flushing is mainly triggered in two scenarios. One is that the upper-layer application actively calls the dirty flushing interface to flush the dirty data to the disk, and the other is that the internal dirty flushing thread periodically flushes the data in the dirty data queue to the disk. After being flushed to the disk, the data is safe and will not be lost after a power outage. The file layer 104 can include the following steps to perform dirty flushing:

[0109] (1) When a dirty flush request is received, a write lock is added to a specified dirty data segment in the dirty flush request; the specified dirty data segment is a data segment in the dirty data queue; the dirty flush request is used to flush the file data in the specified dirty data segment to the disk for storage.

[0110] (2) Determine whether there is any gap in the file data in the specified dirty data segment.

[0111] (3) If there are gaps in the file data in the designated dirty data segment, read the file data corresponding to the gaps in the file data in the designated dirty data segment from the disk to fill the gaps in the file data in the designated dirty data segment.

[0112] Since the minimum memory allocation unit of the memory pool is 4KB, the minimum unit of the data segment is also 4KB. When flushing the dirty data segment, the minimum size of the dirty data segment needs to be 4KB. If the first write operation is to rewrite the interval with the memory address [0,1KB), and the second write operation is to rewrite the interval with the memory address [2KB,3KB), then in the data segment with the memory address [0,4KB), the data with the memory addresses [1KB,2KB) and [3KB,4KB) are missing, that is, there is a hole in the data segment. If the data segment needs to be flushed, the data of the data segment needs to be read back from the disk first, and the empty intervals with the memory addresses [1KB,2KB) and [3KB,4KB) need to be filled. After eliminating the hole in the data segment, the data segment can be flushed to the disk. In the actual dirty flushing process, it can be ensured that the length of all data segments with holes is 4KB. In this way, when flushing data segments with holes, there is no need to read back long data from the disk, thereby improving the dirty flushing performance.

[0113] (4) When there is no gap in the file data within the specified dirty data segment, determine whether there is a data segment in the flushing queue that overlaps with the specified dirty data segment; the flushing queue is used to store the data segment of the file data being flushed to the disk for storage.

[0114] (5) If there is a data segment in the flushing queue that has an overlapping interval with the specified dirty data segment, after the file data in the data segment in the flushing queue that has an overlapping interval with the specified dirty data segment is flushed, the specified dirty data segment is added to the flushing queue, and the file data in the specified dirty data segment is flushed to the disk for storage.

[0115] The dirty data segments being flushed to disk are stored in the flushing queue. If the specified dirty data segment is segment 1, and segment 2 exists in the flushing queue, and there is an overlapping data interval between segment 2 and segment 1, it is necessary to wait until segment 2 is flushed before triggering the flush of segment 1.

[0116] (6) If there is no data segment in the flushing queue that overlaps with the specified dirty data segment, the specified dirty data segment is directly added to the flushing queue, and the file data in the specified dirty data segment is flushed to the disk for storage.

[0117] (7) After the file data in the specified dirty data segment is flushed, the specified dirty data segment is deleted from the flushing queue and the dirty data queue, and the state of the specified dirty data segment is marked as a clean data segment state, and the specified dirty data segment is added to a clean data queue; the clean data queue is used to store data segments marked as clean data segment states.

[0118] (8) Release the write lock of the specified dirty data segment.

[0119] In this way, the operation of flushing the file data in the specified dirty data segment to the disk for storage is completed. The file layer 104 can handle special scenarios such as concurrent flushing and hole writing through the above process to ensure data consistency and flushing efficiency. The dirty data in the memory becomes clean data after being flushed to the disk. Clean data means that the data cached in the memory is consistent with the data stored in the disk. Clean data can be deleted at any time to release memory space.

[0120] It should be noted that dirty data segments cannot be written directly. If the specified write data segment in the file write request contains dirty data, the dirty flush operation will be triggered first. The dirty data in the specified write data segment will be flushed to the disk to become clean data before the file write operation is performed.

[0121] The file layer of the memory cache framework system of the embodiment of the present application adds a read lock to the data segment that needs to be read when reading the file, adds a write lock to the data segment that needs to be written when writing the file, and adds a write lock to the dirty data segment that needs to be flushed when flushing dirty data. Compared with the existing memory cache framework system that adds a read-write lock to the entire file when performing read and write operations, the scope of adding read-write locks is smaller, which can improve the read and write performance and ensure concurrent reading and serial writing of a single file. Moreover, through sophisticated lock control and data segment management, it can effectively balance concurrency performance and data security, ensure data consistency, and is suitable for various complex file reading and writing scenarios.

[0122] The monitoring layer 104 is used to manage the memory usage of the memory pool and monitor the status of data segments; the data segment is a file data interval within a file; the status of the data segment includes a dirty data segment status and a clean data segment status.

[0123] The monitoring layer 104 can manage the memory usage status of the files and memory pools provided by the file layer, count the status of all data segments of different files, record the size of all data segments in different states, etc., and can periodically trigger the dirty data segment queue's dirty flushing operation and the clean data segment queue's clean data segment elimination operation to flush the dirty data to the disk for storage to ensure that the dirty data does not exceed the set threshold, and eliminate the clean data at the same time to ensure that the cached data space does not exceed the maximum limit of the memory pool, so that the memory pool can work in a range with the highest allocation efficiency.

[0124] Figure 5 A schematic diagram of a memory cache framework system for a distributed file system client according to an embodiment of the present application is shown as follows: Figure 5As shown, the memory cache framework system of the embodiment of the present application is respectively a memory pool layer, a data segment layer, a file layer and a monitoring layer from the bottom to the top, and each layer provides a unique solution for a specific problem. The memory pool layer adopts a customized proprietary algorithm, flattens the memory pool through a bitmap, and divides the memory pool into two major areas: free list management memory space and spare memory space, which can realize the rapid allocation and release of file data cache and effectively control the size of the total data cache; the data segment layer links the discrete physical addresses of file data through a bidirectional linked list, and can use the cursor mechanism to achieve efficient data access and positioning; the file layer introduces a read-write lock mechanism to achieve concurrent control of file read and write requests, including core logic such as reading, writing, and flushing dirty, which can effectively handle cache hits and cache misses, and ensure data consistency; the monitoring layer can manage the use of files and memory pools provided by the file layer, monitor the status of dirty data segments in the dirty data queue and clean data segments in the clean data queue, and can trigger dirty data refresh and clean data elimination at regular intervals to ensure that the use of the memory pool is within a reasonable and efficient range. The security of file data cached in memory can be guaranteed by upper-level applications. For example, use the text editor Vim application to open an existing file, edit the file content, save with the wq command and exit. In this process, the Vim application first calls the memory cache read interface to read the data of the file, then calls the memory cache write interface to write the data of the file, and finally calls the memory cache dirty flush interface to flush the dirty data segment in the memory to the disk for storage. After the dirty flush interface returns successfully, the edited dirty data is actually written to the disk. At this time, the edited file data is safe. When the file is opened again, the previously edited content can be read again. The innovative multi-level memory cache framework system proposed in the embodiment of the present application can significantly improve data caching performance, reduce memory usage pressure, and ensure data consistency and security; at the same time, the flexible design of the memory cache framework system enables it to adapt to various complex data caching scenarios, and can provide reliable cache support for high-performance computing and large-scale data processing.

[0125] This application also proposes a memory caching method for a distributed file system client.

[0126] Figure 6 A memory caching method for a distributed file system client according to an embodiment of the present application is shown. The method can be applied to the memory caching framework system for a distributed file system client proposed in the present application, such as Figure 6 As shown, the method may include:

[0127] S601. In response to a first request, allocate a first memory space in a memory pool using the memory pool layer 101; the memory pool is a memory space pre-applied to the distributed file system client; the first request is used to request allocation of a requested memory space required for caching first file data; the first memory space is greater than or equal to the requested memory space; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to the maximum fixed memory space, allocate the first memory space in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, allocate the first memory space in the spare memory space.

[0128] S602: Use the data segment layer 102 to link the memory addresses corresponding to the file data cached in the memory pool through a bidirectional linked list to obtain a logical address corresponding to the file data; the logical address is used to access the file data.

[0129] S603. In response to a file read request, the file layer 103 is used to add a read lock to the specified read data segment to implement a read operation on the specified read data segment, and in response to a file write request, the file layer 103 is used to add a write lock to the specified write data segment to implement a write operation on the specified write data segment; the specified read data segment is the file data interval specified to be read in the file read request; the specified write data segment is the file data interval specified to be written in the file write request.

[0130] S604. Use the monitoring layer 104 to manage the memory usage of the memory pool and monitor the status of the data segment; the data segment is a file data interval in the file; the status of the data segment includes a dirty data segment status and a clean data segment status.

[0131] The specific implementation of steps S601 to S604 can refer to the description of the above-mentioned memory cache framework system embodiment, which will not be repeated here.

[0132] The memory caching method for a distributed file system client in the embodiment of the present application can significantly improve data caching performance, reduce memory usage pressure, realize concurrent reading and serial writing of the same file, and ensure data consistency and security.

[0133] An embodiment of the present application also proposes an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned memory caching method for a distributed file system client or implement the functions of the above-mentioned memory caching framework system for a distributed file system client when executing the instructions stored in the memory.

[0134] Figure 7 1 is a block diagram of an electronic device 1900 according to an embodiment of the present application. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-mentioned memory caching method for a distributed file system client or to implement the above-mentioned memory caching framework system for a distributed file system client.

[0135] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0136] The present application also provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the computer-readable storage medium implements the memory caching method for a distributed file system client or implements the functions of the memory caching framework system for a distributed file system client. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0137] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by a processing component 1922 of an electronic device 1900 to complete the above-mentioned memory caching method for a distributed file system client or to implement the functions of the above-mentioned memory caching framework system for a distributed file system client.

[0138] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned memory caching method for a distributed file system client or implements the functions of the above-mentioned memory caching framework system for a distributed file system client.

[0139] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0140] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0141] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0142] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0143] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0144] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0145] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0146] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0147] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A memory cache framework system for a distributed file system client, characterized in that: include: The memory pool layer is used to allocate a first memory space in the memory pool in response to the first request; The memory pool is a memory space applied for in advance to the distributed file system client; The first request is used to request allocation of requested memory space required for caching the first file data; The first memory space is greater than or equal to the requested memory space; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to the maximum fixed memory space, the first memory space is allocated in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, the first memory space is allocated in the spare memory space; The data segment layer is used to link the memory addresses corresponding to the file data cached in the memory pool through a bidirectional linked list to obtain the logical addresses corresponding to the file data; the logical addresses are used to access the file data; The file layer is used to implement a read operation on a specified read data segment by adding a read lock to the specified read data segment in response to a file read request, and implement a write operation on the specified write data segment by adding a write lock to the specified write data segment in response to a file write request; the specified read data segment is a file data interval specified to be read in the file read request; the specified write data segment is a file data interval specified to be written in the file write request; The monitoring layer is used to manage the memory usage of the memory pool and monitor the status of data segments; the data segment is a file data interval within a file; the status of the data segment includes a dirty data segment status and a clean data segment status.

2. The memory cache framework system according to claim 1, characterized in that: The memory pool layer is further used to record unused fixed memory spaces and their memory addresses through a free list.

3. The memory cache framework system according to claim 2, characterized in that: The memory pool layer is also used for: Determine the size relationship between the requested memory space and the maximum fixed memory space in the free list; If the requested memory space is smaller than the maximum fixed memory space in the free list, determine whether there is a fixed memory space in the free list that is equal to the requested memory space; wherein, If there is a fixed memory space in the free list that is equal to the requested memory space, taking the first fixed memory space in the free list as the first memory space, and deleting the first fixed memory space and its memory address from the free list; the first fixed memory space is any fixed memory space in the fixed memory space in the free list that is equal to the requested memory space; If there is no fixed memory space equal to the requested memory space in the free list, split the second fixed memory space in the free list into a first memory subspace and a second memory subspace, use the first memory subspace as the first memory space, delete the second fixed memory space and its memory address from the free list, and add the second memory subspace and its memory address to the free list; the second fixed memory space is any fixed memory space in the free list that is larger than the requested memory space, and the first memory subspace is larger than or equal to the requested memory space; in the case where the first memory subspace is larger than the requested memory space, add the first memory subspace and its memory address to the free list; or, if there is no fixed memory space equal to the requested memory space in the free list, use the second fixed memory space as the first memory space; If the requested memory space is equal to the maximum fixed memory space in the free list, use the third fixed memory space in the free list as the first memory space, and delete the third fixed memory space and its memory address from the free list; the third fixed memory space is any maximum fixed memory space in the free list; If the requested memory space is larger than the maximum fixed memory space in the free list, determine whether there is unallocated continuous memory space in the spare memory space that is equal to the requested memory space; wherein, If there is unallocated continuous memory space in the spare memory space that is equal to the requested memory space, use the unallocated continuous memory space as the first memory space; If there is no unallocated continuous memory space equal to the requested memory space in the spare memory space, the requested memory space is split into two equal requested memory sub-spaces, each of the requested memory sub-spaces is regarded as the requested memory space, and the size relationship between the requested memory space and the maximum fixed memory space in the free list is determined, and subsequent steps are performed until the first memory space is allocated in the memory pool.

4. The memory cache framework system according to claim 1, characterized in that: If the first memory space is larger than the requested memory space, a cursor is used to mark a starting position of caching the first file data in the first memory space.

5. The memory cache framework system according to claim 1, characterized in that: The memory pool is divided into a plurality of minimum memory allocation units of equal size; the memory pool layer is further used for: The usage status of each of the minimum memory allocation units is marked by a bitmap.

6. The memory cache framework system according to claim 2, characterized in that: The memory pool layer is also used for: In response to the second request, releasing a second memory space in the memory pool; Determining whether there is an unallocated continuous memory space adjacent to the second memory space; If there is an unallocated continuous memory space adjacent to the second memory space, merging the second memory space with the unallocated continuous memory space adjacent to the second memory space to obtain a third memory space; Determining whether the third memory space includes a junction point between the free list management memory space and the standby memory space; If the third memory space contains the intersection point, releasing the third memory space to the spare memory space; If the third memory space does not contain the intersection point, determine whether there is a fixed memory space in the free list that is equal to the third memory space; wherein, If there is a fixed memory space equal to the third memory space in the free list, releasing the third memory space as a fixed memory space to the free list management memory space, and adding the third memory space and its memory address to the free list; If there is no fixed memory space equal to the third memory space in the free list, the third memory space is split into multiple third memory sub-spaces according to the size of the fixed memory space in the free list; wherein, there is a fixed memory space equal to at least one of the third memory sub-spaces in the free list; each of the third memory sub-spaces is released as a fixed memory space to the free list management memory space, and each of the third memory sub-spaces and its memory address are added to the free list.

7. The memory cache framework system according to claim 1, characterized in that: The file read request includes the specified read data segment and the specified incoming space; The file layer is also used for: When receiving the file read request, adding a read lock for the specified read data segment; Determine whether all file data in the specified read data segment is cached in the memory pool; If the memory pool does not cache all the file data in the specified read data segment, adding the vacant data segment to the read-back queue; The vacant data segment is a data segment in the designated read data segment to which file data that is not cached in the memory pool belongs; Requesting the memory pool layer to allocate memory space corresponding to the vacant data segment; Read the file data in the missing data segment from the disk into the memory space corresponding to the missing data segment allocated by the memory pool layer, so that all the file data in the specified read data segment is cached in the memory pool, and delete the missing data segment from the read-back queue; In the case that all the file data in the designated read data segment is cached in the memory pool, filling all the file data in the designated read data segment in the memory pool into the designated incoming space; Release the read lock of the specified read data segment.

8. The memory cache framework system according to claim 1, characterized in that: The file write request includes the specified write data segment, write start position, write length and write data; The file layer is also used for: When receiving the file write request, adding a write lock for the specified write data segment; Determining a writing interval according to the designated writing data segment, the writing starting position and the writing length; Determine whether the memory space corresponding to the write interval in the memory pool matches the memory space required by the write interval; If the memory space corresponding to the write interval does not match the memory space required by the write interval, request the memory pool layer to allocate memory space so that the memory space corresponding to the write interval matches the memory space required by the write interval; In the case where the memory space corresponding to the write interval matches the memory space required by the write interval, the write data is filled into the memory space corresponding to the write interval, and the state of the designated write data segment is marked as a dirty data segment state, and the designated write data segment is added to a dirty data queue; the dirty data queue is used to store data segments marked as dirty data segment states; Release the write lock of the specified write data segment.

9. The memory cache framework system according to claim 8, characterized in that: The file layer is also used for: When a dirty flush request is received, a write lock is added to a specified dirty data segment in the dirty flush request; the specified dirty data segment is a data segment in the dirty data queue; the dirty flush request is used to flush the file data in the specified dirty data segment to a disk for storage; Determine whether there is a gap in the file data in the specified dirty data segment; If there are gaps in the file data in the specified dirty data segment, read the file data corresponding to the gaps in the file data in the specified dirty data segment from the disk, and fill the gaps in the file data in the specified dirty data segment; In the case that there is no gap in the file data in the designated dirty data segment, determining whether there is a data segment in the flushing queue that has an overlapping interval with the designated dirty data segment; The flushing queue is used to store the data segments of the file data being flushed to the disk for storage; If there is a data segment in the flushing queue that has an overlapping interval with the specified dirty data segment, after the file data in the data segment in the flushing queue that has an overlapping interval with the specified dirty data segment is flushed, the specified dirty data segment is added to the flushing queue, and the file data in the specified dirty data segment is flushed to the disk for storage; If there is no data segment in the flushing queue that overlaps with the specified dirty data segment, directly add the specified dirty data segment to the flushing queue, and simultaneously flush the file data in the specified dirty data segment to the disk for storage; After the file data in the specified dirty data segment is flushed, the specified dirty data segment is deleted from the flushing queue and the dirty data queue, and the state of the specified dirty data segment is marked as a clean data segment state, and the specified dirty data segment is added to a clean data queue; the clean data queue is used to store data segments marked as clean data segment states; Release the write lock of the specified dirty data segment.

10. A memory caching method for a distributed file system client, characterized in that: A memory cache framework system applied to any one of claims 1-9; the method comprising: In response to a first request, a first memory space is allocated in a memory pool using a memory pool layer; the memory pool is a memory space applied for in advance to the distributed file system client; the first request is used to request allocation of a requested memory space required for caching first file data; the first memory space is greater than or equal to the requested memory space; wherein the memory pool is divided into a free list management memory space and a spare memory space; the free list management memory space is divided into a plurality of fixed memory spaces; when the requested memory space is less than or equal to a maximum fixed memory space, the first memory space is allocated in the free list management memory space; when the requested memory space is greater than the maximum fixed memory space, the first memory space is allocated in the spare memory space; Using the data segment layer to link the memory addresses corresponding to the file data cached in the memory pool through a bidirectional linked list, a logical address corresponding to the file data is obtained; the logical address is used to access the file data; In response to a file read request, the file layer is used to add a read lock to the specified read data segment to implement a read operation on the specified read data segment, and in response to a file write request, the file layer is used to add a write lock to the specified write data segment to implement a write operation on the specified write data segment; the specified read data segment is the file data interval specified to be read in the file read request; the specified write data segment is the file data interval specified to be written in the file write request; The monitoring layer is used to manage the memory usage of the memory pool and monitor the status of the data segment; the data segment is a file data interval within a file; the status of the data segment includes a dirty data segment status and a clean data segment status.

Images