A cache disk space allocation method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of cloud storage, and discloses a cache disk space allocation method and device, computer equipment and a storage medium, the method comprises the following steps: when the cache disk is formatted, formatting a data area of the cache disk into data blocks of multiple different sizes; receiving cache data, and allocating a target data block for the cache data from the data blocks of the multiple different sizes based on the size of a cache data occupied space; and storing the cache data into the target data block. The technical scheme provided by the application solves the problem of low cache disk space utilization.

Description

Technical Field

[0001] This invention relates to the field of cloud storage technology, specifically to a method, apparatus, computer device, and storage medium for allocating cache disk space. Background Technology

[0002] In the cloud storage field, server storage disks are broadly categorized into cache disks and data disks. Data disks are hard drives in servers that serve the function of storing data, while cache disks store frequently accessed data on the hard drive for faster retrieval on subsequent reads, offering faster read speeds than data disks. Current storage engines primarily manage all input / output of the cache disk through Object-Oriented Units (OBJs), which are the basic units for input / output processing in the underlying storage engine. Within the entire storage engine, the size of a single OBJ is defined as a fixed value. During disk initialization, the formatting program in the storage engine divides the entire disk according to the OBJ size and then numbers all data blocks. Based on this number, the specific location of each block on the disk can be calculated. For the entire storage engine, the size of the OBJ is a defined fixed value. Since the OBJ is the basic unit of data storage requests, each data storage request requires the allocation of an OBJ, regardless of the actual amount of data written. From the disk's perspective, it must occupy the physical disk space corresponding to the OBJ. For example, typically an OBJ (Object Object Class) is defined as 4MB. The physical disk needs to be divided into many 4MB blocks. If an OBJ only writes 4KB of data, it still occupies 4MB of disk space, resulting in only 1 / 1000 of that block's effective disk space utilization. In this mode, the effective utilization of the entire cache disk is greatly reduced. For a relatively small resource like the cache disk, this significant space waste becomes a bottleneck for the entire system. Summary of the Invention

[0003] In view of this, the present invention provides a cache disk space allocation method, apparatus, computer equipment and storage medium to solve the problem of low cache disk space utilization.

[0004] In a first aspect, the present invention provides a method for allocating cache disk space, the method comprising: formatting the data area of ​​the cache disk into multiple data blocks of different sizes when formatting the cache disk; receiving cached data and allocating a target data block for the cached data from the multiple data blocks of different sizes based on the size of the space occupied by the cached data; and storing the cached data into the target data block.

[0005] Based on the above technical means, this embodiment divides the data area of ​​the cache disk into data blocks of different sizes, and allocates data blocks to the data to be cached using the principle of minimum allocation on demand. This makes reasonable use of the free space on the disk, increases the disk utilization rate, improves the cache hit rate of the entire system, and minimizes the cache disk flushing and synchronization operations, which can significantly enhance the input and output throughput of the entire storage system.

[0006] In one optional implementation, the data area of ​​the cache disk is formatted into multiple data blocks of different sizes, including: dividing the data area into data blocks of sizes corresponding to various capacity units, and obtaining a storage ID corresponding to each data block. The storage ID is used to correspond to the physical location of the cache disk, and the storage IDs of each data block are unique; placing the storage ID corresponding to each data block into a free list, where the storage IDs in the free list are used to indicate that the corresponding data block is available; defining the size of OBJs, and mapping the numbered data blocks to different OBJs.

[0007] Based on the aforementioned technical means, this embodiment assigns unique numbers to the divided data blocks to represent the actual physical location of each data block. Then, it defines the size of each Object Object (OBJ) and allocates each data block to a different OBJ. This approach is less complex than defining OBJs of different sizes. When data caching is required, only one OBJ needs to be allocated to handle data storage requests. The OBJ does not directly occupy a large physical space; instead, it selects a smaller data block from the managed data blocks based on the actual size of the data for caching. Unused data blocks can continue to be used in subsequent storage tasks, thus significantly improving the space utilization of the cache disk.

[0008] In one optional implementation, the data area is divided into data blocks of different sizes according to different capacity units, including: dividing the cache disk into three regions; dividing the three regions according to a preset first capacity unit, second capacity unit, and third capacity unit to obtain multiple large data blocks, medium data blocks, and small data blocks, wherein the first capacity unit > the second capacity unit > the third capacity unit.

[0009] Based on the above technical means, the embodiments of the present invention divide the data area of ​​the cache disk into large data blocks, medium data blocks and small data blocks according to three types: large, medium and small. This adapts to a variety of data storage scenarios and avoids too many types of data block divisions, which would increase the complexity of data storage.

[0010] In one optional implementation, receiving cached data and allocating a target data block for the cached data from a variety of data blocks of different sizes based on the size of the space occupied by the cached data includes: detecting the space occupied capacity of the cached data; comparing the space occupied capacity with the capacity of data blocks of various sizes in sequence; determining the smallest target data block capacity from the data block capacities larger than the space occupied capacity based on the comparison results; determining a target storage ID from a free list based on the target data block capacity; determining a target OBJ for using the corresponding target data block based on the target storage ID; receiving cached data through the target OBJ; storing the cached data into the target data block according to the target storage ID through the target OBJ; and deleting the target storage ID from the free list.

[0011] In one optional implementation, the first capacity unit, the second capacity unit, and the third capacity unit are 4M, 1M, and 512K, respectively. The step of receiving cached data and allocating a target data block from multiple data blocks of different sizes based on the space occupied by the cached data includes: detecting the space occupied by the cached data; comparing the space occupied capacity sequentially with the data block capacities of 4M, 1M, and 512K; selecting a target data block capacity from 4M, 1M, and 512K according to the comparison results, wherein the target data block capacity is the smallest capacity greater than the space occupied capacity; determining a target storage ID from the free list based on the target data block capacity; determining a target OBJ corresponding to the target data block based on the target storage ID; receiving the cached data through the target OBJ; storing the cached data into the target data block according to the target storage ID through the target OBJ; and deleting the target storage ID from the free list.

[0012] Based on the above technical means, this embodiment allocates and uses the data block corresponding to the smallest target data block capacity among the data block capacities that are larger than the space occupied capacity according to the size of the cached data. Under the condition of minimal space loss, it also ensures minimal input and output loss, thereby using the corresponding OBJ to process data storage tasks and significantly reducing the resource loss rate of the cache disk.

[0013] In one optional implementation, the method further includes: when the number of data blocks corresponding to a predefined smaller capacity unit is less than a preset threshold, merging the data in the data blocks corresponding to the smaller capacity unit into the data blocks corresponding to the predefined larger capacity unit.

[0014] In one optional implementation, during formatting, each OBJ is converted into an array in the smallest capacity unit using its metadata, and the data in the data block corresponding to the smaller capacity unit is merged into the data block corresponding to the larger capacity unit. This includes: reading the data in the data block corresponding to the smaller capacity unit in the current OBJ according to the order represented by the array; writing the read data into the data block corresponding to the larger capacity unit according to the order represented by the array; rewriting the first storage ID in the current OBJ using the second storage ID, where the second storage ID is the storage ID of the data block corresponding to the larger capacity unit and the first storage ID is the storage ID of the data block corresponding to the smaller capacity unit; deleting the second storage ID from the free list; and restoring the first storage ID to the free list.

[0015] Based on the aforementioned technical methods, the first step is to predefine which data blocks have smaller capacity units and which have larger capacity units. Then, when smaller data blocks are used more frequently, this embodiment also merges some data from smaller data blocks into larger data blocks to solve the problem of insufficient space in small and medium data blocks, further improving the utilization rate of storage resources. Simultaneously, the released storage IDs are added back to the free list to facilitate the repeated use of released storage resources.

[0016] In an optional implementation, the method further includes: receiving a data deletion request; responding to the data deletion request through the current OBJ and clearing the data blocks used by the current OBJ; clearing the metadata information of the current OBJ; restoring the storage ID of the data blocks used by the current OBJ to the free list; when the data in the current OBJ is flushed to the data disk, clearing the data blocks used by the current OBJ; clearing the metadata information of the current OBJ; and restoring the storage ID of the data blocks used by the current OBJ to the free list.

[0017] Secondly, the present invention provides a cache disk space allocation device, the device comprising: a data block partitioning unit, used to format the data area of ​​the cache disk into multiple data blocks of different sizes when formatting the cache disk; a data block allocation unit, used to receive cached data and allocate target data blocks for the cached data from multiple data blocks of different sizes based on the size of the space occupied by the cached data; and a storage unit, used to store the cached data into the target data blocks.

[0018] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the cache disk space allocation method of the first aspect or any corresponding embodiment described above.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the cache disk space allocation method of the first aspect or any corresponding embodiment thereof.

[0020] The technical solution provided by this invention has the following advantages:

[0021] (1) In this embodiment, the data area of ​​the cache disk is divided into data blocks of different sizes, and the data blocks are allocated to the data to be cached according to the principle of minimizing space loss. This makes reasonable use of the free space in the disk, increases the disk utilization rate, improves the cache hit rate of the entire system, and minimizes the cache disk flushing and synchronization operations to the greatest extent, which can significantly enhance the input and output throughput of the entire storage system.

[0022] (2) In this embodiment, the data blocks are numbered uniquely to represent the actual physical location of each data block. Then, the size of the OBJ is defined, and each data block is allocated to a different OBJ. This is less difficult to configure than defining OBJs of different sizes. When data caching is required, only one OBJ needs to be allocated to handle data storage requests. The OBJ does not directly occupy a large physical space, but selects a smaller data block from the managed data blocks according to the actual size of the data to cache the data. Unused data blocks can continue to be used in subsequent storage tasks, thereby significantly improving the space utilization of the cache disk.

[0023] (3) In this embodiment of the invention, the data area of ​​the cache disk is divided into large data blocks, medium data blocks and small data blocks according to three types: large, medium and small, so as to adapt to a variety of data storage scenarios and avoid too many types of data blocks, which would increase the complexity of data storage.

[0024] (4) In this embodiment, the data block corresponding to the smallest target data block capacity among the data block capacities that are larger than the space occupied capacity is allocated and used according to the size of the cached data. Under the condition of minimal space loss, the input and output loss is also minimized, thereby using the corresponding OBJ to process data storage tasks and significantly reducing the resource loss rate of the cache disk.

[0025] (5) When a large number of smaller data blocks are used, this embodiment also merges some data from smaller data blocks into larger data blocks to solve the problem of insufficient space in small and medium data blocks, and further improves the utilization rate of storage resources. At the same time, the released storage IDs are put back into the free list to facilitate the repeated use of released storage resources. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a cache disk space allocation method according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the structure of an OBJ according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of another OBJ according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of another OBJ according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of another OBJ according to an embodiment of the present invention;

[0032] Figure 6 This is a flowchart illustrating the changes in the OBJ structure corresponding to the data block organization according to an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of a cache disk space allocation device according to an embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] According to an embodiment of the present invention, a method for allocating cache disk space is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] This embodiment provides a method for allocating cache disk space, which can be used in the aforementioned computer equipment. Figure 1 This is a flowchart of a cache disk space allocation method according to an embodiment of the present invention, the process including the following steps:

[0038] Step S101: When formatting the cache disk, the data area of ​​the cache disk is formatted into data blocks of various sizes.

[0039] Specifically, the method provided in this embodiment of the invention is mainly applied to the disk management module of the underlying storage engine of the cloud storage platform. It primarily improves the allocation strategy of data blocks within the cache disk to reduce disk wear during I / O (Input-Output) processes, save disk space, and increase disk utilization. Based on this, when formatting the cache disk, this embodiment no longer formats the entire data area of ​​the cache disk to the same size. Instead, it formats the data area into some large data blocks and some small data blocks according to different data block sizes. For example, one area of ​​the data area may be formatted to 4MB, another area to 2MB, and yet another area to 128KB. This embodiment is merely an example and is not limited to this. Thus, various data block sizes are obtained through this scheme. It should be noted that this step is only performed when the disk is formatted for the first time. After the data blocks are divided, subsequent read and write operations do not require formatting.

[0040] Step S102: Receive cached data and allocate target data blocks for the cached data from a variety of data blocks of different sizes based on the size of the space occupied by the cached data.

[0041] Specifically, after the cache disk is formatted, the data sent by the host computer can be stored. When the disk management module receives the cache data, it first determines the size of the cache data, and then selects one or more target data blocks from a variety of data blocks that are closest to the size of the cache data but larger than the cache data space usage, in accordance with the principle of minimizing cache disk space loss, to store the cache data.

[0042] Step S103: Store the cached data into the target data block.

[0043] Specifically, in the final embodiment of the present invention, the cached data is stored in the selected target data block, so that the size of the selected space is as close as possible to the size of the cached data. This avoids the result of blindly allocating a large physical disk space to the cached data according to completely equal OBJs, but actually only using a small amount of physical space, which would result in wasted resources.

[0044] It should be noted that in this embodiment of the invention, each received cached data is defined as not exceeding the maximum data block size. For example, related technologies define the size of OBJ as 4M, which by default means that each cached data will not exceed 4M. In this embodiment of the invention, if the maximum data block size is defined as 4M, then similarly, each cached data will not exceed 4M. This is the same mechanism widely used in related technologies. For scenarios where a cached data exceeds the maximum data block size, such as a cached data exceeding 4M, these scenarios are relatively extreme, and users usually do not define data in this way, so they are not within the scope of this embodiment of the invention.

[0045] This embodiment divides the data area of ​​the cache disk into data blocks of different sizes and allocates data blocks to the data to be cached using the principle of minimum allocation on demand. This makes reasonable use of the free space on the disk, increases the disk utilization rate, improves the cache hit rate of the entire system, and minimizes the cache disk flushing and synchronization operations, which can significantly enhance the input and output throughput of the entire storage system.

[0046] In some alternative implementations, step S101 includes:

[0047] Step a1: Divide the data area into data blocks of different sizes according to different capacity units, and obtain the storage ID corresponding to each data block. The storage ID is used to correspond to the physical location of the cache disk. The storage IDs of each data block are unique.

[0048] Step a2: Add the storage ID corresponding to each data block to the free list. The storage ID in the free list is used to indicate that the corresponding data block is available.

[0049] Step a3: Define the size of OBJ and map the numbered data blocks to different OBJs.

[0050] Specifically, OBJ is the basic unit for the storage engine to process I / O. In other words, as a software structural concept, OBJ can be simply understood as follows: each time the host computer requests cached data, the disk management module provides a response request. This software object corresponds to a physical location of a fixed size on the cache disk. The data area is divided equally according to the fixed-size OBJ. The technology of the disk management module using OBJ as the basic unit to process I / O each time is relatively mature and sound, and the operation is relatively simple. If the OBJ is directly divided into multiple different sizes and then different sizes of OBJ are used to process I / O, this kind of technology is more difficult to implement and requires too much modification to the mechanism of disk management module to process I / O, which is not conducive to the compatibility of the technical solution provided in the embodiments of the present invention on related storage engines.

[0051] Therefore, the solution provided by the embodiments of the present invention first divides the data area into data blocks of the corresponding size of each capacity unit according to different capacity units, and then assigns a unique number to each data block to obtain the storage ID corresponding to each data block. The obtained storage ID is used to mark each actual physical location of the cache disk, so that the storage ID and the physical location have a mapping relationship.

[0052] Next, the storage ID corresponding to each data block is placed into the free list. The free list is a data set used by the disk management module to manage the status of each data block. If there is no data stored in a data block, the storage ID corresponding to the data block will be placed into the free list. If the data block has been allocated storage data, the storage ID corresponding to the data block will be removed from the free list, so that the disk management module knows which data blocks are available and which data blocks are not available.

[0053] Next, the size of the OBJ is defined as a fixed size, for example, 4MB. Then, the numbered data blocks are mapped to different OBJs. This means the disk management module has multiple OBJs available, and different OBJs use different data blocks. When the host computer needs to cache data, the disk management module still follows the storage engine's I / O processing mechanism, first allocating an OBJ for the data storage request. However, the difference is that since each OBJ also manages multiple data blocks of different sizes, it doesn't directly cache the data in a fixed-size physical space. Instead, it analyzes the size of the cached data and then uses the data blocks used by the OBJ to allocate matching actual data blocks for the cached data. Thus, based on the storage ID corresponding to the data blocks used by the OBJ, the cached data is stored in the corresponding physical location. If some data blocks currently used by the OBJ are still unused, the current OBJ can still be allocated and used during the next data I / O processing.

[0054] Based on this, this embodiment combines the OBJ mechanism for handling IO with related technologies, and integrates the data block partitioning method of different sizes provided in this embodiment of the invention to achieve a cache disk space allocation method with higher compatibility and improved storage resource utilization.

[0055] In one embodiment, step a1 specifically includes:

[0056] 1. Divide the cache disk into three regions.

[0057] 2. Divide the three regions into three pre-defined capacity units: first capacity unit, second capacity unit, and third capacity unit, to obtain multiple large data blocks, medium data blocks, and small data blocks, with the first capacity unit > the second capacity unit > the third capacity unit.

[0058] Specifically, in this embodiment of the invention, the data area of ​​the cache disk is divided into large data blocks, medium data blocks, and small data blocks according to three types: large, medium, and small. This adapts to various data storage scenarios and avoids too many types of data block divisions, which would increase the complexity of data storage.

[0059] Based on the above steps, in a specific scenario embodiment, the first capacity unit, the second capacity unit, and the third capacity unit are preferably 4M, 1M (1 / 4 of 4M), and 512K (1 / 8 of 4M), respectively, thus obtaining data blocks of three sizes: 4M, 1M, and 512K. Therefore, in this specific scenario embodiment, during system initialization, the disk is formatted, and the Data area, originally a unified area, is divided into three areas according to a specific ratio: the large data block is divided according to the size of OBJ (4M), the medium data block is divided according to 1 / 4 of the size of OBJ (1M), and the small data block is divided according to 1 / 8 of the size of OBJ (512K). A unique storage ID is assigned to each data block to ensure that the storage IDs of the large, medium, and small data blocks are not duplicated. Assume that the allocated storage IDs [1, 1000] belong to the small block area, [1001, 2000] belong to the medium block area, and [2000, max] belong to the large block area. Then, set up three free lists for small, medium, and large blocks respectively, and put the corresponding storage IDs into their respective free lists to indicate that the above data blocks are all available.

[0060] Since the smallest unit for processing data I / O is not the aforementioned data block, this embodiment also defines an OBJ. Assuming the defined OBJ size is 4MB, the aforementioned data blocks are mapped to different OBJs. Therefore, the solution provided by this embodiment is compatible with most storage engines. The resulting OBJs typically have the following characteristics:

[0061] First, OBJ only manages data blocks of 512KB in size, and can manage one or more, with a maximum of eight, because it cannot exceed the defined limit of 4MB. For example... Figure 2 As shown, when only one block of data smaller than 512KB is written to the entire OBJ, the entire OBJ can occupy only the 512KB data block numbered 112 on the disk, instead of the original 4MB data block. The null positions in the diagram indicate that no other data blocks are currently used in the OBJ.

[0062] Secondly, OBJ only manages data blocks of 1MB in size, and can manage one or more, with a maximum of four, because it cannot exceed the defined limit of 4MB. For example... Figure 3 As shown, when a piece of data smaller than 1M is written, the entire OBJ can occupy only the 1M data block numbered 1002 on the disk, instead of the original 4M data block.

[0063] Third, OBJ only manages data blocks of 4MB in size, and can only manage one, because it cannot exceed the defined limit of 4MB. For example... Figure 4 As shown, when a certain OBJ is written to the entire 4M data block, the entire OBJ occupies the 4M data block numbered 2001.

[0064] Fourth, and most common, is the mixed management scenario for OBJs, where the data blocks managed by OBJs range from 1MB to 512KB in size. For example... Figure 5 As shown, positions 1 and 5 in OBJ specify a 512K data block 2 and 10 respectively. Positions 2 and 3 jointly occupy a 1M data block 1000, and positions 6 and 7 jointly occupy a 1M data block. The entire OBJ actually occupies 3M of disk space.

[0065] In other words, this embodiment of the invention still uses OBJ as the basic unit to process data I / O. Although OBJ is defined as 4M, it does not directly allocate 4M physical space to data. Instead, OBJ can manage a maximum of 4M data blocks. When processing I / O, OBJ reallocates data blocks based on the actual size of the received cached data, and then stores the data in the corresponding location according to the storage ID of the actually allocated data block. For example... Figure 5 If the data block with storage ID 2 is used, but other data blocks are not used, then the current OBJ can continue to be used the next time data is stored.

[0066] Based on this, this embodiment combines the OBJ mechanism for handling IO with related technologies, and integrates the data block partitioning method of different sizes provided in this embodiment of the invention to achieve a cache disk space allocation method with higher compatibility and improved storage resource utilization.

[0067] In some alternative implementations, step S102 includes:

[0068] Step b1: Detect the space usage of cached data.

[0069] Step b2 involves comparing the space occupied capacity with the capacity of data blocks of various sizes in sequence.

[0070] Step b3: Based on the comparison results, determine the smallest target data block capacity from the data block capacities that are greater than the space occupied capacity.

[0071] Step b4: Determine the target storage ID from the free list based on the target data block capacity.

[0072] Step b5: Determine the target OBJ that uses the corresponding target data block based on the target storage ID.

[0073] Step b6: Receive cached data through the target OBJ.

[0074] Step b7: Store the cached data into the target data block according to the target storage ID using the target OBJ.

[0075] Step b8: Remove the target storage ID from the free list.

[0076] Specifically, for certain cached data sizes, there are usually several different data block allocation schemes. For example, if the cached data is 900KB, it can be allocated one 1MB data block or two 512KB data blocks for storage, and the storage space consumption is relatively small. If the cached data is 1.8MB, it can be allocated two 1MB data blocks or one 4MB data block for storage, with the 4MB data block having a slightly larger space consumption. However, if only minimizing space consumption is considered, using schemes like allocating two 1MB data blocks or two 512KB data blocks will cause IO consumption to increase exponentially. In the cloud computing field, where high data interaction speed is required, high IO consumption is also unacceptable.

[0077] Based on this, this embodiment of the invention provides a specific space allocation scheme that minimizes both storage resource consumption and IO consumption. First, the disk management module calculates the space occupied by the cached data to determine its size. Then, the space occupied is compared sequentially with the capacity of various data block sizes. For example, if the cached data is 900K, it is compared sequentially with data block sizes of 512K, 1M, and 4M. Next, the smallest target data block capacity is determined from the data block capacities larger than the space occupied. For example, if there are 1M and 4M data blocks larger than 900K, the smallest 1M is selected as the target data block capacity, and the 1M data block is used to store the 900K data. Then, the target storage ID is determined from the free list based on the target data block capacity. For example, it searches the free list to find which 1M data blocks are available, such as the data block with target storage ID 1000. Since different OBJs use different data blocks, it is necessary to determine the target storage ID based on the target storage ID. The ID determines which OBJ uses the data block corresponding to the target storage ID. Assuming the OBJ using the target data block is OBJ1, then OBJ1 is the target OBJ. Then, cached data is received through the target OBJ; in other words, data I / O is processed by calling OBJ1. Afterward, when the target OBJ processes cached data, it doesn't directly allocate 4MB of physical space. Instead, it stores the cached data into the corresponding target data block according to the aforementioned target storage ID "1000". Finally, the disk management module removes the target storage ID from the free list, making the data block's status unavailable. However, other data blocks in OBJ1 remain available, so when other data blocks in OBJ1 are used, OBJ1 can still be called to process I / O.

[0078] Thus, by employing the above methods, both storage resource consumption and IO consumption are minimized, thereby reducing the overall resource consumption of the storage engine.

[0079] In an optional implementation, when the first capacity unit, the second capacity unit, and the third capacity unit are 4M, 1M, and 512K respectively, step S102 includes:

[0080] Step f1: Detect the space usage of cached data;

[0081] Step f2: Compare the space occupied capacity with the data block capacities of 4M, 1M and 512K in sequence;

[0082] Step f3: Based on the comparison results, select the target data block capacity from 4M, 1M and 512K. The target data block capacity is the minimum capacity that is greater than the space occupied.

[0083] Step f4: Determine the target storage ID from the free list based on the target data block capacity;

[0084] Step f5: Determine the target OBJ that uses the corresponding target data block based on the target storage ID;

[0085] Step f6: Receive buffered data through the target OBJ;

[0086] Step f7: Store the cached data into the target data block according to the target storage ID using the target OBJ;

[0087] Step f8: Remove the target storage ID from the free list.

[0088] Specifically, the principle of the specific cache space allocation scheme provided in this embodiment is the same as that of steps b1 to b8 above, and will not be repeated here. Please refer to the relevant descriptions of steps b1 to b8 above.

[0089] In some optional embodiments, the cache disk space allocation method provided by the present invention further includes the following steps:

[0090] Step c1: When the number of data blocks corresponding to the predefined smaller capacity unit is less than the preset threshold, merge the data in the data blocks corresponding to the smaller capacity unit into the data blocks corresponding to the predefined larger capacity unit.

[0091] Specifically, due to the limited resources of the cache disk, after a large number of read / write I / O operations, there may be insufficient space in small and medium blocks. If an Object (OBJ) performs a small amount of data writing at this time, it will trigger the old processing flow, i.e., requesting a complete large data block for use, resulting in wasted space. Therefore, this embodiment of the invention first defines data blocks below a preset capacity as smaller capacity units and those above the preset capacity as larger capacity units. Then, a disk defragmentation process is added, mainly targeting OBJs that have undergone a large number of reads and writes, where the entire OBJ contains a large amount of data but occupies multiple small data blocks. For such OBJs, a separate large data block is reallocated, all data of the OBJ is read and merged, and then written into the newly allocated large data block. Finally, the occupied small data blocks are released, further saving storage resources.

[0092] In some alternative implementations, step c1 specifically includes:

[0093] In step d1, during formatting, each OBJ is converted into an array with the smallest capacity unit based on its metadata.

[0094] Step d2: Read out the data in the data block corresponding to the smaller capacity unit in the current OBJ according to the order represented by the array.

[0095] Step d3: Write the read data into the data block corresponding to the larger capacity unit in the order represented by the array.

[0096] Step d4: Rewrite the first storage ID in the current OBJ using the second storage ID. The second storage ID is the storage ID of the data block corresponding to the larger capacity unit, and the first storage ID is the storage ID of the data block corresponding to the smaller capacity unit.

[0097] Step d5: Remove the second storage ID from the free list.

[0098] Step d6: Restore the first storage ID to the free list.

[0099] Specifically, in this embodiment of the invention, in order to further facilitate the accurate reading of the organized data, each OBJ is rewritten in array form during disk formatting, as described above. Figures 2 to 5 In the metadata of the OBJ file, the original data block marked with only an ID is changed to an array of 8 elements, using a minimum size unit of 512K. The numbers under the box in the diagram indicate the relative positions of the data within the OBJ. Therefore, when merging data, such as... Figure 6 As shown, first, the data in the data block previously used by OBJ is read out. Then, the array order in which the data is stored in OBJ remains unchanged, i.e., the data from 0K to 512K, from 512K to 1M, ..., from 3M to 4M remain the same. Next, the read data is written into the large data block according to the order represented by the array. Meanwhile, after the data migration is completed, the current OBJ uses the data block corresponding to the first storage ID before data processing, for example... Figure 6 The numbers 1, 2, 1000, 9, 10, and 1001 in the data block refer to the data block with the second storage ID after data processing. For example... Figure 6 The ID 2001 in the first storage ID (OBJ) needs to be rewritten using the second storage ID. Through this method, although the data blocks change before and after data processing, the relative positions of the data remain unchanged. Therefore, when reading data, even with only one large data block ID, individual small data within the large data block can be accurately read based on the relative positions of the data storage represented by the array, solving the problem of inaccurate data reading after data processing.

[0100] Meanwhile, in this embodiment, the second storage ID after data processing is also removed from the free list, indicating that the large data block has been occupied. The first storage ID before data processing is restored to the free list, indicating that the small data block has been released, so that the disk management module can reuse the released small data block to store new data.

[0101] In some optional embodiments, the cache disk space allocation method provided by the present invention further includes the following steps:

[0102] Step e1: Receive a data deletion request.

[0103] Step e2: Request a data deletion in response to the current OBJ and clear the data blocks used by the current OBJ.

[0104] Step e3: Clean up the metadata information of the current OBJ.

[0105] Step e4: Restore the storage ID of the data block currently used by the OBJ to the free list.

[0106] Step e5: When the data in the current OBJ is flushed to the data disk, the data blocks used by the current OBJ are cleared.

[0107] Step e6: Clean up the metadata information of the current OBJ.

[0108] Step e7: Restore the storage ID of the data block currently used by the OBJ to the free list.

[0109] Specifically, in this embodiment of the invention, the release of data blocks includes two aspects in addition to data cleanup. The first is the normal deletion of OBJ data by the disk management module. Based on the received data deletion request, the data blocks currently used by the OBJ are cleared, the metadata information of the OBJ is cleaned up, all data blocks requested by the OBJ are released, and the corresponding storage is put back into the corresponding free list according to the storage ID.

[0110] Secondly, when cached data is flushed to the data disk, the old cached data generally does not need to be retained, which will also trigger resource release. In this case, it should be handled according to the aforementioned OBJ deletion logic.

[0111] The solution provided by the embodiments of the present invention further reduces the storage of redundant data, makes the use of data blocks more flexible, and improves the space utilization of cache disk.

[0112] It is important to note that in some optional implementations, the OBJ data write IO process involves the allocation of data blocks. This embodiment determines whether a new data block needs to be allocated and its required size based on the write location described in the IO request, the data size, and the data block information already obtained in the OBJ's current metadata. Data is then written to the new data block or the original data block is overwritten. Finally, the storage ID is obtained, mapped to the actual disk location based on the storage ID, and the data is written to the disk. Then, the OBJ's metadata information is updated to complete the read / write IO. For example: Figure 5As shown, if the write position described by the previous IO request was in the 512K~1M range, then the data will be written to the second data block. If the write position of the current IO request is in the 2.5M~3M data block, then the data will be written to the newly allocated 10th data block. However, if the write position of the current IO request is also in the 512K~1M range, then a new data block will not be allocated; instead, the new data will overwrite the old data in the second data block. This further improves the flexibility of data reading and writing.

[0113] The solution provided by this invention employs a rational disk allocation and management strategy. During the I / O process, appropriate disk space is allocated to OBJs as needed, reducing disk space loss caused by unreasonable system design and maximizing disk space utilization efficiency. This significantly improves the stability and I / O throughput of the entire storage platform. The allocation of cache disk space utilizes multiple methods, combining I / O requests and current metadata information to request reasonable space for data storage, reducing cache disk wear and thus improving cache hit rate and overall system performance.

[0114] This embodiment also provides a cache disk space allocation device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0115] This embodiment provides a cache disk space allocation device, such as... Figure 7 As shown, it includes:

[0116] The data block partitioning unit 701 is used to format the data area of ​​the cache disk into data blocks of various sizes when formatting the cache disk. For details, please refer to the relevant description of step S101 in the above method embodiment, which will not be repeated here.

[0117] The data block allocation unit 702 is used to receive cached data and allocate target data blocks for the cached data from a variety of data blocks of different sizes based on the size of the space occupied by the cached data. For details, please refer to the relevant description of step S102 in the above method embodiment, which will not be repeated here.

[0118] Storage unit 703 is used to store cached data into the target data block. For details, please refer to the relevant description of step S103 in the above method embodiment, which will not be repeated here.

[0119] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0120] In this embodiment, the cache disk space allocation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0121] This invention also provides a computer device having the above-described features. Figure 7 The cache disk space allocation device shown.

[0122] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0123] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0124] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0125] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0126] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0127] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0128] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0129] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method of allocating cache disk space, characterized by, The method includes: When formatting the cache disk, the data area of ​​the cache disk is formatted into data blocks of various sizes. Formatting the data area of ​​the cache disk into data blocks of various sizes includes: dividing the data area into data blocks of sizes corresponding to different capacity units, and obtaining a storage ID corresponding to each data block. The storage ID is used to correspond to the physical location of the cache disk, and the storage IDs of each data block are unique; placing the storage ID corresponding to each data block into a free list, where the storage IDs in the free list indicate that the corresponding data block is available; defining the size of OBJs and mapping the numbered data blocks to different OBJs; when the number of data blocks corresponding to a predefined smaller capacity unit is less than a preset threshold, merging the data in the data blocks corresponding to the smaller capacity unit into the data blocks corresponding to the predefined larger capacity unit. During formatting, each OBJ is converted into an array using its metadata in the smallest capacity unit. The step of merging the data in the data block corresponding to the smaller capacity unit into a predefined data block corresponding to a larger capacity unit includes: reading the data in the data block corresponding to the smaller capacity unit in the current OBJ according to the order represented by the array; writing the read data into the data block corresponding to the larger capacity unit according to the order represented by the array; rewriting the first storage ID in the current OBJ using a second storage ID, where the second storage ID is the storage ID of the data block corresponding to the larger capacity unit and the first storage ID is the storage ID of the data block corresponding to the smaller capacity unit; deleting the second storage ID from the free list; and restoring the first storage ID to the free list. Receive cached data, and allocate a target data block for the cached data from a variety of data blocks of different sizes based on the size of the space occupied by the cached data; The cached data is stored in the target data block.

2. The method according to claim 1, characterized in that, The process of dividing the data area into data blocks of corresponding sizes according to different capacity units includes: The cache disk is divided into three areas; The three regions are divided according to the preset first capacity unit, second capacity unit and third capacity unit to obtain multiple large data blocks, medium data blocks and small data blocks, with the first capacity unit > the second capacity unit > the third capacity unit.

3. The method according to claim 2, characterized in that, The first capacity unit, the second capacity unit, and the third capacity unit are 4M, 1M, and 512K, respectively. The process of receiving cached data and allocating target data blocks from multiple data blocks of different sizes based on the size of the cached data includes: Detect the space usage of the cached data; The space occupied capacity is compared with the data block capacities of 4M, 1M and 512K in sequence; Based on the comparison results, a target data block capacity is selected from 4M, 1M and 512K, wherein the target data block capacity is the minimum capacity that is greater than the space occupied capacity; The target storage ID is determined from the free list based on the target data block capacity; The target OBJ corresponding to the target data block is determined based on the target storage ID; The cached data is received through the target OBJ; The cached data is stored in the target data block according to the target storage ID using the target OBJ; Remove the target storage ID from the free list.

4. The method according to claim 1, characterized in that, The process of receiving cached data and allocating target data blocks from multiple data blocks of different sizes based on the size of the cached data includes: Detect the space usage of the cached data; The space occupied capacity is compared sequentially with the capacity of data blocks of various sizes; Based on the comparison results, the smallest target data block capacity is determined from the data block capacities that are greater than the space occupancy capacity; The target storage ID is determined from the free list based on the target data block capacity; The target OBJ corresponding to the target data block is determined based on the target storage ID; The cached data is received through the target OBJ; The cached data is stored in the target data block according to the target storage ID using the target OBJ; Remove the target storage ID from the free list.

5. The method according to claim 1, characterized in that, The method further includes: Receive data deletion requests; The data block used by the current OBJ is cleared by responding to the data deletion request. Clear the metadata information of the current OBJ; Restore the storage ID of the data block currently used by the OBJ to the free list; When the data in the current OBJ is flushed to the data disk, the data blocks used by the current OBJ are cleared; Clear the metadata information of the current OBJ; Restore the storage ID of the data block currently used by the OBJ to the free list.

6. A cache disk space allocation device, characterized in that, The device includes: A data block partitioning unit is used to format the data area of ​​the cache disk into data blocks of various sizes during cache disk formatting. Formatting the data area of ​​the cache disk into data blocks of various sizes includes: dividing the data area into data blocks of sizes corresponding to different capacity units, and obtaining a storage ID corresponding to each data block. The storage ID is used to correspond to the physical location of the cache disk, and the storage IDs of each data block are unique; placing the storage ID corresponding to each data block into a free list, where the storage IDs in the free list indicate that the corresponding data block is available; defining the size of an Object Object Class (OBJ) and mapping numbered data blocks to different OBJs; when the number of data blocks corresponding to a predefined smaller capacity unit is less than a preset threshold, merging the data in the data blocks corresponding to the smaller capacity unit into the data blocks corresponding to the predefined larger capacity unit. During formatting, each OBJ is converted into an array using its metadata in the smallest capacity unit. The step of merging the data in the data block corresponding to the smaller capacity unit into a predefined data block corresponding to a larger capacity unit includes: reading the data in the data block corresponding to the smaller capacity unit in the current OBJ according to the order represented by the array; writing the read data into the data block corresponding to the larger capacity unit according to the order represented by the array; rewriting the first storage ID in the current OBJ using a second storage ID, where the second storage ID is the storage ID of the data block corresponding to the larger capacity unit and the first storage ID is the storage ID of the data block corresponding to the smaller capacity unit; deleting the second storage ID from the free list; and restoring the first storage ID to the free list. A data block allocation unit is used to receive cached data and allocate a target data block for the cached data from a variety of data blocks of different sizes based on the size of the space occupied by the cached data; A storage unit is used to store the cached data into the target data block.

7. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the cache disk space allocation method according to any one of claims 1 to 5 by executing the computer instructions.

Images