Storage resource scheduling method and device, electronic equipment and storage medium

Abstract

The storage resource scheduling method and device, electronic equipment and storage medium provided by the application pre-allocate a logical space smaller than the maximum value (i.e. block size BlockSize) of the logical space that can be occupied by a single data block in the storage system for the data block when creating the data block, while allowing the actual physical space used by the data block to exceed the size value of the pre-allocated logical space (the upper limit is BlockSize) when writing and storing data, and in the case of exceeding, temporarily and dynamically additionally allocating the corresponding logical space of the data corresponding to the exceeding part to the data block to meet the logical space requirement of the exceeding part of the data. In this way, the logical space size occupied by the data block in the system is closer to the actual physical space size occupied, which can effectively improve the situation of idle physical space but insufficient logical space in the traditional technology, and improve the physical storage resource utilization of the system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data storage, and particularly relates to a storage resource scheduling method and device, an electronic device and a storage medium. BACKGROUND

[0002] A storage system generally uses a data object (Object) as the minimum storage unit, and the size of an Object is usually from several tens of KB to several MB; and uses a data block (Block) as the minimum management unit in the storage system, and the data block is a collection of multiple data objects. Whether storage resource application or recovery, the data block is the minimum operation unit, and the size of a data block is generally between several hundred MB and several GB.

[0003] In the conventional technology, when a new data block is created, the space of the BlockSize (block size) is allocated in advance for the data block at one time. The space of the BlockSize allocated in advance for the data block by the object storage system when the data block is created is referred to as logical space. In the storage process, whether the data objects in a data block are written to fill the disk space of the BlockSize (i.e., the real disk space occupied by the data block, also referred to as physical space), the system determines that the disk space of the BlockSize in the disk has been allocated to the data block. This easily leads to the situation that the physical space is idle but the logical space is insufficient in the storage system, resulting in low utilization of the physical storage resources (such as disk resources) of the system. SUMMARY

[0004] Therefore, the application provides a storage resource scheduling method and device, an electronic device and a storage medium. By dynamically adjusting the logical space proportion of a data block, the situation that the physical space is idle but the logical space is insufficient in the conventional technology is improved, and the utilization of the physical storage resources of the system is improved.

[0005] The specific technical solutions are as follows:

[0006] A storage resource scheduling method comprises the following steps:

[0007] Obtaining a data block creation request;

[0008] Creating a data block, and pre-allocating a logical space with a predetermined size value for the created data block; the predetermined size value is less than the maximum value of the logical space that can be occupied by a single data block in the storage system;

[0009] When data is written for storage, if the data amount of the data block exceeds the predetermined size value, then:

[0010] Allocating a logical space with a size value not less than the excess amount to the data block;

[0011] write data of the data block which does not exceed the predetermined size value into a physical space corresponding to a pre-allocated logical space of the data block, and write data exceeding the predetermined size value into a physical space corresponding to at least a part of an additionally allocated logical space of the data block;

[0012] wherein the exceeding amount is an amount by which the data amount of the data block exceeds the predetermined size value; and a total size of the pre-allocated logical space and the additionally allocated logical space of the data block does not exceed the maximum value.

[0013] Optionally, the pre-allocated logical space of the created data block with the predetermined size value comprises:

[0014] allocating the pre-allocated logical space of the created data block with the predetermined size value from a remaining logical space of the storage system, if the remaining logical space is not less than the predetermined size value.

[0015] wherein the remaining logical space comprises a result obtained by removing logical spaces occupied by all data blocks from a total logical space of the storage system.

[0016] If a size of a physical space actually occupied by the data block does not exceed the predetermined size value, a size of a logical space occupied by the data block is the predetermined size value; if the size of the physical space actually occupied by the data block exceeds the predetermined size value, the size of the logical space occupied by the data block is the size of the physical space actually occupied by the data block.

[0017] Optionally, after the data block is created, the method further comprises:

[0018] monitoring a recycling operation on a physical space occupied by the data block.

[0019] if the recycling operation is monitored, determining whether a first size value of a logical space currently occupied by a target data block corresponding to the recycling operation monitored exceeds the predetermined size value.

[0020] if the first size value exceeds the predetermined size value, releasing a part of the logical space currently occupied by the target data block according to the predetermined size value and a second size value of a physical space actually occupied by the target data block after the recycling operation is executed.

[0021] Optionally, the releasing the part of the logical space currently occupied by the target data block according to the predetermined size value and the second size value of the physical space actually occupied by the target data block after the recycling operation is executed comprises:

[0022] if the second size value is not greater than the predetermined size value, releasing a part of the logical space currently occupied by the target data block exceeding the predetermined size value.

[0023] if the second size value is greater than the predetermined size value, releasing a part of the logical space currently occupied by the target data block that exceeds the second size value.

[0024] Optionally, the method further includes:

[0025] obtaining an adjustment instruction for the predetermined size value;

[0026] adjusting the predetermined size value in the storage system to a size value indicated by the adjustment instruction based on a hot loading manner, to obtain an adjusted size value of the pre-allocated logical space; and performing a matching logical space adjustment operation on the logical space occupied by the data block in the storage system based on the adjusted size value of the pre-allocated logical space.

[0027] A storage resource scheduling apparatus includes:

[0028] a creating unit configured to obtain a data block creating request and create a data block;

[0029] an allocating unit configured to pre-allocate a logical space with a predetermined size value for the created data block; and in a case where a data amount of the data block exceeds the predetermined size value when data is written for storage, additionally allocate a logical space with a size value not less than the excess amount for the data block;

[0030] a writing unit configured to write data of the data block that does not exceed the predetermined size value into a physical space corresponding to the pre-allocated logical space for the data block; and write data of the excess part into a physical space corresponding to at least part of the additionally allocated logical space for the data block;

[0031] wherein the excess amount is an amount by which the data amount of the data block exceeds the predetermined size value; and a total size of the pre-allocated logical space and the additionally allocated logical space does not exceed the maximum value.

[0032] Optionally, the allocating unit, in terms of pre-allocating a logical space with a predetermined size value for the created data block, is specifically configured to:

[0033] in a case where a remaining logical space of the storage system is not less than the predetermined size value, allocate the logical space with the predetermined size value for the created data block from the remaining logical space;

[0034] wherein the remaining logical space includes a result obtained by removing logical spaces occupied by all data blocks from all logical spaces of the storage system;

[0035] If the size of the physical space actually occupied by the data block does not exceed the predetermined size value, the size of the logical space occupied by the data block is the predetermined size value; if the size of the physical space actually occupied by the data block exceeds the predetermined size value, the size of the logical space occupied by the data block is the size value of the physical space actually occupied by the data block.

[0036] Optionally, the apparatus further includes:

[0037] a monitoring unit configured to monitor a recycling operation on the physical space occupied by the data block;

[0038] a determining unit configured to, in a case where the recycling operation is monitored, determine whether a first size value of the logical space currently occupied by a target data block corresponding to the monitored recycling operation exceeds the predetermined size value;

[0039] a releasing unit configured to, in a case where the first size value exceeds the predetermined size value, release part of the logical space currently occupied by the target data block according to the predetermined size value and a second size value of the physical space actually occupied by the target data block after the recycling operation is performed.

[0040] Optionally, the releasing unit is specifically configured to:

[0041] if the second size value is not greater than the predetermined size value, release part of the logical space currently occupied by the target data block that exceeds the predetermined size value;

[0042] if the second size value is greater than the predetermined size value, release part of the logical space currently occupied by the target data block that exceeds the second size value.

[0043] Optionally, the apparatus further includes:

[0044] an adjusting unit configured to:

[0045] obtain an adjusting instruction for the predetermined size value;

[0046] adjust the predetermined size value in the storage system to a size value indicated by the adjusting instruction based on a hot loading manner, to obtain an adjusted size value of the pre-allocated logical space; and perform a matching logical space adjusting operation on the logical space occupied by the data block in the storage system based on the adjusted size value of the pre-allocated logical space.

[0047] An electronic device, characterized by comprising:

[0048] a memory configured to store a computer instruction set;

[0049] a processor configured to implement the storage resource scheduling method according to any one of the preceding embodiments by executing the instruction set stored in the memory.

[0050] A computer readable storage medium, the computer readable storage medium has a computer instruction set stored therein, the computer instruction set is executed by a processor to implement the storage resource scheduling method according to any one of the above.

[0051] According to the above scheme, the storage resource scheduling method, device, electronic equipment and storage medium provided by the application, when creating a data block, a logical space smaller than the maximum value of the logical space that can be occupied by a single data block in the storage system (that is, the block size BlockSize) is pre-allocated to the data block, while allowing the actual physical space used by the data block to exceed the size value of the pre-allocated logical space (the upper limit is BlockSize) when writing and storing data, and in the case of exceeding, the corresponding size of the logical space corresponding to the exceeding part of the data is temporarily dynamically appended to the data block to meet the logical space requirement of the exceeding part of the data. In this way, the logical space size occupied by the data block in the system is closer to the actual physical space size occupied by the data block, which can effectively improve the situation that the physical space is idle but the logical space is insufficient in the traditional technology, and improve the physical storage resource utilization of the system. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0053] Figure 1 is a flowchart of the storage resource scheduling method provided by the embodiment of the present application;

[0054] Figure 2 is a schematic diagram of the logical space size occupied by the data block being the pre-allocated logical space size provided by the embodiment of the present application;

[0055] Figure 3 is a schematic diagram of the logical space size occupied by the data block being the actual physical space size used by the data block provided by the embodiment of the present application;

[0056] Figure 4 is another flowchart of the storage resource scheduling method provided by the embodiment of the present application;

[0057] Figure 5 is a schematic diagram of the pre-allocated and occupied storage space of the data block before the execution of the recycling operation provided by the embodiment of the present application;

[0058] Figure 6is a schematic diagram of pre-allocated and occupied storage space of a data block after a recovery operation is executed, provided by an embodiment of the present application;

[0059] Figure 7 is another schematic diagram of pre-allocated and occupied storage space of a data block after a recovery operation is executed, provided by an embodiment of the present application;

[0060] Figure 8 is a structural schematic diagram of a storage resource scheduling apparatus, provided by an embodiment of the present application;

[0061] Figure 9 is another structural schematic diagram of a storage resource scheduling apparatus, provided by an embodiment of the present application;

[0062] Figure 10 is yet another structural schematic diagram of a storage resource scheduling apparatus, provided by an embodiment of the present application;

[0063] Figure 11 is a structural schematic diagram of an electronic device, provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0065] In the conventional technology, a data block (Block) is the smallest management unit in a storage system, and the Block is the smallest operation unit for the application or recovery of storage resources. When a new data block is created, the space of a BlockSize is allocated in advance for the data block at one time, and the space of the BlockSize allocated in advance for the data block by the storage system when the data block is created is referred to as a logical space. In the storage process, whether the data objects in a data block are written to fill the disk space of the BlockSize (i.e., the disk space actually occupied by the Block, also referred to as a physical space) or not, the system determines that the disk space of the BlockSize in the disk has been allocated to the data block. When the data objects in the data block are deleted, the system performs a recovery operation (Compaction) to recover the physical space corresponding to the deleted data objects. The inventors have found that the logical space of the BlockSize allocated for the data block in the conventional technology is not released with the recovery of the physical space occupied by part of the data objects in the data block, and only when all the data objects in the data block are deleted and all the physical space occupied by the data objects is recovered, the entire logical space of the BlockSize allocated for the data block is released. This easily leads to a situation in which the physical space is idle but the logical space is insufficient in the storage system, and accordingly leads to a low utilization rate of the physical storage resources (such as disk resources) of the system.

[0066] The above problems of the storage system can further lead to other derivative problems in a specific application scenario. A typical application scenario is an object storage system. Object storage is a storage method for unstructured data, and the object storage system often adopts a storage structure of "cluster-data node-Block-Object". This structure can provide high concurrency capability of object storage and improve the bandwidth utilization rate of the object storage system. Based on the storage structure, the object storage system can be used to provide massive, low-cost, strong security, high-reliability distributed cloud storage services and data read-write services, and solve related complex problems such as storage expansion, data reliability and security, and distributed access for users.

[0067] When data migration needs to be performed due to machine relocation, data scale reduction of the cluster requiring downsizing, or over-provisioned machines in the cluster requiring to be offline, it is usually desired to concentrate data on a small number of machines, for example, to concentrate data of multiple machines in the cluster on a small number of machines to achieve cluster downsizing or offline of over-provisioned machines, or to concentrate data of multiple machines in the cluster on a small number of machines to achieve convenient machine relocation. However, based on the current situation in the prior art that a data block is used as the minimum operation unit for application and recovery of storage space, the data to be migrated cannot be concentrated on a small number of machines due to a case that the physical space is sufficient but the logical space is insufficient in the machine, which hinders data migration and brings inconvenience to machine relocation, and at the same time, the cluster downsizing cannot be achieved by reducing the number of machines, and the offline process cannot be well supported when the cluster faces offline of part of the machines (equivalent to cluster downsizing in nature).

[0068] Therefore, embodiments of the present application provide a storage resource scheduling method and device, an electronic device, and a storage medium, which improve the situation that the physical space is idle but the logical space is insufficient in the prior art by dynamically adjusting the logical space proportion of a data block, and improve the utilization rate of physical storage resources of the system.

[0069] Referring to Figure 1 A flowchart of a storage resource scheduling method provided by an optional embodiment of the present application is shown in FIG. 1. The method can be applied to, but is not limited to, a terminal device such as a mobile device or a personal computer, or a physical machine (such as a server or a cache device) corresponding to a cluster node of a distributed storage cluster.

[0070] As shown in FIG. 1, the storage resource scheduling method specifically includes the following steps. Figure 1

[0071] Step 101: Obtain a data block creation request.

[0072] The data block creation request can be a data block creation request triggered when a data write operation is initially performed in a process of writing and storing data in a front end to a storage system, or can be a request for creating a new data block triggered when the physical space of a current data corresponding block size (BlockSize) is full during the write operation.

[0073] In the embodiments of the present application, the block size BlockSize refers to a FullBlock Size, which represents the maximum value of the storage space that can be occupied by a single data block in the storage system.

[0074] Step 102: Create a data block and pre-allocate a logical space with a predetermined size value to the created data block.

[0075] ​For the data storage scene, the application proposes a technical idea of dynamically adjusting the logical space proportion of the data block. By dynamically adjusting the logical space proportion of the data block, the size of the logical space allocated and occupied by the data block in the storage system is closer to the actual size of the physical space occupied, and the utilization rate of the system physical storage resources is improved.

[0076] In this technical idea, when a data block is created in response to a data block creation request, a predetermined size value of logical space is pre-allocated to the created data block, which is smaller than the maximum value of the storage space that can be occupied by a single data block in the storage system, i.e., smaller than the block size BlockSize.

[0077] In this application, the size of the pre-allocated logical space for the data block is the minimum value of the logical space occupied in the data block life cycle (provided that the system does not adjust the predetermined size value), and the size of the logical space occupied by the data block can be dynamically adjusted between the predetermined size value of the pre-allocated logical space and BlockSize according to the data storage requirements of the data block.

[0078] In implementation, a logical space proportion (e.g., 80%, 50%, etc.) used when pre-allocating logical space can be determined and set according to actual needs, which represents the proportion of the size of the pre-allocated logical space to the block size BlockSize, and the pre-allocation of logical space for the data block is performed according to the set logical space proportion.

[0079] For example:

[0080] Suppose the original cluster size is: including x machines (or x nodes); now want to control the cluster size to y (x>y) to realize downsizing; that means the number of data blocks on a single machine in the cluster needs to be x / y times of the original, then the downsizing ratio z=y / x can be determined based on the calculation formula BlockSize*n=z*BlockSize*(x / y)*n, where n represents the number of original data blocks on a machine in the cluster.

[0081] The downsizing ratio z=y / x can be used as the logical space proportion used when pre-allocating logical space for the data block during the downsizing process. Based on the logical space proportion z, the number of data blocks on each machine in the cluster can be increased to x / y times of the original as much as possible, thereby realizing cluster downsizing.

[0082] In addition, in implementation, the predetermined size value of logical space needs to be allocated to the created data block from the remaining logical space of the storage system under the condition that the remaining logical space of the storage system is not less than the above-mentioned predetermined size value.

[0083] The remaining logical space of the storage system includes the logical space obtained by removing the logical space occupied by the data blocks from the total logical space of the storage system.

[0084] When writing data for storage, if the data amount of the data block exceeds the predetermined size value (it is easy to understand that the data amount of the data block does not exceed the BlockSize), then:

[0085] Step 103, the data block is additionally allocated with a logical space whose size is not less than the excess amount.

[0086] The excess amount is the amount by which the data amount of the data block exceeds the predetermined size value; and the total size of the pre-allocated logical space and the additionally allocated logical space does not exceed the maximum value (BlockSize).

[0087] In the dynamic addition of the logical space, an optional way can be that the data block is additionally allocated with a logical space whose size is equal to the above-mentioned excess amount.

[0088] Alternatively, the data block can be additionally allocated with a logical space whose size is greater than the above-mentioned excess amount. In order to reduce the logical space occupation of the storage system as much as possible under the premise of meeting the data storage requirement, in this embodiment, preferably, under the premise that the total size of the pre-allocated logical space and the additionally allocated logical space does not exceed the BlockSize, the difference (or ratio) between the size of the additionally allocated logical space of the data block and the above-mentioned excess amount is less than the set upper limit.

[0089] Step 104, the data of the data block which does not exceed the predetermined size value is written into the physical space corresponding to the pre-allocated logical space of the data block; and the data of the excess part is written into the physical space corresponding to at least part of the additionally allocated logical space of the data block.

[0090] In the case where the data block is additionally allocated with a logical space whose size is equal to the above-mentioned excess amount, the system allocates a physical space whose size is equal to that of the additionally allocated logical space from the remaining physical space of the underlying storage (such as a disk) resource, and writes the data of the excess part into the physical space, so as to meet the writing and storage requirement of the data of the excess part.

[0091] In the case that the append allocated logical space is allocated with a size greater than the above excess amount, the system allocates a physical space with a size equal to the above excess amount from the remaining physical space of the underlying storage (such as a disk) resource for the append allocated logical space (i.e., only part of the append allocated logical space is allocated with a corresponding physical space, instead of all of the append allocated logical space), and writes the data of the excess part into the physical space, thus satisfying the write and storage requirement of the data of the excess part.

[0092] Based on this, the system can support that the actual used physical space of a data block exceeds the size value (the upper limit is BlockSize) of the pre-allocated logical space during the data write and storage process. Thus, based on the logical space with the predetermined size value pre-allocated for the data block, there are two cases in the actual data storage:

[0093] One is that the actual used physical space of the data block does not exceed the predetermined size value of the pre-allocated logical space. In this case, the predetermined size value of the pre-allocated logical space is used to participate in the calculation of the overall logical space ratio (or remaining amount) in the system, and the logical space allocation for the data block is also based on this. That is, the logical space size occupied by the data block in this case is the size of the pre-allocated logical space, as shown in Figure 2

[0094] The other is that the actual used physical space of the data block exceeds the predetermined size value of the pre-allocated logical space. In this case, the size value of the actual used physical space of the data block is used to participate in the calculation of the overall logical space ratio (or remaining amount) in the system, and the logical space allocation for the data block is also based on this. That is, the logical space size occupied by the data block in this case is the size of the actual used physical space, as shown in Figure 3

[0095] In the storage system, the actual data amount of each data block is uneven, which can be any data amount between 0-BlockSize, and for the same data block, the data amount can change over time, for example, the data amount is less than the predetermined size value of the pre-allocated logical space at a certain historical period, and then the data amount exceeds the predetermined size value due to continuous data write.

[0096] ​​The present application can make the data block with smaller actual data amount occupy a smaller logical space of the system, and the data block with larger actual data amount (such as larger than the size of the pre-allocated logical space) occupy a larger logical space close to the data amount of the data block in the system, which obviously compresses the proportion of the overall logical space of the system. In addition, for the data block with larger actual data amount, the dynamic addition of the logical space makes the data block occupy a larger logical space close to the data amount requirement of the data block when the data amount exceeds the size of the pre-allocated logical space, rather than occupying the logical space in advance, which also compresses the proportion of the logical space of the system from the time perspective.

[0097] According to the above scheme, the storage resource scheduling method provided by the present application pre-allocates a logical space smaller than the maximum value of the logical space that can be occupied by a single data block in the storage system (i.e. the block size BlockSize) when creating a data block, and allows the actual physical space used by the data block to exceed the size of the pre-allocated logical space (the upper limit is BlockSize) when writing and storing data, and in the case of exceeding, the corresponding size of the logical space of the data exceeding part is temporarily and dynamically allocated to the data block to meet the logical space requirement of the data exceeding part. In this way, the logical space occupied by the data block in the system is closer to the actual physical space size occupied by the data block, which can effectively improve the situation of idle physical space but insufficient logical space in the traditional technology, and improve the physical storage resource utilization of the system.

[0098] In an optional embodiment of the present application, referring to Figure 4 The storage resource scheduling method can further include the following steps after step 104:

[0099] Step 105, monitoring the recycling operation of the physical space occupied by the data block.

[0100] When the data object in the data block is deleted, the system will perform the recycling operation (Compaction) to recycle the physical space corresponding to the deleted data object.

[0101] The step 105 can monitor the recycling operation of the physical space occupied by the data block in real time or periodically at a set time interval. The purpose of monitoring the recycling operation of the physical space in the storage system is to discover the recycling situation of the physical storage resource (such as the disk resource) in the storage system in time, so as to determine whether the target data block corresponding to the recycling operation needs to perform a matching logical space release operation.

[0102] Step 106, in the case of monitoring the recycling operation, determining whether the first size value of the logical space currently occupied by the target data block corresponding to the monitored recycling operation exceeds the predetermined size value of the logical space.

[0103] In combination with reference to Figure 2 and Figure 3 When the actual physical space used by the target data block does not exceed the size of the logical space pre-allocated for it, the logical space currently occupied by the target data block is the logical space obtained by pre-allocation; when the actual physical space used by the target data block exceeds the size of the logical space pre-allocated for it, the size of the logical space currently occupied by the target data block is the size of the actual physical space used by it.

[0104] Correspondingly, the first size value of the logical space currently occupied by the target data block may be equal to the predetermined size value of the pre-allocated logical space, or may exceed the predetermined size value.

[0105] If the first size value of the logical space currently occupied by the target data block is equal to the predetermined size value, in view of the feature that the size of the logical space pre-allocated for the data block in the present application is the minimum value of the logical space occupied by the data block in its life cycle, the logical space currently occupied by the target data block no longer has compression space for further compression.

[0106] If the first size value of the logical space currently occupied by the target data block exceeds the predetermined size value, after the recycling operation, two situations may occur:

[0107] 1) The actual physical space occupied by the data block after the recycling operation is smaller than the pre-allocated logical space;

[0108] 2) The actual physical space occupied by the data block after the recycling operation is still greater than the pre-allocated logical space.

[0109] For the above two situations, the logical space currently occupied by the target data block has compression space for further compression.

[0110] Therefore, in the case of monitoring the recycling operation, the present step 106 further determines whether the first size value of the logical space currently occupied by the target data block corresponding to the monitored recycling operation exceeds the predetermined size value of the pre-allocated logical space, to know whether the logical space currently occupied by the target data block has compression space for further compression.

[0111] Step 107, if it exceeds, according to the predetermined size value and the second size value of the actual physical space occupied by the target data block after the recycling operation is performed, releasing part of the logical space in the logical space currently occupied by the target data block.

[0112] If the first size value of the logical space currently occupied by the target data block exceeds the predetermined size value of the pre-allocated logical space, the logical space currently occupied by the target data block has compression space that can be further compressed, and thus, part of the logical space currently occupied by the target data block can be released to further compress the logical space occupied by the target data block.

[0113] If the second size value of the physical space actually occupied by the target data block is not greater than the predetermined size value of the pre-allocated logical space for the target data block, part of the logical space currently occupied by the target data block that exceeds the predetermined size value is released, which can be specifically understood with reference to FIG. 5 and FIG. 6. Figure 6

[0114] If the second size value of the physical space actually occupied by the target data block is greater than the predetermined size value, part of the logical space currently occupied by the target data block that exceeds the second size value is released, which can be specifically understood with reference to FIG. 5 and FIG. 6. Figure 7

[0115] If the first size value of the logical space currently occupied by the target data block is equal to the predetermined size value of the pre-allocated logical space for the target data block, the logical space currently occupied by the target data block does not have compression space that can be further compressed, and thus, the operation of releasing part of the logical space currently occupied by the target data block is not necessary.

[0116] The embodiment is directed to the physical space recycling operation of the storage system, and the monitoring, the condition determination, and the releasing operation of the logical space occupied by the data block when the condition is met are used to further compress the overall logical space occupancy of the storage system under the premise that the storage requirement of the data in the data block is met, and thus, the situation that the physical space is idle but the logical space is insufficient in the conventional technology is further improved, and the utilization rate of the physical storage resource of the system is improved.

[0117] Optionally, the predetermined size value of the pre-allocated logical space for the data block or the occupancy ratio of the predetermined size value to the BlockSize can be adjusted according to requirements. For example, if the amount of the released logical space in the system is still not enough based on the existing occupancy ratio (assuming 80%), the requirement cannot be met, and thus, the occupancy ratio can be further reduced (for example, to 50%) to further release the logical space.

[0118] Therefore, in an optional embodiment, the storage resource scheduling method provided in the present application can further include the following processing.

[0119] An adjustment instruction for the predetermined size value is obtained.

[0120] ​​The predetermined size value corresponding to the pre-allocated logical space of each data block in the storage system is adjusted to the size value indicated by the adjustment instruction based on the hot loading mode, and the logical space adjustment operation is performed on the logical space occupied by the data block in the storage system based on the adjusted size value of the pre-allocated logical space.

[0121] The hot loading mode refers to adjusting the predetermined size value corresponding to the pre-allocated logical space to the size value indicated by the adjustment instruction without restarting the machine, and performing the logical space adjustment operation on the logical space occupied by each data block in the storage system based on the adjusted value.

[0122] Specifically, if the logical space size value indicated by the adjustment instruction is smaller than the existing pre-allocated logical space size value of the system, then:

[0123] 1) For the case that the actual physical space occupied by the data block is not greater than the logical space size value indicated by the adjustment instruction, the logical space occupied by the data block is adjusted to the logical space size value indicated by the adjustment instruction.

[0124] 2) For the case that the actual physical space occupied by the data block is greater than the logical space size value indicated by the adjustment instruction and smaller than the existing pre-allocated logical space size value of the system, the logical space occupied by the data block is adjusted to the actual physical space size occupied by the data block.

[0125] 3) For the case that the actual physical space occupied by the data block is not smaller than the existing pre-allocated logical space size value of the system, the logical space occupied by the data block is not adjusted.

[0126] If the logical space size value indicated by the adjustment instruction is greater than the existing pre-allocated logical space size value of the system, then:

[0127] 1) For the case that the actual physical space occupied by the data block is smaller than the logical space size value indicated by the adjustment instruction, the logical space occupied by the data block is adjusted to the logical space size value indicated by the adjustment instruction.

[0128] 2) For the case that the actual physical space occupied by the data block is not smaller than the logical space size value indicated by the adjustment instruction, the logical space occupied by the data block is not adjusted.

[0129] After adjusting the pre-allocated logical space size of the data block in the storage system based on the adjustment instruction, the pre-allocated logical space of the new data block is performed based on the adjusted pre-allocated logical space size, and the logical space release operation is performed when the recycling operation is monitored and the determination condition is met.

[0130] The embodiment realizes adjusting the size of the pre-allocated logical space of the data block by the hot loading mode. When the proportion of the pre-allocated logical space to the BlockSize is further reduced after adjustment, the proportion of the overall logical space of the storage system can be further reduced based on the hot loading mode, the storage system can create a larger number of data blocks, and when the overall use proportion of the system physical storage space (such as the disk space) exceeds the proportion of the system pre-allocated logical space in the overall logical space, the overall logical space and the physical space of the system tend to be occupied.

[0131] When the method of the application is applied to the object storage system in the form of the distributed cluster described above, based on the scheme of dynamically adjusting the logical space allocation and the proportion of the application, the system can flexibly allocate the cluster logical space, flexibly adjust the storage resource utilization rate of the machines in the cluster, thereby supporting the system to flexibly reduce the cluster capacity, and concentrating the data of multiple machines in the cluster by using a smaller number of machines, thereby providing technical support for convenient machine relocation and machine offline.

[0132] Corresponding to the storage resource scheduling method described above, the embodiment of the application further provides a storage resource scheduling device, such as Figure 8 The structural schematic diagram of the device is shown, which can include:

[0133] The creating unit 801 is configured to obtain a data block creation request and create a data block.

[0134] The allocating unit 802 is configured to pre-allocate a logical space with a predetermined size value for the created data block, and when the data amount of the data block exceeds the predetermined size value when data is written for storage, the allocating unit 802 is configured to additionally allocate a logical space with a size value not less than the excess amount for the data block.

[0135] The writing unit 803 is configured to write the data of the data block that does not exceed the predetermined size value into the physical space corresponding to the pre-allocated logical space of the data block, and write the excess data into the physical space corresponding to the at least partially additionally allocated logical space of the data block.

[0136] The excess amount is the amount of the data amount of the data block exceeding the predetermined size value, and the total size of the pre-allocated logical space and the additionally allocated logical space of the data block does not exceed the maximum value.

[0137] In an optional implementation of the embodiment of the application, the allocating unit 802, in terms of pre-allocating a logical space with a predetermined size value for the created data block, is specifically configured to:

[0138] allocating, from a remaining logical space of the storage system, a logical space of the predetermined size value for the created data block in a case where the remaining logical space is not less than the predetermined size value;

[0139] The remaining logical space includes a result obtained by removing logical spaces occupied by the data blocks from a total logical space of the storage system.

[0140] If a size of the physical space actually occupied by the data block does not exceed the predetermined size value, a size of the logical space occupied by the data block is the predetermined size value; if the size of the physical space actually occupied by the data block exceeds the predetermined size value, the size of the logical space occupied by the data block is the size of the physical space actually occupied by the data block.

[0141] In an optional embodiment of the present application, referring to Figure 9 The storage resource scheduling apparatus can further include:

[0142] The monitoring unit 804 is configured to monitor a recycling operation on the physical space occupied by the data block.

[0143] The determining unit 805 is configured to determine, in a case where the recycling operation is monitored, whether a first size value of the logical space currently occupied by a target data block corresponding to the monitored recycling operation exceeds the predetermined size value.

[0144] The releasing unit 806 is configured to release, in a case where the first size value exceeds the predetermined size value, part of the logical space currently occupied by the target data block according to the predetermined size value and a second size value of the physical space actually occupied by the target data block after the recycling operation is performed.

[0145] In an optional embodiment of the present application, the releasing unit 806 is specifically configured to:

[0146] If the second size value is not greater than the predetermined size value, release part of the logical space currently occupied by the target data block that exceeds the predetermined size value.

[0147] If the second size value is greater than the predetermined size value, release part of the logical space currently occupied by the target data block that exceeds the second size value.

[0148] In an optional embodiment of the present application, referring to Figure 10 The storage resource scheduling apparatus can further include:

[0149] The adjusting unit 807 is configured to:

[0150] Obtain an adjusting instruction for the predetermined size value.

[0151] Based on the hot loading mode, the predetermined size value in the storage system is adjusted to the size value indicated by the adjustment instruction, to obtain an adjusted size value of the pre-allocated logical space; and based on the adjusted size value of the pre-allocated logical space, a matching logical space adjustment operation is performed on the logical space occupied by the data block in the storage system.

[0152] For the storage resource scheduling apparatus disclosed in the embodiments of the present application, since it corresponds to the storage resource scheduling method disclosed in the above method embodiments, the description is relatively simple, and the relevant similarities can be found in the above method embodiments, which will not be described in detail here.

[0153] The embodiments of the present application also disclose an electronic device, which can be, but is not limited to, a terminal device such as a mobile device or a personal computer, or a physical machine (such as a server or a cache device) corresponding to a cluster node of a distributed storage cluster.

[0154] As shown in the structural schematic diagram of the electronic device, at least includes: Figure 11 As shown in the structural schematic diagram of the electronic device, at least includes:

[0155] The memory 1101 is used to store a computer instruction set.

[0156] The computer instruction set can be implemented in the form of a computer program.

[0157] The memory 1101 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0158] The processor 1102 is used to implement the storage resource scheduling method of any one of the above method embodiments by executing the instruction set stored on the memory.

[0159] The processor 1102 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready programmable gate array (FPGA), or other programmable logic devices, etc.

[0160] In addition, the scheduling device can also include a communication interface, a communication bus and other components. The memory, the processor and the communication interface complete the communication among each other through the communication bus.

[0161] The communication interface is configured to schedule the communication between the device and other devices (e.g., a control device in a CDN scheduling system). The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0162] In the embodiment, the processor in the electronic device pre-allocates a logical space smaller than a maximum value of a logical space that can be occupied by a single data block in the storage system (i.e., a block size BlockSize) for the data block when the data block is created, while allowing the actual physical space used by the data block to exceed the size of the pre-allocated logical space (with an upper limit of BlockSize) when the data block performs data writing and storage, and temporarily dynamically additionally allocating a logical space corresponding to the data in the exceeding part to the data block to meet the logical space requirement of the data in the exceeding part when the exceeding occurs. In this way, the size of the logical space occupied by the data block in the system is closer to the actual size of the physical space occupied by the data block, which effectively improves the situation that the physical space is idle but the logical space is insufficient in the traditional technology, and improves the utilization rate of the physical storage resources of the system.

[0163] In addition, the embodiment of the present application also discloses a computer readable storage medium, which stores a computer instruction set. The computer instruction set is executed by a processor to implement the storage resource scheduling method disclosed in any of the above method embodiments.

[0164] Specifically, when the computer instruction set in the computer readable storage medium of the embodiment is executed by the processor, a logical space smaller than a maximum value of a logical space that can be occupied by a single data block in the storage system (i.e., a block size BlockSize) is pre-allocated for the data block when the data block is created, while allowing the actual physical space used by the data block to exceed the size of the pre-allocated logical space (with an upper limit of BlockSize) when the data block performs data writing and storage, and temporarily dynamically additionally allocating a logical space corresponding to the data in the exceeding part to the data block to meet the logical space requirement of the data in the exceeding part when the exceeding occurs. In this way, the size of the logical space occupied by the data block in the system is closer to the actual size of the physical space occupied by the data block, which effectively improves the situation that the physical space is idle but the logical space is insufficient in the traditional technology, and improves the utilization rate of the physical storage resources of the system.

[0165] It should be noted that each of the above-described examples of the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be mutually referred to.

[0166] For the convenience of description, the above system or device is described in various modules or units in terms of functions. Of course, in the implementation of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0167] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that in this paper, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0169] The above is only the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A storage resource scheduling method, characterized in that, The method comprises: obtaining a data block creation request; creating a data block, and pre-allocating a logical space with a predetermined size value for the created data block; the predetermined size value is smaller than a maximum value of a logical space that can be occupied by a single data block in a storage system, and the logical space is a block size space that the storage system allocates in advance for the data block when creating the data block; when writing data for storage, if the data amount of the data block exceeds the predetermined size value, then: allocating additional logical space with a size value not less than the excess amount for the data block; writing data of the data block that does not exceed the predetermined size value into a physical space corresponding to the pre-allocated logical space of the data block; writing the excess data into a physical space corresponding to at least part of the additional allocated logical space of the data block, and the physical space is a disk space actually occupied by the data block; wherein the excess amount is the amount by which the data amount of the data block exceeds the predetermined size value, and the total size of the pre-allocated logical space and the additional allocated logical space for the data block does not exceed the maximum value; the pre-allocating a logical space with a predetermined size value for the created data block comprises: allocating the logical space with the predetermined size value for the created data block from the remaining logical space of the storage system, if the remaining logical space is not less than the predetermined size value; wherein the remaining logical space comprises a result obtained by removing the logical space occupied by each data block from the total logical space of the storage system; if the size of the physical space actually occupied by the data block does not exceed the predetermined size value, then the size of the logical space occupied by the data block is the predetermined size value; if the size of the physical space actually occupied by the data block exceeds the predetermined size value, then the size of the logical space occupied by the data block is the size value of the physical space actually occupied by the data block.

2. The method of claim 1, wherein, after creating the data block, the method further comprises: monitoring a recycling operation on the physical space occupied by the data block; determining whether the first size value of the logical space currently occupied by the target data block corresponding to the monitored recycling operation exceeds the predetermined size value; if so, releasing part of the logical space currently occupied by the target data block according to the predetermined size value and the second size value of the physical space actually occupied by the target data block after the recycling operation is performed.

3. The method of claim 2, wherein, the releasing part of the logical space currently occupied by the target data block according to the predetermined size value and the second size value of the physical space actually occupied by the target data block after the recycling operation is performed comprises: if the second size value is not greater than the predetermined size value, releasing the part of the logical space currently occupied by the target data block that exceeds the predetermined size value; if the second size value is greater than the predetermined size value, releasing the part of the logical space currently occupied by the target data block that exceeds the second size value.

4. The method of claim 2, wherein, the method further comprises: obtaining an adjustment instruction for the predetermined size value; Adjust the predetermined size value in the storage system to a size value indicated by the adjustment instruction based on a hot loading mode, to obtain an adjusted size value of the pre-allocated logical space; and perform a matching logical space adjustment operation on the logical space occupied by the data block in the storage system based on the adjusted size value of the pre-allocated logical space.

5. A storage resource scheduling apparatus, characterized by comprising: Comprise: A creating unit configured to obtain a data block creation request and create a data block; An allocating unit configured to pre-allocate a logical space with a predetermined size value for the created data block; The predetermined size value is smaller than a maximum value of a logical space that can be occupied by a single data block in the storage system, and when data is written for storage, if the data amount of the data block exceeds the predetermined size value, a logical space with a size value not smaller than the excess amount is additionally allocated for the data block, the logical space being a block size space pre-allocated for the data block by the storage system when the data block is created; A writing unit configured to write data of the data block that does not exceed the predetermined size value into a physical space corresponding to the pre-allocated logical space of the data block; Write the data of the excess part into a physical space corresponding to at least part of the additionally allocated logical space of the data block, the physical space being a disk space actually occupied by the data block; The excess amount is the amount by which the data amount of the data block exceeds the predetermined size value; and the total size of the pre-allocated logical space and the additionally allocated logical space does not exceed the maximum value; The allocating unit, in terms of pre-allocating a logical space with a predetermined size value for the created data block, is specifically configured to: Allocate the logical space with the predetermined size value for the created data block from a remaining logical space of the storage system if the remaining logical space is not lower than the predetermined size value; The remaining logical space comprises a result obtained by removing the logical space occupied by each data block from the total logical space of the storage system; If the size of the physical space actually occupied by the data block does not exceed the predetermined size value, the size of the logical space occupied by the data block is the predetermined size value; and if the size of the physical space actually occupied by the data block exceeds the predetermined size value, the size of the logical space occupied by the data block is the size value of the physical space actually occupied by the data block.

6. The apparatus of claim 5, wherein, Further comprise: A monitoring unit configured to monitor a recycling operation on the physical space occupied by the data block; A determining unit configured to determine whether a first size value of the logical space currently occupied by a target data block corresponding to the monitored recycling operation exceeds the predetermined size value in a case where the recycling operation is monitored; A releasing unit configured to release part of the logical space currently occupied by the target data block according to the predetermined size value and a second size value of the physical space actually occupied by the target data block after the recycling operation is executed in a case where the first size value exceeds the predetermined size value.

7. The apparatus of claim 6, wherein, The releasing unit is specifically configured to: Release the part of the logical space currently occupied by the target data block that exceeds the predetermined size value if the second size value is not larger than the predetermined size value. If the second size value is greater than the predetermined size value, releasing a part of the logical space currently occupied by the target data block exceeding the second size value.

8. The apparatus of claim 6, wherein, Further comprising: an adjusting unit, configured to: obtain an adjusting instruction for the predetermined size value; adjust the predetermined size value in the storage system to a size value indicated by the adjusting instruction based on a hot loading manner, to obtain an adjusted size value of the pre-allocated logical space; and perform a matching logical space adjusting operation on the logical space occupied by the data block in the storage system based on the adjusted size value of the pre-allocated logical space.

9. An electronic device, comprising: comprising: a memory, configured to store a computer instruction set; a processor, configured to execute the computer instruction set stored on the memory, to implement the storage resource scheduling method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer instruction set, and the computer instruction set is executed by the processor to implement the storage resource scheduling method according to any one of claims 1-4.

Images