A method and system for optimizing block storage scheduling strategy

Abstract

The application relates to the technical field of cloud computing, in particular to a block storage scheduling strategy optimization method which comprises the following steps: deploying an Openstack environment; completing the butt joint of Cinder and two pools supporting thin provisioning, requiring that the two pools correspond to two volume backends but use the same volume type; beneficial effects are as follows: the block storage scheduling strategy optimization method and system 1 can schedule to a suitable storage pool under the condition that an environment butt joints multiple thin storage pools and the sizes of the storage pools are inconsistent, can guarantee that the scheduling direction can make the actual residual spaces of different storage pools of the same volume type similar, still supports the scheduling direction making the actual residual spaces of different storage pools of the same volume type similar for a storage backend which cannot set max_over_subscription_ratio as auto, and max_over_subscription_ratio can also be set as auto after using the optimization strategy, without influencing original functions.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, specifically to a method and system for optimizing block storage scheduling strategies. Background Technology

[0002] In government cloud environments, it's common for Cinder to connect to multiple storage systems or multiple thin-walled storage pools within the same storage system. When these storage pools all use the same volume type, scheduling between different storage pools becomes necessary. The scheduling method involves checking the free capacity of all pools and selecting the pool with the smallest free capacity for scheduling.

[0003] In existing technologies, the free capacity in the current scheduling logic is calculated as "total capacity * maximum thin provisioning ratio - provisioned capacity". When the storage pool sizes are inconsistent, this calculation method may cause the scheduler to always schedule to the storage pool with the larger capacity. The community provides a method to automatically adjust max_over_subscription_ratio. By setting max_over_subscription_ratio = auto, Cinder will automatically calculate the max_over_subscription_ratio value based on the pool's provisioned_capacity_gb, total_capacity_gb, and free_capacity_gb attributes.

[0004] However, the above only applies to storage systems that support setting `max_over_subscription_ratio` to `auto`. Some SAN storage backends can only set `max_over_subscription_ratio` to a specific number. There are also some special cases, such as Ceph. If the `rbd_exclusive_cinder_pool` parameter is enabled and `max_over_subscription_ratio` is set to `auto`, the calculated value of `max_over_subscription_ratio` may be negative due to an imperfect high availability mechanism of the Cinder scheduler, resulting in scheduling failure. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for optimizing block storage scheduling strategies to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for optimizing block storage scheduling strategies, the method comprising the following steps:

[0007] Deploy an OpenStack environment;

[0008] Complete the integration between Cinder and two Pools that support thin provisioning. These two Pools should correspond to two volume backends, but use the same volume type.

[0009] In Cinder, the calculation method for available capacity in CapacityWeigher is changed to "(free_capacity_gb-math.floor(total*reserved))*max_over_subscription_ratio";

[0010] Use the OpenStack client to create a 100GB volume of this volume type and check if the volume is scheduled to a storage pool with ample remaining storage space.

[0011] Preferably, a multi-node environment is deployed, with well-planned control nodes and compute nodes. When deploying the OpenStack environment, the OpenStack environment is divided into different nodes according to their functions, such as control nodes, compute nodes, and storage nodes.

[0012] Preferably, two storage pools of different sizes need to be prepared, for example, Pool0 is 26T and Pool1 is 13T. Thick standby volumes are manually created to occupy storage space. For example, if a 18.8T thick standby volume is created in Pool0, its available space is 26.18 - 18.8, or 7.38T; if a 2.25T thick standby volume is created in Pool1, its available space is 13.09 - 2.25, or 10.84T.

[0013] Preferably, the process of creating a volume using Cinder is as follows: The Cinder API initiates a workflow, and upon completion, notifies the Cinder Scheduler. The Cinder Scheduler executes a scheduling algorithm, selecting the optimal storage node through filters and weights. Specifically, it undergoes a series of filtering steps using AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter, and CapacityWeigher, ultimately scheduling the volume to a suitable storage backend. The Cinder Scheduler then sends a message to the Cinder Volume, instructing it to create the volume. The Cinder Volume also initiates a Flow to complete the volume creation process, and the volume is created on the storage backend selected by the Cinder Scheduler.

[0014] Preferably, after creating a 100G volume, check the volume's properties and use os-vol-host-attr:host to see which storage pool the volume will be scheduled to.

[0015] A system for optimizing block storage scheduling strategies, comprising a deployment module, an interface module, a computing module, and a creation module;

[0016] The deployment module is used to deploy an OpenStack environment;

[0017] The interface module is used to connect Cinder with two Pools that support thin provisioning. These two Pools must correspond to two volume backends, but use the same volume type.

[0018] The calculation module is used to change the calculation method of available capacity in CapacityWeigher in Cinder to "(free_capacity_gb-math.floor(total*reserved))*max_over_subscription_ratio";

[0019] Create a module to use the OpenStack client to create a 100GB volume of this volume type and check if the volume is scheduled to a storage pool with a large amount of remaining storage space.

[0020] Preferably, in the deployment module, a multi-node environment is deployed, control nodes and computing nodes are planned, and when deploying the OpenStack environment, the OpenStack environment is divided into different nodes according to function, such as control nodes, computing nodes and storage nodes.

[0021] Preferably, in the docking module, two storage pools of different sizes need to be prepared, for example, Pool0 is 26T and Pool1 is 13T; thick standby volumes are manually created to occupy storage space. For example, if a 18.8T thick standby volume is created in Pool0, its available space is 26.18-18.8, or 7.38T; if a 2.25T thick standby volume is created in Pool1, its available space is 13.09-2.25, or 10.84T.

[0022] Preferably, in the computing module, the process of creating a volume using Cinder is as follows: Cinder-api initiates a workflow and notifies Cinder-scheduler upon completion; Cinder-scheduler executes a scheduling algorithm, selecting the optimal storage node through Filter and Weight; specifically, it undergoes layer-by-layer filtering using AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter, and CapacityWeigher, ultimately being scheduled to a suitable storage backend; Cinder-scheduler sends a message to Cinder-volume, instructing it to create the volume. Cinder-volume also initiates a Flow to complete the volume creation process, and the volume is created on the storage backend selected by Cinder-scheduler.

[0023] Preferably, in the creation module, after creating a 100G volume, the volume's attributes are viewed, and the storage pool to which the volume is scheduled is determined based on os-vol-host-attr:host.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] The block storage scheduling strategy optimization method and system proposed in this invention enable scheduling to a suitable storage pool when the environment is connected to multiple thin storage pools with inconsistent sizes, ensuring that the scheduling direction makes the actual remaining space of different storage pools of the same volume type similar; for storage backends that cannot set max_over_subscription_ratio to auto, it still supports scheduling direction that makes the actual remaining space of different storage pools of the same volume type similar; after using the optimization strategy, max_over_subscription_ratio can also be set to auto without affecting the original functionality. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In the description of this invention, it should be noted that the terms "center," "middle," "upper," "lower," "left," "right," "inner," "outer," "top," "bottom," "side," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "a," "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0030] For purposes of simplicity and illustration, the principles of the embodiments are described primarily by way of example. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that these embodiments may not be limited to these specific details in practice. In some instances, well-known methods and structures have not been described in detail to avoid unnecessarily obscuring these embodiments. Furthermore, all embodiments can be used in combination with each other.

[0031] Example 1

[0032] Please see Figure 1 The present invention provides a technical solution: a method for optimizing block storage scheduling strategy, the method comprising the following steps:

[0033] 1. Deploy an OpenStack environment. It is recommended to deploy a multi-node environment and plan the control nodes and compute nodes accordingly.

[0034] When deploying an OpenStack environment, the OpenStack environment is divided into different nodes according to their functions, such as control nodes, compute nodes, and storage nodes. The purpose of doing this is to reduce the pressure on the cluster, while making the cluster architecture clearer and making it easier to expand the cluster in the future.

[0035] 2. Integrate Cinder with two pools that support thin provisioning. These two pools should correspond to two volume backends, but use the same volume type.

[0036] Two storage pools of different sizes need to be prepared, for example, Pool0 is 26T and Pool1 is 13T. Thick provisioned volumes are manually created to occupy storage space. For example, if a 18.8T thick provisioned volume is created in Pool0, its available space is 26.18 - 18.8, or 7.38T; if a 2.25T thick provisioned volume is created in Pool1, its available space is 13.09 - 2.25, or 10.84T. The purpose of this is to test whether the newly created volume can be scheduled to the storage pool with more remaining storage space, rather than to the storage pool with a larger capacity.

[0037] 3. In Cinder, the calculation method for available capacity in CapacityWeigher will be changed to "(free_capacity_gb - math.floor(total * reserved)) * max_over_subscription_ratio".

[0038] The current process for creating a volume using Cinder is as follows: The Cinder API initiates a workflow, and upon completion, notifies the Cinder scheduler. The Cinder scheduler executes a scheduling algorithm, selecting the optimal storage node through filters and weights. Specifically, it involves multiple layers of filtering using AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter, and CapacityWeigher, ultimately scheduling the volume to a suitable storage backend. The Cinder scheduler then sends a message to the Cinder volume, instructing it to create the volume. The Cinder volume also initiates a flow to complete the volume creation process, and the volume is created within the storage backends selected by the Cinder scheduler. The relevant filters and weights in this patent are CapacityFilter and CapacityWeigher. CapacityFilter selects backends with sufficient available capacity, while CapacityWeigher selects the backend with the largest capacity from these selected backends. The original formula, total * max_over_subscription_ratio - provisioned_capacity - total * reserved, is flawed. Subtracting math.floor(total * reserved) from the formula fails to account for the reserved capacity also being multiplied by the max_over_subscription_ratio coefficient, ultimately causing the reserved_percentage parameter to have no real effect. For example, a pool has a total_capacity_gb of 100GB and a max_over_subscription_ratio of 20. If a 2GB thick provisioned volume is created in this pool, the pool's provisioned_capacity_gb will be 2GB and free_capacity_gb will be 98GB. If reserved_percentage is set to 10, the original formula yields 100 * 20 - 2 - (100 * 0.1) = 2000 - 2 - 10 = 1988.This means the pool has 1988GB of free space. However, if `reserved_percentage` is set, the desired free capacity is 100*20-2-(100*0.1)*20=2000-2-200=1798GB. This ensures that 10% of the storage space is reserved. The formula used in `CapacityWeigher` to calculate free capacity differs from that used in `CapacityFilter`. In `CapacityFilter`, the formula is "(free_capacity_gb-math.floor(total*reserved))*max_over_subscription_ratio"; while in `CapacityWeigher`, it's "total_capacity_gb*max_over_subscription_ratio-provisioned_capacity_gb-math.floor(total*reserved)". Therefore, based on the above two considerations, the original formula was modified, and the calculation method of available capacity in CapacityWeigher was changed to "(free_capacity_gb-math.floor(total*reserved))*max_over_subscription_ratio", so that scheduling always schedules to the storage pool with more remaining storage space.

[0039] 4. Use the OpenStack client to create a 100GB volume of this volume type and check if the volume is scheduled to a storage pool with ample free space.

[0040] After creating the 100GB volume, you can view its properties. The `os-vol-host-attr:host` attribute indicates which storage pool the volume is scheduled to. Alternatively, you can log in to the storage pool to view this information.

[0041] Example 2

[0042] A system for optimizing block storage scheduling strategies, comprising a deployment module, an interface module, a computing module, and a creation module;

[0043] The deployment module is used to deploy an OpenStack environment; it deploys multi-node environments, plans control nodes and compute nodes, and divides the OpenStack environment into different nodes according to their functions, such as control nodes, compute nodes, and storage nodes.

[0044] The interface module is used to connect Cinder with two pools that support thin provisioning. These two pools must correspond to two volume backends but use the same volume type. For storage, two storage pools of different sizes are required, for example, Pool0 is 26T and Pool1 is 13T. Thick provisioned volumes are manually created to occupy storage space. For example, creating an 18.8T thick provisioned volume in Pool0 will give it 26.18 - 18.8T, or 7.38T, of usable space; creating a 2.25T thick provisioned volume in Pool1 will give it 13.09 - 2.25T, or 10.84T, of usable space.

[0045] The calculation module transforms the calculation method for available capacity in CapacityWeigher within Cinder into "(free_capacity_gb-math.floor(total*reserved))*max_over_subscription_ratio". The process of creating a volume using Cinder is as follows: The cinder-api initiates a workflow and notifies the cinder-scheduler upon completion; the cinder-scheduler executes a scheduling algorithm, selecting the optimal storage node through filters and weights; specifically, it undergoes a multi-layered filtering process using AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter, and CapacityWeigher, ultimately scheduling the volume to a suitable storage backend; the cinder-scheduler sends a message to cinder-volume, instructing it to create the volume. cinder-volume also initiates a flow to complete the volume creation process, and the volume is created on the storage backend selected by the cinder-scheduler.

[0046] Create a module to use the OpenStack client to create a 100GB volume of this volume type and check if the volume is scheduled to a storage pool with a large amount of remaining storage space. After creating the 100GB volume, check the volume's attributes and use os-vol-host-attr:host to see which storage pool the volume is scheduled to.

[0047] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing block storage scheduling strategies, characterized in that: The block storage scheduling strategy optimization method comprises the following steps: Deploy an Openstack environment; Complete the connection of Cinder and 2 pools supporting thin provisioning, which requires that the 2 pools correspond to two volume backends, but use the same volume type; Change the calculation method of the available capacity in the CapacityWeigher in Cinder to "(free_capacity_gb-math.floor(total*reserved))*max_over_subscript ion_ratio"; Use the openstack client to create a 100G volume with this volume type, and check whether the volume is scheduled to the storage pool with the largest remaining space; The process of creating a volume (volume) with cinder is as follows: cinder-api starts a workflow, notifies cinder-scheduler after completing the work; cinder-scheduler executes the scheduling algorithm, selects the optimal storage node through Filter and Weigher; specifically, it is filtered layer by layer through AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter and CapacityWeigher, and finally scheduled to the appropriate storage backend; cinder-scheduler sends a message to cinder-volume to create a volume; cinder-volume also starts a Flow to complete the volume creation work, and the volume will be created in the storage backend selected by cinder-scheduler. 2.The method of claim 1, wherein: Deploy a multi-node environment, plan the control node and the compute node, and deploy the Openstack environment. When deploying the Openstack environment, divide the Openstack environment into different nodes according to the functions, including the control node, the compute node and the storage node. 3.The method of claim 2, wherein: Two storage pools of different sizes need to be prepared on the storage, and thick provisioning volumes are manually created on the storage to occupy the storage space. 4.The method of claim 1, wherein: After creating a 100G volume, check the properties of the volume, and according to os-vol-host-attr:host, it can be seen that the volume is scheduled to which storage pool.

5. A system using the block storage scheduling policy optimization method of any one of claims 1-4, characterized in that: The system comprises a deployment module, an interface module, a calculation module and a creation module; The deployment module is used for deploying an Openstack environment; The interface module is used for completing the connection of Cinder and 2 pools supporting thin provisioning, which requires that the 2 pools correspond to two volume backends, but use the same volume type; The calculation module changes the calculation method of the available capacity in the CapacityWeigher in Cinder to (free_capacity_gb-math.floor(total*reserved))*max_over_subscript ion_ratio”; Create a module to create a 100G volume using the openstack client with this volume type, and check if the volume is scheduled to the storage pool with more remaining space. In the computing module, the process of creating a volume (volume) by cinder is as follows: cinder-api starts a workflow, and after the work is completed, it notifies cinder-scheduler; cinder-scheduler executes the scheduling algorithm, selects the optimal storage node through Filter and Weigher; specifically, it is filtered layer by layer through AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter and CapacityWeigher, and finally scheduled to the appropriate storage backend; cinder-scheduler sends a message to cinder-volume to create a volume; cinder-volume also starts a Flow to complete the volume creation work, and the volume will be created in the storage backend selected by cinder-scheduler. 6.The system for block storage scheduling policy optimization of claim 5, wherein: In the deployment module, a multi-node environment is deployed, and the control node and the computing node are planned. When deploying the Openstack environment, the Openstack environment is divided into different nodes according to the function, including the control node, the computing node and the storage node. 7.The system for block storage scheduling policy optimization of claim 6, wherein: In the docking module, two storage pools with different sizes need to be prepared on the storage, and thick backup volumes are manually created on the storage to occupy the storage space. 8.The system of claim 6, wherein: In the creating module, after creating a 100G volume, the properties of the volume are checked, and according to os-vol-host-attr:host, it can be seen that the volume is scheduled to which storage pool.

Images