Volume mapping management method, apparatus and device

Abstract

The present disclosure provides a volume mapping management method, which can be applied to the technical field of computers. The method comprises: in response to a mapping request of a new volume mapping host, obtaining hardware configuration data of a distributed storage cluster, wherein the distributed storage cluster comprises a plurality of storage nodes, and the plurality of storage nodes are used for processing read and write tasks from the host; determining respective resource consumption information of the plurality of storage nodes based on storage configuration data of historical volumes having a home relationship with the plurality of storage nodes; and determining a home node for the new volume from the plurality of storage nodes based on the hardware configuration data and the resource consumption information, so that the home node processes the read and write tasks from the host after the new volume is mapped to the host. The present disclosure also provides a volume mapping management device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to a volume mapping management method, apparatus, and device. Background Technology

[0002] In a distributed storage cluster, each volume has a home node attribute, meaning each volume belongs to a specific storage node within the cluster. When creating a volume, the home node must be specified. Under normal circumstances, only the storage node to which the volume belongs can handle read and write tasks from the corresponding host in a distributed storage cluster.

[0003] However, when multiple volumes are created and evenly distributed across the storage nodes of a distributed storage cluster, it is difficult to achieve load balancing across the entire distributed storage cluster because each volume handles different front-end services and these services have different performance requirements for the back-end storage. This affects the reliability and service performance of the distributed storage cluster. Summary of the Invention

[0004] In view of the above problems, this disclosure provides volume mapping management methods, apparatus, devices, media, and program products.

[0005] According to a first aspect of this disclosure, a volume mapping management method is provided, comprising: in response to a mapping request for a new volume mapping host, obtaining hardware configuration data of a distributed storage cluster, wherein the distributed storage cluster includes multiple storage nodes, the multiple storage nodes being used to process read and write tasks from the host; determining resource consumption information of each of the multiple storage nodes based on storage configuration data of historical volumes that have affiliation with the multiple storage nodes; and determining, based on the hardware configuration data and resource consumption information, an affiliation node for the new volume from the multiple storage nodes, so that the affiliation node can process read and write tasks from the host after the new volume is mapped to the host.

[0006] According to embodiments of this disclosure, storage configuration data includes the type of historical volumes; based on the storage configuration data of historical volumes that have affiliation with multiple storage nodes, determining the resource consumption information of each of the multiple storage nodes includes: determining the resource consumption information of each of the multiple storage nodes based on the type of historical volumes and a first predetermined mapping relationship, wherein the first predetermined mapping relationship characterizes the mapping relationship between the type of volume and the resource consumption information.

[0007] According to embodiments of this disclosure, the volume mapping management method further includes: determining resource consumption information of the ownership node of different types of volumes based on a historical data set, wherein the historical data set is obtained by testing read and write tasks for different types of volumes; and determining a first predetermined mapping relationship based on different types of volumes and resource consumption information.

[0008] According to embodiments of this disclosure, determining the home node for a new volume from multiple storage nodes based on hardware configuration data and resource consumption information includes: determining the resource utilization rate of each of the multiple storage nodes based on hardware configuration data and resource consumption information; and determining the home node for a new volume from the multiple storage nodes based on the resource utilization rate of each of the multiple storage nodes.

[0009] According to embodiments of this disclosure, resource consumption information indicates theoretical resource consumption; determining the resource utilization rate of each of the multiple storage nodes based on hardware configuration data and resource consumption information includes: determining the hardware performance estimate of each of the multiple storage nodes based on the hardware configuration data of each of the multiple storage nodes and a second predetermined mapping relationship, wherein the second predetermined mapping relationship characterizes the mapping relationship between hardware configuration data and hardware performance estimate; determining the theoretical resource consumption rate of each of the multiple storage nodes based on the theoretical resource consumption of each of the multiple storage nodes and a preset coefficient used to determine the theoretical resource consumption rate; and determining the resource utilization rate of each of the multiple storage nodes based on the ratio of the theoretical resource consumption rate of each of the multiple storage nodes to the hardware performance estimate.

[0010] According to embodiments of this disclosure, the volume mapping management method further includes: obtaining hardware usage information of multiple storage nodes during a processing cycle, wherein the multiple storage nodes process read and write tasks from the host during the processing cycle; and updating a preset coefficient based on the hardware usage information and resource utilization of the multiple storage nodes.

[0011] According to embodiments of this disclosure, the storage configuration data includes the read / write performance of historical volumes; based on the resource utilization of multiple storage nodes, determining the home node for adding a new volume from among the multiple storage nodes includes: if it is determined that the resource utilization of all storage nodes among the multiple storage nodes is less than a security threshold, determining the storage node with the lowest resource utilization among the multiple storage nodes as the home node for adding a new volume; if it is determined that there is a storage node among the multiple storage nodes with a resource utilization greater than or equal to the security threshold, determining the storage node with a resource utilization greater than or equal to the security threshold as a volume migration node; and based on the read / write performance of historical volumes, migrating at least one historical volume belonging to the volume migration node to a storage node with a resource utilization less than the security threshold.

[0012] According to embodiments of this disclosure, the volume mapping management method further includes: in response to a mapping request, obtaining a configuration file, wherein the configuration file indicates the performance requirements of the terminal for the distributed storage cluster; and determining a security threshold based on the configuration file.

[0013] A second aspect of this disclosure provides a volume mapping management apparatus, comprising: an acquisition module, configured to acquire hardware configuration data of a distributed storage cluster in response to a mapping request for a new volume mapping host, wherein the distributed storage cluster includes multiple storage nodes, the multiple storage nodes being used to process read and write tasks from the host; a first determination module, configured to determine resource consumption information of each of the multiple storage nodes based on storage configuration data of historical volumes that have affiliation with the multiple storage nodes; and a second determination module, configured to determine, based on the hardware configuration data and resource consumption information, the affiliation node for the new volume from the multiple storage nodes, so that the affiliation node can process read and write tasks from the host after the new volume is mapped to the host.

[0014] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this disclosure, when a new volume is mapped to a host, the owner node for the new volume is determined from multiple storage nodes in the distributed storage cluster based on the hardware configuration of the storage node and the resource consumption of historical volumes with ownership relationships with the storage node. Since the owner node for the new volume is determined when the new volume is mapped to the host, and is determined from the distributed storage cluster based on the hardware configuration of the storage node and the resource consumption of historical volumes with ownership relationships with the storage node, the load on the storage nodes can be reasonably allocated, which is beneficial for load balancing of the distributed storage cluster and improves the stability and service performance of the storage system. Compared to specifying the owner node when creating a volume, it avoids the problem of unbalanced storage node load caused by increases in the number of volumes, expansion of volume capacity, and changes in host read / write tasks during the processing of read / write tasks. Furthermore, since the resource consumption information of the storage node is a theoretical consumption determined based on storage configuration data, the response speed of the storage system can be improved. Attached Figure Description

[0018] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration schematically depicts application scenarios of volume mapping management methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure;

[0020] Figure 2 A flowchart illustrating a volume mapping management method according to an embodiment of the present disclosure is shown schematically;

[0021] Figure 3 A schematic diagram illustrating a first predetermined mapping relationship according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram illustrating the determination of storage node resource utilization according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A schematic diagram illustrating the updating of preset coefficients according to an embodiment of the present disclosure is shown.

[0024] Figure 6 A schematic block diagram of a volume mapping management apparatus according to an embodiment of the present disclosure is shown; and

[0025] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a volume mapping management method according to an embodiment of the present disclosure. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without this specific detail. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When information uses expressions such as "at least one of A, B, and C", it should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0030] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0032] In implementing the embodiments of this disclosure, it was found that storage node load balancing can ensure that under a large amount of data storage and access requests, the load is evenly distributed among the storage nodes, improving the stability and service performance of the storage system. However, in the event of subsequent business operations, such as an increase in the number of volumes, expansion of volume capacity, and changes in host read / write tasks, specifying the owner node when creating a volume can lead to uneven load distribution among the storage nodes when handling read / write tasks from the host.

[0033] Embodiments of this disclosure provide a volume mapping management method, including: in response to a mapping request for a new volume mapping host, obtaining hardware configuration data of a distributed storage cluster, wherein the distributed storage cluster includes multiple storage nodes, and the multiple storage nodes are used to process read and write tasks from the host; determining resource consumption information of each of the multiple storage nodes based on storage configuration data of historical volumes that have affiliation with the multiple storage nodes; and determining the affiliation node for the new volume from the multiple storage nodes based on the hardware configuration data and resource consumption information, so that the affiliation node can process read and write tasks from the host after the new volume is mapped to the host.

[0034] Figure 1 The illustration schematically depicts an application scenario of a volume mapping management method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0035] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a host 101, a management node 102, and a distributed storage cluster 103.

[0036] The interaction between host 101, management node 102, and distributed storage cluster 103 is achieved through a network, which serves as the medium for providing communication links between host 101, management node 102, and distributed storage cluster 103. The network can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0037] Host 101 can refer to a network node authorized to access a specific volume, such as a server, workstation, or other type of computer device. An operating system and other applications are installed inside host 101. During volume mapping, host 101 can act as an accessor and user, communicating with the management node and storage node in the storage system through specific network protocols and requesting access to data on the volume. For example, host 101 can interact with the volume belonging to the node through a file system or storage driver to perform read and write operations on data.

[0038] The distributed storage cluster 103 may include multiple storage nodes, which can be used as the owner nodes of volumes and handle read and write tasks from the host 101. A storage node can be the owner node of multiple volumes. A storage node may include storage devices, such as a redundant array of independent disks, a disk cluster, or one or more interconnected disk drives of storage.

[0039] The management node 102 can be used to manage the storage nodes and hosts 101 in the distributed storage cluster 103. The management node 102 may include servers, other types of computer equipment, etc.

[0040] It should be noted that the volume mapping management method provided in this embodiment can generally be executed by the management node 102. Correspondingly, the volume mapping management device provided in this embodiment can generally be located in the management node 102. The volume mapping management method provided in this embodiment can also be executed by a management node or management node cluster that is different from the management node 102 and can communicate with the host 101 and / or the distributed storage cluster 103. Correspondingly, the volume mapping management device provided in this embodiment can also be located in a management node or management node cluster that is different from the management node 102 and can communicate with the host 101 and / or the distributed storage cluster 103.

[0041] It should be understood that Figure 1 The number of hosts, management nodes, and distributed storage clusters shown is merely illustrative. Depending on implementation needs, any number of hosts, management nodes, and distributed storage clusters can be included.

[0042] The following will be based on Figure 1 The described scene, through Figures 2-6 The volume mapping management method of the disclosed embodiments will be described in detail.

[0043] Figure 2 A flowchart illustrating a volume mapping management method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the volume mapping management method of this embodiment includes operations S210 to S230.

[0045] In operation S210, in response to a mapping request for a new volume mapping host, the hardware configuration data of the distributed storage cluster is obtained.

[0046] In operation S220, based on the storage configuration data of historical volumes that have affiliation with multiple storage nodes, the resource consumption information of each of the multiple storage nodes is determined.

[0047] In operation S230, based on hardware configuration data and resource consumption information, the home node for the new volume is determined from multiple storage nodes, so that the home node can handle read and write tasks from the host after the new volume is mapped to the host.

[0048] In this embodiment, a newly added volume may be a volume created based on creation configuration information in a storage or data management scenario, but without a designated home node. Storage or data management scenarios may include, but are not limited to, persistent storage requirements, data backup and recovery, storage expansion, performance optimization, data isolation, and management. Creation configuration information may include, but is not limited to, host attribute information and volume size. When storage services need to be provided to a host, the newly added volume can be mapped to the corresponding host based on the host's attribute information. The mapping request may include, but is not limited to, host attribute information. This attribute information may be, for example, a host number. The distributed storage cluster may include multiple storage nodes, which can be used to handle read and write tasks from the host. The hardware configuration data of the distributed storage cluster may include the hardware configuration data of each storage node. The hardware configuration data can be used to indicate the storage configuration of the storage nodes.

[0049] For example, when an application, database, or file system needs to provide persistent storage, a new volume can be created based on the creation configuration information. When storage services need to be provided to a host with a corresponding host number, the host can send a mapping request to the management node. After receiving the mapping request, the management node can verify the host number, and if the verification is successful, obtain the hardware configuration data of the distributed storage cluster.

[0050] In this embodiment of the disclosure, each of the multiple storage nodes can serve as the home node for at least one historical volume. A historical volume can be understood as a volume that has been configured with a home node. Storage configuration data can be used to indicate the attributes of the historical volume. The attributes of the historical volume may include, but are not limited to, the basic attributes and performance attributes of the historical volume. Resource consumption information can be used to indicate the resource consumption of the storage nodes.

[0051] For example, for historical volumes with different basic attributes and / or different performance attributes, the corresponding storage nodes that have affiliation with the historical volume consume different resources.

[0052] For example, storage capacity can be determined based on storage configuration. Based on storage capacity, multiple storage nodes are sorted in descending order. Then, based on resource consumption, the multiple storage nodes are sorted again in ascending order to obtain a sorting result for the multiple storage nodes. The storage node ranked first in the sorting result is then determined as the node to which the new volume belongs.

[0053] According to embodiments of this disclosure, when a new volume is mapped to a host, the owner node for the new volume is determined from multiple storage nodes in the distributed storage cluster based on the hardware configuration of the storage node and the resource consumption of historical volumes with ownership relationships with the storage node. Since the owner node for the new volume is determined when the new volume is mapped to the host, and is determined from the distributed storage cluster based on the hardware configuration of the storage node and the resource consumption of historical volumes with ownership relationships with the storage node, the load on the storage nodes can be reasonably allocated, which is beneficial for load balancing of the distributed storage cluster and improves the stability and service performance of the storage system. Compared to specifying the owner node when creating a volume, it avoids the problem of unbalanced storage node load caused by increases in the number of volumes, expansion of volume capacity, and changes in host read / write tasks during the processing of read / write tasks. Furthermore, since the resource consumption information of the storage node is a theoretical consumption determined based on storage configuration data, the response speed of the storage system can be improved.

[0054] Figure 3 A schematic diagram illustrating a first predetermined mapping relationship according to an embodiment of the present disclosure is shown.

[0055] In the process of implementing the embodiments of this disclosure, it was also found that in normal business scenarios, the number of storage nodes in a distributed storage cluster can be hundreds or thousands. If the resource consumption of each storage node is monitored dynamically in real time, it will consume a lot of monitoring resources, affect the response speed of the storage system, and even if the monitoring of a certain storage node fails, it will reduce the accuracy of determining the attribution node, thereby affecting the load balancing of the distributed storage cluster.

[0056] Based on this, in the embodiments of this disclosure, the storage configuration data may include the type of historical volume. Regarding the above... Figure 2 The operation S220 shown determines the resource consumption information of each of the multiple storage nodes based on the storage configuration data of the historical volumes that have affiliation with multiple storage nodes. It may include the operation of determining the resource consumption information of each of the multiple storage nodes based on the type of the historical volume and a first predetermined mapping relationship.

[0057] In this embodiment of the disclosure, the first predetermined mapping relationship can characterize the mapping relationship between the volume type and resource consumption information.

[0058] like Figure 3 As shown, the volume type can include type 1, type 2... type N. For example, based on the resource consumption information configured for different types of volumes by the storage system's management client, the following can be obtained: Figure 3 The types shown are mapped to resource consumption information 1, type 2 to resource consumption information 2, and so on, with type N being mapped to resource consumption information N. N is a positive integer.

[0059] According to embodiments of this disclosure, resource consumption information is determined based on the type of historical volume and a predetermined mapping relationship, resulting in high data conversion efficiency. This improves the response speed of the storage system and avoids load imbalance issues caused by monitoring failures. Furthermore, determining resource consumption information based on the type of historical volume enhances the versatility of the volume.

[0060] According to one embodiment of this disclosure, the volume mapping management method may include, in addition to, the following: Figure 2 In addition to operations S210 to S230, the following operations may also be included: determining the resource consumption information of the owner node for different types of volumes based on historical data sets; and determining a first predetermined mapping relationship based on the different types of volumes and the resource consumption information.

[0061] In this embodiment of the disclosure, the historical data set can be obtained by testing read and write tasks for different types of volumes. The historical data set may include, but is not limited to, resource consumption information of the node to which each type of volume belongs. Resource consumption information may include resource consumption ratios. Resource consumption ratios may include, for example, CPU consumption ratios and memory consumption ratios. Different types of volumes correspond to different CPU consumption ratios and memory consumption ratios. Volumes may include various types, such as, but not limited to, ordinary volumes, thin volumes, compressed volumes, dual-active volumes, snapshot volumes, and Network Attached Storage (NAS) volumes.

[0062] For example, using the created volumes of the aforementioned types, and assuming each volume type has the same performance attribute of read / write operations per second (IOPS), read / write tasks from the host can be tested for each volume type to obtain the CPU consumption ratio and memory consumption ratio corresponding to each volume type. The CPU consumption ratio and memory consumption ratio of any volume type can be mapped to a maximum CPU consumption and maximum memory consumption in the form of a score. Based on this mapping relationship for that volume type, and according to the CPU consumption ratio and memory consumption ratio of other volume types, the maximum CPU consumption and maximum memory consumption in the form of a score can be determined.

[0063] Taking a regular volume as an example, the CPU consumption ratio and memory consumption ratio of a regular volume can each be mapped to a maximum CPU consumption score of 10 and a maximum memory consumption score of 10, respectively. Based on this mapping for regular volumes, the maximum CPU consumption and maximum memory consumption in score form for thin volumes, compressed volumes, dual-active volumes, snapshot volumes, and NAS volumes can be deduced, resulting in the first predetermined mapping relationship between type, IOPS, maximum CPU consumption (score), and maximum memory consumption (score) shown in Table 1 below.

[0064]

[0065] According to embodiments of this disclosure, by testing read and write tasks for different types of volumes, mapping relationships are determined, thereby improving the accuracy of resource consumption information.

[0066] According to embodiments of this disclosure, regarding the above... Figure 2 The operation S230 shown, which determines the home node for the new volume from multiple storage nodes based on hardware configuration data and resource consumption information, may include the following operations: determining the resource utilization rate of each of the multiple storage nodes based on the hardware configuration data and resource consumption information; and determining the home node for the new volume from the multiple storage nodes based on the resource utilization rate of each of the multiple storage nodes.

[0067] For example, hardware configuration data can indicate hardware resource configuration information. For each storage node: resource utilization can be obtained based on hardware resource configuration information and resource consumption information. By comparing the resource utilization of each storage node, the storage node with the lowest resource utilization is determined as the home node.

[0068] According to embodiments of this disclosure, by determining the resource utilization of each of multiple storage nodes, the node to which a new volume belongs is determined, thereby optimizing resource allocation and improving the stability and service performance of the storage system.

[0069] In another embodiment of this disclosure, the resource consumption information indicates the theoretical resource consumption.

[0070] Figure 4 A schematic diagram illustrating the determination of storage node resource utilization according to an embodiment of the present disclosure is shown.

[0071] For each storage node: determining the resource utilization of a storage node can be done as follows Figure 4 As shown, based on the hardware configuration data 401 of the storage node and the second predetermined mapping relationship 402, the hardware performance estimate 403 of the storage node is determined. Based on the theoretical resource consumption 404 of the storage node and the preset coefficient 405 used to determine the theoretical resource consumption rate 406, the theoretical resource consumption rate 406 of the storage node is determined. Based on the ratio of the theoretical resource consumption rate 406 of the storage node to the hardware performance estimate 403, the resource utilization rate 407 of the storage node is determined.

[0072] In this embodiment, the second predetermined mapping relationship 402 can characterize the mapping relationship between hardware configuration data and hardware performance estimates. Hardware configuration data 401 may include, but is not limited to, CPU type, memory type, and memory capacity. Hardware performance estimates 403 may include CPU performance estimates and memory performance estimates. Different CPU types correspond to different CPU performance estimates, and different memory types correspond to different memory performance estimates. For the same memory type, different capacities result in different memory performance estimates. CPU performance estimates and memory performance estimates can be determined based on the factory configuration of the CPU and memory. For example, Table 2 below shows the CPU performance estimates corresponding to CPU types using Intel as an example. Table 3 below shows the memory performance estimates corresponding to memory types and memory capacities using a sense amplifier (SA) as an example.

[0073]

[0074]

[0075] For example, the theoretical resource consumption rate 406 of a storage node can be determined by multiplying the theoretical resource consumption 404 of the storage node by a preset coefficient 405 used to determine the theoretical resource consumption rate. The theoretical resource consumption 404 may include theoretical CPU consumption and theoretical memory consumption. The resource utilization rate 407 may include CPU utilization and memory utilization.

[0076] For example, CPU performance estimates and memory performance estimates matching the CPU type, memory type, and memory capacity of the storage node can be determined from Tables 2 and 3 above. The theoretical CPU consumption and theoretical memory consumption can be calculated based on the type, quantity, and IOPS value (average of a single historical volume over 5 minutes) of the historical volumes associated with the storage node, corresponding to Table 1 above. The CPU utilization rate is obtained by dividing the product of the theoretical CPU consumption and a preset coefficient used to determine the theoretical CPU consumption rate by the CPU performance estimate. Similarly, the memory utilization rate is obtained by dividing the product of the theoretical memory consumption and a preset coefficient used to determine the theoretical memory consumption rate by the memory performance estimate. The preset coefficients for determining the theoretical memory consumption rate and the theoretical CPU consumption rate can be determined based on the performance requirements of the distributed storage cluster from the terminal or based on the experience of the storage system administrators. Multiple storage nodes can be sorted in ascending order according to CPU utilization, then sorted again in ascending order according to memory utilization, and finally, the storage node ranked first in the sorted list is determined as the associated node.

[0077] According to embodiments of this disclosure, the hardware performance estimates of multiple storage nodes are determined through a predetermined mapping relationship, thereby determining resource utilization, which is accurate and has high data conversion efficiency.

[0078] According to one embodiment of this disclosure, in addition to including such Figure 2 In addition to operations S210 to S230, the operation may also include: obtaining hardware usage information of multiple storage nodes during the processing cycle; and updating preset coefficients based on the hardware usage information and resource utilization of the multiple storage nodes.

[0079] In this embodiment of the disclosure, multiple storage nodes process read and write tasks from the host within a processing cycle. After determining the owner node using the method of this disclosure, the newly added volume is assigned the attribute of the owner node. Then, the newly added volume is mapped to the host. After the newly added volume is mapped to the host, a preset time period is used for the storage nodes of the owner node of the newly added volume to process read and write tasks from the host, which is one processing cycle. Hardware usage information can be used to indicate hardware resource utilization.

[0080] Figure 5 A schematic diagram illustrating the updating of preset coefficients according to an embodiment of the present disclosure is shown.

[0081] For example, such as Figure 5 As shown, updating the preset coefficients can include operations S501 to S506.

[0082] In operation S501, hardware usage information of multiple storage nodes is obtained within a predetermined number of processing cycles.

[0083] In operation S502, it is determined whether the hardware resource utilization rate indicated by the hardware usage information is consistently lower than the resource utilization rate within a predetermined number of processing cycles.

[0084] When operating S503, reduce the preset coefficient.

[0085] In operation S504, it is determined whether the hardware resource utilization rate indicated by the hardware usage information is consistently greater than the resource utilization rate within a predetermined number of processing cycles.

[0086] In operation S505, increase the preset coefficient.

[0087] When operating S506, the preset coefficients are not updated.

[0088] In this embodiment of the disclosure, if it is determined that the hardware resource utilization rate indicated by the hardware usage information is continuously less than the resource utilization rate within a predetermined number of processing cycles, it indicates that the actual hardware resource utilization rate, such as the actual CPU utilization rate and the actual memory utilization rate, is less than the theoretically determined resource utilization rate, such as the CPU utilization rate and the memory utilization rate. This indicates that within the current predetermined number of processing cycles, the read / write model of the volume with the ownership relationship of the storage node is a type that does not easily consume resources, such as a 4K 100% read read / write model. Operation S503 can be executed to reduce the preset coefficient, thereby reducing the theoretical resource consumption rate, increasing the service load carried by the storage node, and making reasonable use of resources.

[0089] If the hardware resource utilization rate indicated by the hardware usage information is consistently not less than the resource utilization rate within a predetermined number of processing cycles, execute operation S504. If the hardware resource utilization rate indicated by the hardware usage information is consistently greater than the resource utilization rate within a predetermined number of processing cycles, it indicates that the actual hardware resource utilization rate is greater than the theoretically determined resource utilization rate. This means that within the current predetermined number of processing cycles, the read / write model of the volume belonging to this storage node is a resource-intensive type, such as a 1M 100% write read / write model. In this case, execute operation S505 to increase the preset coefficient, thereby increasing the theoretical resource consumption rate, reducing the workload carried by the storage node, and ensuring the stability of the storage system. If the hardware resource utilization rate indicated by the hardware usage information is consistently not greater than the resource utilization rate within a predetermined number of processing cycles, execute operation S506.

[0090] For example, the predetermined quantity can be determined based on the experience of the storage system administrators.

[0091] According to embodiments of this disclosure, based on the hardware usage and resource utilization of multiple storage nodes within a processing cycle, preset coefficients are updated to dynamically adjust the theoretical resource consumption rate, thereby adjusting the workload carried by the storage nodes, ensuring the stability of the storage system, and achieving rational utilization of resources.

[0092] According to embodiments of this disclosure, storage configuration data may include the read / write performance of historical volumes. Determining the home node for a new volume from among multiple storage nodes based on their respective resource utilization rates may include the following operations: if it is determined that the resource utilization rates of all storage nodes are less than a safety threshold, the storage node with the lowest resource utilization rate among the multiple storage nodes is determined as the home node for the new volume. If it is determined that the resource utilization rate of any storage node among the multiple storage nodes is greater than or equal to the safety threshold, the storage node with a resource utilization rate greater than or equal to the safety threshold is determined as a volume migration node. Based on the read / write performance of historical volumes, at least one historical volume belonging to the volume migration node is migrated to the storage node with a resource utilization rate less than the safety threshold.

[0093] In this embodiment, the security threshold can be determined based on the experience of the storage system administrator. The method of volume migration is not specifically limited in this disclosure.

[0094] According to embodiments of this disclosure, determining the ownership node solely based on a comparison of the resource utilization rates of multiple storage nodes can lead to situations where the minimum resource utilization rate exceeds safety requirements, resulting in storage node overload or even storage node failure, thus affecting the normal operation of the storage system. Determining the ownership node based on a comparison of the storage node's resource utilization rate and a safety threshold ensures load balancing across multiple storage nodes and improves the operational stability of the storage system.

[0095] In the process of implementing the embodiments of this disclosure, it was also found that different user types have different business needs for storage products. When the resource utilization of each storage node is greater than the safety threshold, how to improve the user experience while ensuring the load balance of the distributed storage system has become an urgent technical problem to be solved.

[0096] Based on this, in the embodiments of this disclosure, when it is determined that the resource utilization of multiple storage nodes is greater than a security threshold, a prompt message can be generated. The prompt message can be used to indicate multiple predetermined strategies for determining the home node. The prompt message is sent to the terminal, and after receiving the prompt message, the terminal can return feedback information based on the prompt message. The feedback information can be used to indicate a target predetermined strategy determined from multiple predetermined strategies. After receiving the feedback information, the home node is determined based on the feedback information.

[0097] For example, the pre-defined strategy may include, but is not limited to, volume partitioning strategies and storage node expansion strategies. A volume partitioning strategy might involve dividing a new volume into multiple sub-volumes based on its size, and then assigning storage nodes with the highest resource utilization to each sub-volume as their home nodes. A storage node expansion strategy might involve expanding the distributed storage cluster with additional storage nodes, designating these expanded nodes as home nodes. Administrators managing the storage system can determine the target pre-defined strategy from multiple strategies based on the user type corresponding to the storage product and provide feedback through a terminal.

[0098] According to embodiments of this disclosure, by generating prompts and sending them back to the terminal, it is possible to determine the assigned node in a personalized manner for different user types corresponding to the storage product, thereby improving the user experience while ensuring load balancing of the distributed storage system.

[0099] According to embodiments of this disclosure, the volume mapping management method may include, in addition to, the following: Figure 2In addition to operations S210 to S230, the operation may also include: obtaining a configuration file in response to a mapping request; and determining a security threshold based on the configuration file.

[0100] In this embodiment of the disclosure, the configuration file can indicate the performance requirements of the terminal for the distributed storage cluster. The performance requirements of the terminal for the distributed storage cluster can be determined according to different types of users.

[0101] For example, financial users, who prioritize product stability and may require that the business processes handled by each storage node remain within a secure range, can be configured as high-tier users in the configuration file. For high-tier users, the corresponding security threshold can be lower than the experienced value. Conversely, some small business users, who prioritize product processing speed under a certain security probability, can be configured as low-tier users in the configuration file. For low-tier users, the corresponding security threshold can be higher than the experienced value.

[0102] According to embodiments of this disclosure, a security threshold is determined based on the terminal's performance requirements for the distributed storage cluster, thereby meeting the terminal's personalized needs and enhancing the user experience.

[0103] According to embodiments of this disclosure, preset coefficients can also be determined based on configuration files.

[0104] For example, for high-end users, the default configuration factor can be higher than the empirical value. For low-end users, the default configuration factor can be lower than the empirical value. Since high-end users require storage nodes to handle more volumes, a lower default factor reduces the theoretical resource consumption rate. Low-end users prioritize the security and stability of the storage system, with metadata write speed being less of a priority; therefore, a higher default factor increases the theoretical resource consumption rate.

[0105] Based on the above-described volume mapping management method, this disclosure also provides a volume mapping management apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0106] Figure 6 A schematic block diagram of a volume mapping management apparatus according to an embodiment of the present disclosure is shown.

[0107] like Figure 6 As shown, the volume mapping management device 600 of this embodiment includes an acquisition module 610, a first determination module 620, and a second determination module 630.

[0108] The acquisition module 610 is used to acquire hardware configuration data of the distributed storage cluster in response to a mapping request for a newly added volume mapping host. The distributed storage cluster includes multiple storage nodes, which are used to handle read and write tasks from the host. In one embodiment, the acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0109] The first determining module 620 is used to determine the resource consumption information of each of the multiple storage nodes based on the storage configuration data of historical volumes that have affiliation with multiple storage nodes. In one embodiment, the first determining module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0110] The second determining module 630 is used to determine the home node for the newly added volume from multiple storage nodes based on hardware configuration data and resource consumption information, so that the home node can process read and write tasks from the host after the newly added volume is mapped to the host. In one embodiment, the second determining module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0111] According to embodiments of this disclosure, storage configuration data may include the types of historical volumes. The first determining module 620 may include a mapping unit. The mapping unit is used to determine the resource consumption information of multiple storage nodes based on the types of historical volumes and a first predetermined mapping relationship, wherein the first predetermined mapping relationship characterizes the mapping relationship between volume types and resource consumption information.

[0112] According to embodiments of this disclosure, the volume mapping management device 600 may further include a third determining module and a fourth determining module. The third determining module is used to determine the resource consumption information of the home node for different types of volumes based on a historical data set, wherein the historical data set is obtained by performing test read / write tasks on different types of volumes. The fourth determining module is used to determine a first predetermined mapping relationship based on the different types of volumes and the resource consumption information.

[0113] According to embodiments of this disclosure, the second determining module 630 may include a resource utilization determining unit and a home node determining unit. The resource utilization determining unit is used to determine the resource utilization of each of the multiple storage nodes based on hardware configuration data and resource consumption information. The home node determining unit is used to determine the home node for the newly added volume from among the multiple storage nodes based on the resource utilization of each of the multiple storage nodes.

[0114] According to embodiments of this disclosure, resource consumption information indicates theoretical resource consumption. Determining the resource utilization rate of multiple storage nodes based on hardware configuration data and resource consumption information may include: determining the hardware performance estimate of each of the multiple storage nodes based on the hardware configuration data of each of the multiple storage nodes and a second predetermined mapping relationship, wherein the second predetermined mapping relationship characterizes the mapping relationship between hardware configuration data and hardware performance estimate; determining the theoretical resource consumption rate of each of the multiple storage nodes based on the theoretical resource consumption of each of the multiple storage nodes and a preset coefficient used to determine the theoretical resource consumption rate; and determining the resource utilization rate of each of the multiple storage nodes based on the ratio of the theoretical resource consumption rate of each of the multiple storage nodes to the hardware performance estimate.

[0115] According to embodiments of this disclosure, the volume mapping management device 600 may further include a hardware usage information acquisition module and an update module. The hardware usage information acquisition module is used to acquire hardware usage information of multiple storage nodes during a processing cycle, wherein the multiple storage nodes process read / write tasks from the host during the processing cycle. The update module is used to update a preset coefficient based on the hardware usage information and resource utilization of the multiple storage nodes.

[0116] According to embodiments of this disclosure, the storage configuration data includes the read / write performance of historical volumes. Determining the home node for a new volume from among multiple storage nodes based on their respective resource utilization rates may include: if the resource utilization rates of all storage nodes are less than a security threshold, determining the storage node with the lowest resource utilization rate as the home node for the new volume; if the resource utilization rate of at least one storage node is greater than or equal to the security threshold, determining the storage node with the resource utilization rate greater than or equal to the security threshold as the volume migration node; and migrating at least one historical volume belonging to the volume migration node to a storage node with a resource utilization rate less than the security threshold based on the historical volume's read / write performance.

[0117] According to embodiments of this disclosure, the volume mapping management device 600 may further include a configuration file acquisition module and a threshold determination module. The configuration file acquisition module is used to acquire a configuration file in response to a mapping request, wherein the configuration file indicates the performance requirements of the terminal for the distributed storage cluster. The threshold determination module is used to determine a security threshold based on the configuration file.

[0118] According to embodiments of this disclosure, any plurality of modules among the acquisition module 610, the first determining module 620, and the second determining module 630 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 610, the first determining module 620, and the second determining module 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 610, the first determining module 620, and the second determining module 630 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0119] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a volume mapping management method according to an embodiment of the present disclosure.

[0120] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0121] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0122] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0123] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0124] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0125] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0126] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0127] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0128] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0129] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0132] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A volume mapping management method, characterized in that, The method includes: In response to a mapping request for a new volume mapping host, the hardware configuration data of the distributed storage cluster is obtained, wherein the distributed storage cluster includes multiple storage nodes, and the multiple storage nodes are used to process read and write tasks from the host. Based on the storage configuration data of historical volumes that have affiliation with the multiple storage nodes, determine the resource consumption information of each of the multiple storage nodes; Based on the hardware configuration data and the resource consumption information, the resource utilization rate of each of the plurality of storage nodes is determined. The hardware configuration data indicates the storage configuration or hardware resource configuration information of the storage node, and the resource consumption information indicates the theoretical resource consumption. Based on the resource utilization of each of the plurality of storage nodes, a home node for the newly added volume is determined from the plurality of storage nodes, so that the home node processes read and write tasks from the host after the newly added volume is mapped to the host.

2. The method according to claim 1, characterized in that, The storage configuration data includes the type of the historical volume; The determination of resource consumption information for each of the multiple storage nodes based on storage configuration data of historical volumes that have affiliation with the multiple storage nodes includes: Based on the type of the historical volume and the first predetermined mapping relationship, the resource consumption information of each of the plurality of storage nodes is determined, wherein the first predetermined mapping relationship represents the mapping relationship between the type of the volume and the resource consumption information.

3. The method according to claim 2, characterized in that, The method further includes: Based on historical data sets, resource consumption information of the nodes owning different types of volumes is determined, wherein the historical data sets are obtained by performing test read / write tasks on the different types of volumes; and Based on the different types of volumes and the resource consumption information, the first predetermined mapping relationship is determined.

4. The method according to claim 1, characterized in that, The step of determining the resource utilization rate of each of the multiple storage nodes based on the hardware configuration data and the resource consumption information includes: Based on the hardware configuration data of each of the plurality of storage nodes and the second predetermined mapping relationship, the hardware performance estimate of each of the plurality of storage nodes is determined, wherein the second predetermined mapping relationship represents the mapping relationship between the hardware configuration data and the hardware performance estimate; Based on the theoretical resource consumption of each of the plurality of storage nodes and a preset coefficient used to determine the theoretical resource consumption rate, the theoretical resource consumption rate of each of the plurality of storage nodes is determined; and The resource utilization rate of each of the multiple storage nodes is determined based on the ratio of their respective theoretical resource consumption rate to their estimated hardware performance.

5. The method according to claim 4, characterized in that, The method further includes: Obtain hardware usage information for each of the plurality of storage nodes during the processing cycle, wherein the plurality of storage nodes process read and write tasks from the host during the processing cycle; and The preset coefficients are updated based on the hardware usage information and resource utilization of each of the multiple storage nodes.

6. The method according to claim 1, characterized in that, The storage configuration data includes the read and write performance of the historical volume; The step of determining the node to which the newly added volume belongs from the plurality of storage nodes based on the resource utilization rates of the respective storage nodes includes: If it is determined that the resource utilization rate of all storage nodes is less than the safety threshold, the storage node with the lowest resource utilization rate among the multiple storage nodes shall be determined as the node to which the new volume belongs. If it is determined that among the plurality of storage nodes, there is a storage node whose resource utilization is greater than or equal to the security threshold, the storage node whose resource utilization is greater than or equal to the security threshold is identified as a volume migration node; and Based on the read / write performance of the historical volume, at least one historical volume belonging to the volume migration node is migrated to a storage node with a resource utilization rate less than the security threshold.

7. The method according to claim 6, characterized in that, The method further includes: In response to the mapping request, a configuration file is obtained, wherein the configuration file indicates the terminal's performance requirements for the distributed storage cluster; and The security threshold is determined based on the configuration file.

8. A volume mapping management device, characterized in that, The device includes: The acquisition module is used to acquire hardware configuration data of the distributed storage cluster in response to a mapping request for a newly added volume mapping host. The distributed storage cluster includes multiple storage nodes, which are used to process read and write tasks from the host. The first determining module is used to determine the resource consumption information of each of the multiple storage nodes based on the storage configuration data of historical volumes that have affiliation with the multiple storage nodes; and The second determining module is used to determine the resource utilization rate of each of the plurality of storage nodes based on the hardware configuration data and the resource consumption information, wherein the hardware configuration data indicates the storage configuration or hardware resource configuration information of the storage nodes, and the resource consumption information indicates the theoretical resource consumption; and based on the resource utilization rate of each of the plurality of storage nodes, to determine the home node for the newly added volume from the plurality of storage nodes, so that the home node processes read and write tasks from the host after the newly added volume is mapped to the host.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images