A method for managing a heterogeneous storage cluster and a storage medium

Abstract

This specification discloses a method for managing a heterogeneous storage cluster and a storage medium. In this method, all storage resources of the heterogeneous storage cluster can be centrally managed and dynamically allocated through the global resource pool of the management device, and seamless data migration across different cloud computing environments and multiple storage clusters is supported, enhancing the flexibility and adaptability of the system, enabling users to perform complete life cycle management of storage resources from creation, allocation, usage to recycling through a user interface, without the need to concern themselves with the underlying complex hardware configuration or network topology, thereby helping to reduce the complexity of operation and maintenance, reduce human errors, and significantly reduce the costs of the computing power center in storage operation.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and particularly to a method for managing a heterogeneous storage cluster and a storage medium. Background Art

[0002] With the rapid development of intelligent computing technology, as the core infrastructure supporting massive data processing and complex computing tasks, the computing power center has become increasingly important. During the construction and operation of the computing power center, storage devices from different storage manufacturers are usually purchased and used as needed. These storage devices from different manufacturers have significant differences in technical architecture, interface protocols, management tools, and performance characteristics, showing obvious heterogeneity. This heterogeneity greatly increases the complexity of resource management. Since the management methods of devices from different manufacturers vary, managers need to be familiar with and master a variety of management tools and interfaces to effectively monitor, configure, and maintain various storage devices, resulting in relatively high management costs.

[0003] Therefore, how to reduce the management cost of a heterogeneous storage cluster is an urgent problem to be solved. Summary of the Invention

[0004] This specification provides a method for managing a heterogeneous storage cluster and a storage medium to partially solve the above problems existing in the prior art.

[0005] This specification adopts the following technical solutions:

[0006] This specification provides a method for managing a heterogeneous storage cluster.

[0007] The method is applied to a management device of a heterogeneous storage cluster, and the method includes:

[0008] Determine demand information according to the user's storage request;

[0009] Select a target heterogeneous cluster group that matches the demand information from the heterogeneous cluster groups included in a preset global resource pool; the heterogeneous cluster groups are obtained by pre-dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters, and the storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster.

[0010] Select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and allocate storage resources for the user from the target storage cluster.

[0011] Optionally, the storage resource feature information includes at least one of the following: the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster.

[0012] Optionally, selecting a target storage cluster from the target heterogeneous cluster group according to a preset selection policy specifically includes:

[0013] When it is determined that the requirement information is a target storage type, a target storage cluster is selected from the target heterogeneous cluster group according to a preset first selection policy, and the first selection policy is used to select a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

[0014] Optionally, selecting a target storage cluster from the target heterogeneous cluster group according to a preset selection policy specifically includes:

[0015] When it is determined that the requirement information is a resource parameter, a target storage cluster is selected from the target heterogeneous cluster group according to a preset second selection policy, and the second selection policy is used to select a target storage cluster according to the storage resource utilization rate distribution characteristics of each storage cluster in the target heterogeneous cluster group. The resource parameters include: protocol type, throughput, and number of input / output operations per second.

[0016] Optionally, allocating storage resources for the user from the target storage cluster specifically includes:

[0017] Sending a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources for the user according to the resource request and return the resource information of the storage resources allocated for the user;

[0018] Creating a storage instance according to the resource information, so that the user can access the storage resources through the storage instance.

[0019] Optionally, it further includes:

[0020] Managing data at different levels stored in the storage resources allocated for the user according to the received data management operation instruction, and the data management operation instruction is determined according to the interaction operation performed by the user on the data at different levels stored in the storage resources in a preset user interface.

[0021] Optionally, the method further includes:

[0022] When it is determined that the expiration period of the storage resources allocated to the user is lower than a preset threshold, the storage resources are processed according to the expiration management policy corresponding to the storage resources allocated to the user; the expiration management policy includes: a first expiration management policy for instructing to prohibit the user from having at least partial access rights to the storage resources, a second expiration management policy for instructing to recycle the storage resources, and a third expiration management policy for instructing to migrate the data saved in the storage resources to other storage resources, where the performance level of the other storage resources is lower than the performance level of the storage resources.

[0023] Optionally, the method further includes:

[0024] When it is determined that a preset data migration condition is met, a migration task is created according to the obtained data source information and the destination address, and the source data corresponding to the data source information is migrated to the storage resources corresponding to the destination address for storage according to the migration policy matching the migration task.

[0025] This specification provides a multi - heterogeneous storage cluster management system, and the multi - heterogeneous storage cluster management system includes: a management device and multiple storage clusters;

[0026] The management device is configured to determine demand information according to the storage request of the user, select a target heterogeneous cluster group matching the demand information from different heterogeneous cluster groups included in a preset global resource pool, and select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and send a resource request to the target storage cluster through the adapter of the target storage cluster, and create a storage instance according to the resource information returned by the target storage cluster according to the resource request, so that the user can access the storage resources through the storage instance; the different heterogeneous cluster groups are obtained by pre - dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters, and the storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster.

[0027] The storage cluster is configured to receive the resource request sent by the management device, allocate storage resources for the user according to the resource request, and send the resource information of the storage resources allocated to the user to the management device.

[0028] This specification provides a computer - readable storage medium, and the storage medium stores a computer program, and when the computer program is executed by a processor, the above - mentioned multi - heterogeneous storage cluster management method is implemented.

[0029] This specification provides a management device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned multi - heterogeneous storage cluster management method is implemented.

[0030] The above - mentioned at least one technical solution adopted in this specification can achieve the following beneficial effects:

[0031] In the multi - heterogeneous storage cluster management method provided in this specification, first, demand information is determined according to the user's storage request, and a target heterogeneous cluster group matching the demand information is selected from different heterogeneous cluster groups included in a preset global resource pool. Among them, different heterogeneous cluster groups are obtained by pre - dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters. The storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster. Then, according to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

[0032] It can be seen from the above method that all storage resources of the multi - heterogeneous storage cluster can be centrally managed and dynamically allocated through the global resource pool of the management device, so that users do not need to care about the underlying complex hardware configuration or network topology. Furthermore, it can help reduce the operation and maintenance complexity, reduce human errors, and significantly reduce the cost of managing the multi - heterogeneous storage cluster. Description of the Drawings

[0033] The drawings described herein are used to provide a further understanding of this specification, and constitute a part of this specification. The schematic embodiments of this specification and their descriptions are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0034] Figure 1 It is a schematic flow chart of a multi - heterogeneous storage cluster management method provided in this specification;

[0035] Figure 2 It is a schematic diagram of different heterogeneous cluster groups provided in this specification;

[0036] Figure 3 It is a schematic diagram of the process of adding a new storage cluster provided in this specification;

[0037] Figure 4A It is a schematic diagram of the user interface for file management provided in this specification;

[0038] Figure 4B It is a schematic diagram of the user interface for storage instance management provided in this specification;

[0039] Figure 5A schematic diagram of a multi-heterogeneous storage cluster management system provided in this specification;

[0040] Figure 6 A schematic diagram of a multi-heterogeneous storage cluster management device provided in this specification;

[0041] Figure 7 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0043] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0044] With the rapid development of intelligent computing technology, the importance of computing centers as core infrastructure supporting massive data processing and complex computing tasks has become increasingly prominent. In the construction and operation of computing centers, storage devices from different storage vendors are usually purchased and used on demand. These storage devices from different storage vendors have significant differences in technical architecture, interface protocols, management tools, and performance characteristics, showing obvious heterogeneity. This heterogeneity has brought many challenges to the storage resource management of computing centers. On the one hand, the complexity of resource management has increased significantly. Due to the different management methods of devices from different manufacturers, managers need to be familiar with and master a variety of management tools and interfaces in order to effectively monitor, configure and maintain various storage devices. This not only increases management costs, but also easily leads to unreasonable resource allocation and difficulty in troubleshooting equipment failures due to operational errors or poor management. On the other hand, the heterogeneity of storage devices also seriously affects the efficiency of use. Performance bottlenecks between different devices, incompatible data formats, and inflexible resource scheduling make it impossible to fully and efficiently utilize storage resources, which in turn affects the overall computing performance and business response speed of the computing center.

[0045] At present, solutions for managing heterogeneous storage often have problems such as poor compatibility, imperfect functions or performance bottlenecks, which make it difficult to meet the growing requirements of efficient and intelligent management of computing centers. Therefore, how to uniformly and efficiently manage heterogeneous storage clusters in new computing centers to reduce the cost of managing heterogeneous storage clusters has become a technical problem that needs to be solved in the field of intelligent computing.

[0046] Figure 1 This is a schematic flowchart of a multi - heterogeneous storage cluster management method provided in this specification, including the following steps:

[0047] S101: Determine the requirement information according to the user's storage request.

[0048] In this specification, the above - mentioned multi - heterogeneous storage cluster management method is applied to the management device of the multi - heterogeneous storage cluster. That is, the multi - heterogeneous storage cluster management method can be executed by the management device of the multi - heterogeneous storage cluster to manage the multi - heterogeneous storage cluster.

[0049] Among them, the multi - heterogeneous storage cluster can refer to storage devices and storage technologies of different types and architectures, such as: Network Attached Storage (NAS) of traditional storage server vendors, object storage services, General Parallel File System (GPFS), and NAS, object storage, and elastic block storage in private cloud environments, etc.

[0050] In the above content, NAS is a storage solution that provides data access through a network and is usually used for file - level storage management. It provides simple shared file storage functions, supports multi - user access, and is widely used in scenarios such as file sharing, backup, and recovery. NAS devices usually have good scalability and flexibility and can meet the needs of a large amount of data access and storage, but there are certain bottlenecks in terms of performance and scalability.

[0051] Object storage is a storage method based on objects, which does not rely on the traditional file system structure and is suitable for massive data storage and management. Object storage services provide high scalability and high availability and can support the storage and management of various data types (such as pictures, videos, log files, etc.). It accesses data through a unique identifier (such as an object ID), which is different from the traditional file system. Object storage is widely used in fields such as big data storage and cloud computing platforms.

[0052] GPFS is a high - performance parallel file system that is widely used in large - scale computing clusters and supports high - speed data access and sharing. GPFS can combine multiple nodes in the storage cluster to provide high - bandwidth and low - latency data reading and writing performance, and is suitable for large - scale parallel processing environments that require high - performance computing (HPC). Its advantage lies in being able to efficiently process massive data and support distributed computing and large - scale parallel data access.

[0053] Elastic Block Storage: Elastic Block Storage is a block storage service based on the cloud environment, commonly used for the persistent storage of virtual machines, and can elastically expand the storage capacity according to requirements. Block storage supports high-performance and low-latency storage requirements and is suitable for database and high-performance computing applications. Its elasticity and scalability make block storage a common storage solution in cloud computing platforms.

[0054] From the above content, it can be seen that the business platform can combine multiple storage technologies through the management device to meet different storage requirements, such as file storage, object storage, block storage, etc. And it can manage the storage resources of each storage cluster with different technical backgrounds, functional characteristics, and suppliers, thus avoiding the heterogeneity between different storage devices from making the work of resource scheduling, load balancing, performance optimization, fault detection, etc. more complex, and further leading to a situation where the management cost of the multi-source heterogeneous storage cluster is relatively high.

[0055] Specifically, the management device can receive the storage request sent by the user through the application layer through the preset resource gateway management module, and determine the demand information of the user for the required storage resources through the resource center in the preset resource management module according to the received storage request. Furthermore, according to the received demand information, storage resources can be allocated to the user.

[0056] Among them, the above-mentioned management device can refer to a device integrated with a software program module for managing a multi-source heterogeneous storage cluster.

[0057] The above-mentioned application layer is used to interact with the user based on the unified storage application programming interface (Application Programming Interface, API) gateway provided by the management device.

[0058] S102: Select a target heterogeneous cluster group that matches the demand information from the heterogeneous cluster groups included in the preset global resource pool; the heterogeneous cluster groups are obtained by pre-dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters, and the storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster.

[0059] In this specification, after obtaining the user's demand information, the management device can select a target heterogeneous cluster group that matches the user's demand information from the heterogeneous cluster groups included in the preset global resource pool through the resource management module.

[0060] It should be noted that the above-mentioned heterogeneous cluster groups are obtained by pre-dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters.

[0061] Among them, the above storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster. The above storage resource characteristic information may be software information and / or hardware information, including at least one of the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster.

[0062] The above storage resource characteristic information may be, for example: information about storage nodes included in the storage cluster and / or information about storage pools (composed of at least some storage nodes), logs, resource statistics information, etc.

[0063] In the above content, the information of the storage node may include: identification information of the node, storage medium type, capacity information, I / O performance, read / write speed, operating status (such as normal, faulty, under maintenance, etc.), network information (such as: network interface, bandwidth, data transmission protocol and other network configuration information), etc.

[0064] The information of the storage pool may include: identification information of the storage pool, information about the component nodes, total capacity information and available capacity information, overall performance indicators (such as: throughput, latency, etc.), redundancy strategy (such as: RAID, replica mechanism, erasure code, etc.), storage protocol support, etc.

[0065] The resource statistics information may include: storage resource allocation information of the storage cluster, total capacity and available capacity of the storage cluster, storage resource utilization trend information of the storage cluster (such as: average storage utilization in the past month, etc.), load balancing status information (such as: disk I / O, IOPS and other parameters of each storage node or storage pool), resource migration record information, etc.

[0066] Specifically, the management device may, for each storage cluster to be partitioned, determine the performance level corresponding to the storage cluster according to the storage resource characteristic information of the storage cluster to be partitioned.

[0067] For example: A storage cluster that uses an efficient data transmission protocol and whose storage medium type is a solid-state drive can be set to a higher performance level.

[0068] For another example: A storage cluster that uses the NFS data transmission protocol and whose storage medium type is a traditional mechanical hard disk can be set to a lower performance level.

[0069] Furthermore, the management device may partition each storage cluster to be partitioned into different heterogeneous cluster groups according to the performance level corresponding to each storage cluster to be partitioned, specifically as Figure 2 shown.

[0070] Figure 2 is a schematic diagram of each heterogeneous cluster group provided in this specification.

[0071] Combined with Figure 2 It can be seen that the management device can determine the performance level corresponding to each storage cluster in advance according to the storage resource characteristic information of different storage clusters, and then, according to the different performance levels of each storage cluster, divide each storage cluster into different heterogeneous cluster groups. In Figure 2 it, level0 represents high-performance storage resources, and the performance level decreases sequentially from 0 to N.

[0072] It should be noted that in the actual application scenario, the user can select the performance level of the required storage resources through a preset user interface as demand information, so that the management device can determine the performance level of the storage resources required by the user according to the demand information, and determine the target heterogeneous cluster group according to the performance level of the storage resources required by the user.

[0073] Of course, the user can also input the resource type, resource parameters, etc. of the required storage resources as demand information, so that the management device can determine the performance level of the storage resources required by the user according to the user's demand information, and determine the target heterogeneous cluster group according to the performance level of the storage resources required by the user.

[0074] For the sake of easy understanding, the following will be combined with Figure 3 to elaborate in detail on the process of how to manage adding a newly added storage cluster to the above-mentioned global resource pool.

[0075] Figure 3 This is a schematic diagram of the process of adding a newly added storage cluster provided in this specification.

[0076] Combined with Figure 3 It can be seen that when it is necessary to add a newly added storage cluster to the global resource pool for management by the management device, the adapter corresponding to the storage cluster (each manufacturer's storage corresponds to an adapter) can be obtained first through the resource management module of the management device, and the adapter of the storage resources can be saved, so as to obtain the storage resource characteristic information of the storage cluster, determine the performance level corresponding to the storage cluster according to the storage resource characteristic information of the storage cluster, and add the storage cluster to the heterogeneous cluster group that matches the performance level of the domain storage cluster according to the performance level corresponding to the storage cluster.

[0077] In the process of managing each heterogeneous cluster group through the global resource pool, the management device can also regularly perform resource calibration with each storage cluster through the resource management module, that is, for each storage cluster, detect whether the storage resource characteristic information of this storage cluster stored in the global resource pool is consistent with the actual storage resource characteristic information of this storage cluster, and if it is inconsistent, it can be corrected.

[0078] S103: Select a target storage cluster from the target heterogeneous cluster groups according to a preset selection strategy, and allocate storage resources for the user from the target storage cluster.

[0079] In this specification, after determining the target heterogeneous cluster groups, the management device can select a target storage cluster from the target heterogeneous cluster groups according to a preset selection strategy, and allocate storage resources for the user from the target storage cluster.

[0080] Among them, there can be multiple such selection strategies. For example: a first selection strategy for selecting a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster groups.

[0081] For another example: a second selection strategy for selecting a target storage cluster according to the distribution characteristics of the storage resource utilization rate of each storage cluster in the target heterogeneous cluster groups, etc. (here, the distribution characteristics of the storage resource utilization rate are used to reflect the usage situation and its change pattern of the storage resources within a certain time period, such as: average utilization rate, peak utilization rate, utilization rate variance, etc.). By selecting a target storage cluster according to the distribution characteristics of the storage resource utilization rate of each storage cluster in the target heterogeneous cluster groups, it is possible to avoid allocating too much data to a single highly loaded storage cluster, thereby balancing the workload of each storage cluster.

[0082] In an actual application scenario, in order to improve the efficiency of the user to manage the heterogeneous storage clusters through the management device, the user can also, according to actual needs, choose to input the type of the required storage resources through a preset user interface as the target storage resource type, so that when the management device determines according to the obtained requirement information that the user needs to use the storage resources of the target storage resource type, it can select a target storage cluster from the target heterogeneous cluster groups according to the preset first selection strategy.

[0083] Among them, through the first selection strategy, the storage cluster with the largest remaining capacity can be selected from each storage cluster in the target heterogeneous cluster groups according to the remaining capacity of each storage cluster in the target heterogeneous cluster groups as the target storage cluster.

[0084] In addition, the user can also, according to actual needs, choose to input the resource parameters of the required storage resources through a preset user interface, so that when the management device determines according to the obtained requirement information that the user needs to use the storage resources that meet the above resource parameters, it can select a target storage cluster from the target heterogeneous cluster groups according to the preset second selection strategy.

[0085] Among them, the target storage cluster can be selected according to the distribution characteristics of the storage resource utilization rate of each storage cluster in the target heterogeneous cluster group through the second selection strategy. That is, an adaptive algorithm can be used to select the storage cluster with a lower resource utilization rate as the target storage cluster according to the distribution characteristics of the storage resource utilization rate of each storage cluster, so as to balance the use of storage resources of each storage cluster in the target heterogeneous cluster group.

[0086] Furthermore, after determining the target storage cluster, the management device can obtain the adapter corresponding to the target storage cluster from the pre-saved adapters, and then can send a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources for the user according to the resource request and return the resource information of the storage resources allocated for the user. Then, a storage instance can be created according to the resource information of the storage resources allocated for the user, so that the user can access the storage resources through the storage instance.

[0087] Among them, the above storage instance can be a file directory or a Bucket with a specified capacity, etc.

[0088] It should be noted that after the management device creates a storage instance for the user, the resource gateway management module can manage the storage instance according to the operation instructions input by the user through the user interface, specifically as Figure 4A 、 Figure 4B shown.

[0089] Figure 4A This is a schematic diagram of the user interface for file management provided in this specification.

[0090] Combined with Figure 4A it can be seen that after a storage instance is created for the user, the user can perform data management operations such as file upload and file download for the files in the storage instance through the page component provided in the user interface for managing the files in the storage instance.

[0091] Figure 4B This is a schematic diagram of the user interface for storage instance management provided in this specification.

[0092] Combined with Figure 4B it can be seen that after a storage instance is created for the user, the user can perform change operations such as expanding, editing, and deleting the storage instance through the page component provided in the user interface for managing the storage instance, and synchronize the change operations performed by the user to the global resource pool.

[0093] Among them, the management module can also manage data at different levels saved in the storage resources allocated to the user according to the received data management operation instructions, where the data management operation instructions are determined according to the interaction operations performed by the user on the data at different levels saved in the storage resources in a preset user interface.

[0094] The above-mentioned data at different levels may refer to directory-level data or file-level data. Here, the directory-level data may refer to the directory structure data included in the storage instance, and the file-level data may refer to the content data in the specific files included in the storage instance.

[0095] It can be seen from the above that when the user manages the storage resources in the heterogeneous storage cluster through the management device, the user can directly execute management operations through the user interface without considering the significant differences in technical architecture, interface protocol, management tools, and performance characteristics of the storage devices of different storage vendors.

[0096] Furthermore, after creating a storage instance for the user so that the user can access the storage resources allocated to the user through the storage instance, the management device can also obtain the metering information of the storage resources determined to be allocated to the user, so that when it is determined according to the metering information of the storage resources allocated to the user that the expiration period of the storage resources allocated to the user is lower than the preset threshold, the storage resources can be processed according to the expiration management policy corresponding to the storage resources allocated to the user.

[0097] Among them, the above-mentioned expiration management policies include: a first expiration management policy for instructing to prohibit the user from having at least partial access rights to the storage resources, a second expiration management policy for instructing to recycle the storage resources, and a third expiration management policy for instructing to migrate the data saved in the storage resources to other storage resources, where the performance level of the above-mentioned other storage resources is lower than the performance level of the storage resources.

[0098] It should be noted that the method by which the management device manages the storage instance and the data at different levels in the storage instance according to the operation instructions input by the user through the user interface can be implemented on the K8s / ACK cloud-native infrastructure.

[0099] Specifically, for each storage instance, when the management device creates the storage instance, it can also create a deployment unit Pod for the storage instance in k8s, so as to further manage each storage instance separately through the Pod, so that separate lifecycle management can be set for each storage instance, and resource isolation can be performed between each storage instance and other storage instances.

[0100] Among them, the lifecycle management of each of the above storage instances can be set according to actual requirements. For example, when a user obtains storage resources for model training, high-performance storage resources can be provided for the computing power cluster used by the user during model training. When feedback information indicating the end of model training is received from the computing power cluster, it is considered that the expiration period of the storage resources allocated to the user is lower than a preset threshold. Furthermore, the storage resources can be processed according to the expiration management policy corresponding to the storage resources allocated to the user.

[0101] In addition, when the management device determines that the preset data migration conditions are met, a migration task can also be created by the preset data migration module according to the obtained data source information and destination address, and the source data corresponding to the data source information can be migrated to the storage resources corresponding to the destination address for storage according to the migration strategy matching the migration task.

[0102] Among them, the above data migration conditions can be determined according to actual requirements. For example, after a user obtains a storage instance allocated by the management device, the data stored in other storage devices can be migrated to the above storage instance. At this time, the user can configure the data source address and destination address through the user interface. Furthermore, when the management device receives a data migration request sent by the user through the user interface, it can be considered that the preset data migration conditions are met. Then, according to the data migration request, the data source address and destination address configured by the user can be obtained, and a migration task can be created. Furthermore, the source data corresponding to the data source information can be migrated to the storage resources corresponding to the destination address for storage according to the migration strategy matching the migration task.

[0103] For another example, when the management device determines that the expiration period of the storage resources allocated to the user is lower than the preset threshold, it can be considered that the preset data migration conditions are met. Then, the data source information corresponding to the storage resources allocated to the user can be obtained, and a preset destination address can be obtained to create a migration task. Furthermore, the source data corresponding to the data source information can be migrated to the storage resources corresponding to the destination address for storage according to the migration strategy matching the migration task.

[0104] In the above content, the number of migration threads used for data migration according to different migration strategies and / or the maximum bandwidth used during migration are different. Among them, the migration strategy here can be pre-configured by the user according to actual requirements.

[0105] It should be noted that in actual application scenarios, the management device often needs to perform data migration between different types of storage systems. For example, data migration between file systems, data migration between file systems and object systems, data migration between object systems, and data migration between object systems and file systems.

[0106] Therefore, in addition to the address where the data to be migrated is located (which can be the directory or path where the data to be migrated is located) or the bucket where the data to be migrated is located, the above data source information can also include: protocol information required to access the data to be migrated, such as: Network File System (NFS) protocol, Server Message Block (SMB) protocol, object storage protocol S3, etc.

[0107] Furthermore, after obtaining the above data source information, the management device can, based on the data source information, determine the address where the file to be migrated is located and determine the destination address, and then can create a migration task according to the protocol information required for data migration determined based on the data source information, so as to store the source data corresponding to the data source information in the storage resource corresponding to the destination address according to the migration strategy matching the migration task.

[0108] It is worth noting that when the management device executes the migration task for data migration, it can perform protocol conversion through a preset migration gateway management module to ensure data interoperability between different storage clusters and handle data conversion of different formats and different encoding methods to ensure compatibility of data between different storage systems.

[0109] In an actual application scenario, the user can send a storage request to the management device through the application layer, so that the management device allocates storage resources for the user according to the storage request sent by the user, and then can connect the storage resources allocated for the user to the computing power cluster, so that the user can execute tasks (such as: model training task execution) through the computing power cluster.

[0110] During this process, since different computing power clusters are usually composed of various different types of computing resources, such as CPUs, GPUs, FPGAs, ASICs, etc., and there are significant differences in architecture, performance characteristics, and application scenarios among these resources, different computing power clusters have different access methods for storage resources. Therefore, the management device can also, through the resource gateway management module, select a target access method from the preset access methods for different computing power clusters and connect the storage resources allocated for the user to the computing power cluster according to the target access method.

[0111] As can be seen from the above method, all storage resources of the heterogeneous storage cluster can be centrally managed and dynamically allocated through the global resource pool of the management device, and seamless data migration across different cloud computing environments and multiple storage clusters is supported, enhancing the flexibility and adaptability of the system. This enables users to perform complete life cycle management of storage resources from creation, allocation, use to recycling through the user interface, without having to worry about the underlying complex hardware configuration or network topology. This can help reduce the complexity of operation and maintenance, reduce human errors, and significantly reduce the cost of the computing power center in storage operation.

[0112] For ease of understanding, this specification also provides a heterogeneous storage cluster management system, specifically as Figure 5 shown.

[0113] Figure 5 It is a schematic diagram of a heterogeneous storage cluster management system provided in this specification.

[0114] Combined with Figure 5 it can be seen that the above heterogeneous storage cluster management system may include: a management device, and multiple storage clusters.

[0115] Among them, the above management device is used to determine demand information according to the user's storage request, select a target heterogeneous cluster group that matches the demand information from different heterogeneous cluster groups included in the preset global resource pool, and select a target storage cluster from the target heterogeneous cluster group according to the preset selection strategy. Then, through the adapter of the target storage cluster, a resource request is sent to the target storage cluster, and a storage instance is created according to the resource information returned by the target storage cluster in response to the resource request, so that the user can access the storage resources through the storage instance; the different heterogeneous cluster groups are obtained by pre-dividing different storage clusters according to the storage resource characteristic information corresponding to different storage clusters, and the storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster.

[0116] The storage cluster is used to receive the resource request sent by the management device, allocate storage resources for the user according to the resource request, and send the resource information of the storage resources allocated for the user to the management device.

[0117] Specifically, the above management device integrates a resource management module, a resource gateway management module, a unified storage API gateway, and a data migration module.

[0118] As can be seen from the above, all storage resources of a heterogeneous storage cluster can be centrally managed and dynamically allocated through software program modules such as the resource management module, resource gateway management module, unified storage API gateway, and data migration module included in the management device, and seamless data migration across different cloud computing environments and multiple storage clusters is supported, enhancing the flexibility and adaptability of the system, so that users can perform complete storage resource lifecycle management from creation, allocation, use to recycling through the user interface, without caring about the underlying complex hardware configuration or network topology, which can help reduce the operation and maintenance complexity, reduce human errors, and significantly reduce the storage operation cost of the computing power center.

[0119] The above is one or more embodiments of the method for managing a heterogeneous storage cluster in this specification. Based on the same idea, this specification also provides a corresponding heterogeneous storage cluster management device, as Figure 6 shown.

[0120] Figure 6 The figure is a schematic diagram of a heterogeneous storage cluster management device provided in this specification, including:

[0121] A determination module 601, configured to determine requirement information according to a user's storage request;

[0122] A selection module 602, configured to select a target heterogeneous cluster group that matches the requirement information from different heterogeneous cluster groups included in a preset global resource pool; the different heterogeneous cluster groups are obtained by pre-dividing different storage clusters according to storage resource characteristic information corresponding to different storage clusters, and the storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster;

[0123] A management module 603, configured to select a target storage cluster from the target heterogeneous cluster group according to a preset selection policy, and allocate storage resources for the user from the target storage cluster.

[0124] Optionally, the storage resource characteristic information includes at least one of the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster.

[0125] Optionally, the selection module 602 is specifically configured to, when determining that the requirement information is the target storage type, select a target storage cluster from the target heterogeneous cluster group according to a preset first selection policy, and the first selection policy is used to select a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

[0126] Optionally, the selection module 602 is specifically configured to, when determining that the requirement information is a resource parameter, select a target storage cluster from the target heterogeneous cluster groups according to a preset second selection strategy, where the second selection strategy is used to select a target storage cluster according to the storage resource utilization rate distribution characteristics of each storage cluster in the target heterogeneous cluster groups, and the resource parameters include: protocol type, throughput, and number of input / output operations per second.

[0127] Optionally, the management module 603 is specifically configured to send a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources for the user according to the resource request and return the resource information of the storage resources allocated for the user; create a storage instance according to the resource information, so that the user can access the storage resources through the storage instance.

[0128] Optionally, the management module 603 is further configured to manage different levels of data stored in the storage resources allocated for the user according to the received data management operation instruction, where the data management operation instruction is determined according to the interaction operation performed by the user on different levels of data stored in the storage resources in a preset user interface.

[0129] Optionally, the management module 603 is further configured to, when determining that the expiration period of the storage resources allocated for the user is lower than a preset threshold, process the storage resources according to the expiration management policy corresponding to the storage resources allocated for the user; the expiration management policy includes: a first expiration management policy for instructing to prohibit at least part of the access rights of the user to the storage resources, a second expiration management policy for instructing to recycle the storage resources, and a third expiration management policy for instructing to migrate the data stored in the storage resources to other storage resources, where the performance level of the other storage resources is lower than the performance level of the storage resources.

[0130] Optionally, the management module 603 is further configured to, when determining that a preset data migration condition is met, create a migration task according to the obtained data source information and the destination address, and migrate the source data corresponding to the data source information to the storage resources corresponding to the destination address for storage according to a migration strategy matching the migration task.

[0131] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 provided method for managing a multi-source heterogeneous storage cluster.

[0132] This specification also provides Figure 7A schematic structural diagram of an electronic device corresponding to Figure 1 is shown. As Figure 7 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 multi-element heterogeneous storage cluster management method described above. Of course, in addition to the software implementation method, this specification does not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0133] The improvement of a technology can be clearly distinguished as either a hardware improvement (e.g., improvement of circuit structures such as diodes, transistors, switches, etc.) or a software improvement (improvement of method flows). However, with the development of technology, many improvements of method flows today can be regarded as direct improvements of hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structures by programming the improved method flows into the hardware circuits. Therefore, it cannot be said that an improvement of a method flow cannot be implemented by a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is such an integrated circuit whose logic function is determined by the user's programming of the device. The designer can program by himself / herself to "integrate" a digital system on a piece of PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0134] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0135] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0136] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0137] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0138] This specification is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 one or more of the blocks for implementing the specified functions.

[0139] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 one or more of the blocks.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 one or more of the blocks.

[0141] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0142] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0147] The various embodiments in this specification are described in a progressive manner. For the same or similar parts among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiments.

[0148] The above description is only for the embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A multi-heterogeneous storage cluster management method, characterized in that: The method is applied to a management device of a multi-heterogeneous storage cluster, and the method comprises: Determining demand information according to a storage request of a user, wherein the demand information is a performance level of a storage resource selected by the user through a preset user interface; A target heterogeneous cluster group matching the demand information is selected from the heterogeneous cluster groups included in the preset global resource pool; the heterogeneous cluster groups are obtained by dividing the storage clusters to be divided according to the performance level corresponding to each storage cluster to be divided, and the performance level is determined according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; the storage resource feature information includes: at least one of the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster; According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

2. The method according to claim 1, characterized in that According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, specifically including: When it is determined that the demand information is a target storage type, a target storage cluster is selected from the target heterogeneous cluster group according to a preset first selection strategy, wherein the first selection strategy is used to select a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

3. The method according to claim 1, characterized in that According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, specifically including: When it is determined that the demand information is a resource parameter, a target storage cluster is selected from the target heterogeneous cluster group according to a preset second selection strategy, and the second selection strategy is used to select the target storage cluster according to the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group, and the resource parameters include: protocol type, throughput, and number of input and output operations per second.

4. The method according to claim 1, characterized in that Allocating storage resources to the user from the target storage cluster specifically includes: Sending a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources to the user according to the resource request, and return resource information of the storage resources allocated to the user; A storage instance is created according to the resource information, so that the user can access the storage resource through the storage instance.

5. The method according to claim 1, characterized in that Also includes: According to the received data management operation instructions, the different levels of data stored in the storage resources allocated to the user are managed, and the data management operation instructions are determined based on the interactive operations performed by the user in a preset user interface on the different levels of data stored in the storage resources.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that the validity period of the storage resources allocated to the user is lower than a preset threshold, the storage resources are processed according to the expiration management policy corresponding to the storage resources allocated to the user; the expiration management policy includes: a first expiration management policy for indicating that the user is prohibited from having at least part of the access rights to the storage resources, a second expiration management policy for indicating that the storage resources are reclaimed, and a third expiration management policy for indicating that the data stored in the storage resources are migrated to other storage resources, wherein the performance level corresponding to the other storage resources is lower than the performance level corresponding to the storage resource.

7. The method according to claim 1, characterized in that The method further comprises: When it is determined that the preset data migration conditions are met, a migration task is created based on the acquired data source information and destination address, and the source data corresponding to the data source information is migrated to the storage resource corresponding to the destination address for storage according to a migration strategy matching the migration task.

8. A multi-heterogeneous storage cluster management system, characterized in that: The multi-heterogeneous storage cluster management system includes: a management device, a plurality of storage clusters; The management device is used to determine demand information according to a user's storage request, select a target heterogeneous cluster group matching the demand information from each heterogeneous cluster group included in a preset global resource pool, and select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and send a resource request to the target storage cluster through an adapter of the target storage cluster, and create a storage instance according to the resource information returned by the target storage cluster according to the resource request, so that the user can access the storage resource through the storage instance; the demand information is the performance level of the storage resource selected by the user through a preset user interface; the heterogeneous cluster groups are obtained by dividing each storage cluster to be divided according to the performance level corresponding to each storage cluster to be divided, and the performance level is determined according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; the storage resource feature information includes: at least one of the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster; The storage cluster is used to receive the resource request sent by the management device, allocate storage resources to the user according to the resource request, and send resource information of the storage resources allocated to the user to the management device.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images