Multivariate heterogeneous storage cluster management method and storage medium

Abstract

The invention discloses a multivariate heterogeneous storage cluster management method and a storage medium. In the method, all storage resources of the multi-element heterogeneous storage cluster can be managed and dynamically distributed in a centralized manner through the global resource pool of the management equipment, seamless data migration across different cloud computing environments and among a plurality of storage clusters is supported, the flexibility and adaptability of the system are enhanced, and the system reliability is improved. Therefore, the user can perform complete storage resource life cycle management from the aspects of creation, allocation, use, recovery and the like through the user interface without concerning the complex hardware configuration or network topology structure of the bottom layer, so that the operation and maintenance complexity can be reduced, human errors are reduced, and the user experience is improved. And the cost of the computing power center in the aspect of storage operation is obviously reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to a multi-heterogeneous storage cluster management method and storage medium. Background Art

[0002] With the rapid development of intelligent computing technology, computing centers are becoming increasingly important as core infrastructure supporting massive data processing and complex computing tasks. In the construction and operation of computing centers, storage devices from different storage vendors are usually purchased and used on demand. These storage devices from different storage vendors have significant differences in technical architecture, interface protocols, management tools, and performance characteristics, showing obvious heterogeneity. This heterogeneity greatly increases the complexity of resource management. Due to the different management methods of devices from different vendors, managers need to be familiar with and master a variety of management tools and interfaces in order to effectively monitor, configure, and maintain various storage devices, which leads to high management costs.

[0003] Therefore, how to reduce the cost of managing a multi-heterogeneous storage cluster is an urgent problem to be solved. Summary of the invention

[0004] This specification provides a multi-heterogeneous storage cluster management method and storage medium to partially solve the above-mentioned problems existing in the prior art.

[0005] This manual adopts the following technical solutions: This specification provides a multi-heterogeneous storage cluster management method. The method is applied to a management device of a multi-heterogeneous storage cluster, and the method comprises: Determine required information based on user's storage request; A target heterogeneous cluster group matching the demand information is selected from the heterogeneous cluster groups included in the preset global resource pool; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

[0006] Optionally, the storage resource characteristic information includes: at least one of: a network type used by the storage cluster, a storage medium type corresponding to the storage cluster, and a file system type used by the storage cluster.

[0007] Optionally, selecting a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy specifically includes: When it is determined that the demand information is a target storage type, a target storage cluster is selected from the target heterogeneous cluster group according to a preset first selection strategy, wherein the first selection strategy is used to select a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

[0008] Optionally, selecting a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy specifically includes: When it is determined that the demand information is a resource parameter, a target storage cluster is selected from the target heterogeneous cluster group according to a preset second selection strategy, and the second selection strategy is used to select the target storage cluster according to the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group, and the resource parameters include: protocol type, throughput, and number of input and output operations per second.

[0009] Optionally, allocating storage resources to the user from the target storage cluster specifically includes: Sending a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources to the user according to the resource request, and return resource information of the storage resources allocated to the user; A storage instance is created according to the resource information, so that the user can access the storage resource through the storage instance.

[0010] Optionally, it also includes: According to the received data management operation instructions, the different levels of data stored in the storage resources allocated to the user are managed, and the data management operation instructions are determined based on the interactive operations performed by the user in a preset user interface on the different levels of data stored in the storage resources.

[0011] Optionally, the method further comprises: When it is determined that the validity period of the storage resources allocated to the user is lower than a preset threshold, the storage resources are processed according to the expiration management policy corresponding to the storage resources allocated to the user; the expiration management policy includes: a first expiration management policy for indicating that the user is prohibited from having at least part of the access rights to the storage resources, a second expiration management policy for indicating that the storage resources are reclaimed, and a third expiration management policy for indicating that the data stored in the storage resources are migrated to other storage resources, wherein the performance level corresponding to the other storage resources is lower than the performance level corresponding to the storage resource.

[0012] Optionally, the method further comprises: When it is determined that the preset data migration conditions are met, a migration task is created based on the acquired data source information and destination address, and the source data corresponding to the data source information is migrated to the storage resource corresponding to the destination address for storage according to a migration strategy matching the migration task.

[0013] This specification provides a multi-heterogeneous storage cluster management system, the multi-heterogeneous storage cluster management system includes: a management device, a plurality of storage clusters; The management device is used to determine demand information according to the user's storage request, select a target heterogeneous cluster group that matches the demand information from the heterogeneous cluster groups included in the preset global resource pool, and select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and send a resource request to the target storage cluster through the adapter of the target storage cluster, and create a storage instance according to the resource information returned by the target storage cluster according to the resource request, so that the user can access the storage resource through the storage instance; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; The storage cluster is used to receive the resource request sent by the management device, allocate storage resources to the user according to the resource request, and send resource information of the storage resources allocated to the user to the management device.

[0014] This specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned multi-heterogeneous storage cluster management method is implemented.

[0015] This specification provides a management device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multi-heterogeneous storage cluster management method when executing the program.

[0016] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the multi-heterogeneous storage cluster management method provided in this specification, the demand information is first determined according to the user's storage request, and a target heterogeneous cluster group matching the demand information is selected from the heterogeneous cluster groups contained in the preset global resource pool, wherein each heterogeneous cluster group is obtained by pre-dividing different storage clusters according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster, and then according to the preset selection strategy, the target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

[0017] It can be seen from the above method that all storage resources of a multi-heterogeneous storage cluster can be centrally managed and dynamically allocated through the global resource pool of the management device, so that users do not need to worry about the underlying complex hardware configuration or network topology, which can help reduce the complexity of operation and maintenance, reduce human errors, and significantly reduce the cost of managing a multi-heterogeneous storage cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 A flowchart of a multi-heterogeneous storage cluster management method provided in this specification; Figure 2 A schematic diagram of the heterogeneous cluster groupings provided in this specification; Figure 3 A schematic diagram of the process of adding a storage cluster provided in this manual; Figure 4A A schematic diagram of a user interface for file management provided in this manual; Figure 4B A schematic diagram of a user interface for storage instance management provided in this specification; Figure 5 A schematic diagram of a multi-heterogeneous storage cluster management system provided in this specification; Figure 6 A schematic diagram of a multi-heterogeneous storage cluster management device provided in this specification; Figure 7 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0021] With the rapid development of intelligent computing technology, the importance of computing centers as core infrastructure supporting massive data processing and complex computing tasks has become increasingly prominent. In the construction and operation of computing centers, storage devices from different storage vendors are usually purchased and used on demand. These storage devices from different storage vendors have significant differences in technical architecture, interface protocols, management tools, and performance characteristics, showing obvious heterogeneity. This heterogeneity has brought many challenges to the storage resource management of computing centers. On the one hand, the complexity of resource management has increased significantly. Due to the different management methods of devices from different manufacturers, managers need to be familiar with and master a variety of management tools and interfaces in order to effectively monitor, configure and maintain various storage devices. This not only increases management costs, but also easily leads to unreasonable resource allocation and difficulty in troubleshooting equipment failures due to operational errors or poor management. On the other hand, the heterogeneity of storage devices also seriously affects the efficiency of use. Performance bottlenecks between different devices, incompatible data formats, and inflexible resource scheduling make it impossible to fully and efficiently utilize storage resources, which in turn affects the overall computing performance and business response speed of the computing center.

[0022] At present, solutions for managing heterogeneous storage often have problems such as poor compatibility, imperfect functions or performance bottlenecks, which make it difficult to meet the growing requirements of efficient and intelligent management of computing centers. Therefore, how to uniformly and efficiently manage heterogeneous storage clusters in new computing centers to reduce the cost of managing heterogeneous storage clusters has become a technical problem that needs to be solved in the field of intelligent computing.

[0023] Figure 1 A schematic diagram of a multi-heterogeneous storage cluster management method provided in this specification includes the following steps: S101: Determine demand information according to a storage request of a user.

[0024] In this specification, the above-mentioned multi-heterogeneous storage cluster management method is applied to a management device of a multi-heterogeneous storage cluster, that is, the multi-heterogeneous storage cluster management method can be executed by the management device of the multi-heterogeneous storage cluster to manage the multi-heterogeneous storage cluster.

[0025] Among them, a multi-heterogeneous storage cluster can refer to storage devices and storage technologies of different types and architectures, such as: Network Attached Storage (NAS), object storage services, General Parallel File System (GPFS) of traditional storage server vendors, as well as NAS, object storage and elastic block storage in private cloud environments.

[0026] In the above content, NAS is a storage solution that provides data access through the network, usually used for file-level storage management. It provides simple shared file storage functions, can support multi-user access, and is widely used in scenarios such as file sharing, backup and recovery. NAS devices usually have good scalability and flexibility, and can meet the needs of large amounts of data access, but there are certain bottlenecks in performance and scalability.

[0027] Object storage is an object-based storage method that does not rely on traditional file system structures and is suitable for the storage and management of massive data. Object storage services provide high scalability and high availability, and can support the storage and management of multiple data types (such as pictures, videos, log files, etc.). It accesses data through unique identifiers (such as object IDs), which is different from traditional file systems. Object storage is widely used in big data storage, cloud computing platforms and other fields.

[0028] GPFS is a high-performance parallel file system that is widely used in large-scale computing clusters and supports high-speed data access and sharing. GPFS can combine multiple nodes in a storage cluster to provide high-bandwidth, low-latency data read and write performance, and is suitable for large-scale parallel processing environments that require high-performance computing (HPC). Its advantages are that it can efficiently process massive amounts of data and support distributed computing and large-scale parallel data access.

[0029] Elastic Block Storage: Elastic Block Storage is a cloud-based block storage service that is commonly used for persistent storage of virtual machines and can elastically expand storage capacity based on demand. Block storage supports high-performance, low-latency storage requirements and is suitable for database and high-performance computing applications. Its elasticity and scalability make block storage a common storage solution in cloud computing platforms.

[0030] From the above content, it can be seen that the business platform can combine multiple storage technologies through management devices to meet different storage needs, such as file storage, object storage, block storage, etc. It can also manage storage resources for storage clusters with different technical backgrounds, functional characteristics and suppliers, thereby avoiding the heterogeneity between different storage devices making resource scheduling, load balancing, performance optimization, fault detection and other tasks more complicated, which in turn leads to high costs for managing multiple heterogeneous storage clusters.

[0031] Specifically, the management device can receive storage requests sent by users through the application layer through a preset resource gateway management module, and determine the user's demand information for the required storage resources based on the received storage requests through the resource center in the preset resource management module, and then allocate storage resources to the user based on the received demand information.

[0032] The above-mentioned management device may refer to a device integrated with a software program module for managing a multi-heterogeneous storage cluster.

[0033] The above application layer is used to interact with users based on the unified storage application programming interface (Application Programming Interface, API) gateway provided by the management device.

[0034] S102: Select a target heterogeneous cluster group that matches the demand information from the heterogeneous cluster groups included in the preset global resource pool; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster.

[0035] In this specification, after acquiring the user's demand information, the management device can select a target heterogeneous cluster group matching the user's demand information from various heterogeneous cluster groups included in a preset global resource pool through a resource management module.

[0036] It should be noted that the above-mentioned heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to storage resource feature information corresponding to different storage clusters.

[0037] Among them, the above-mentioned storage resource characteristic information is used to characterize the data access efficiency and data management method of the storage cluster. The above-mentioned storage resource characteristic information can be software information and / or hardware information, including: at least one of the network type used by the storage cluster, the storage medium type corresponding to the storage cluster, and the file system type used by the storage cluster.

[0038] The above-mentioned storage resource characteristic information may be, for example: information of storage nodes included in the storage cluster and / or information of a storage pool (composed of at least some storage nodes), logs, resource statistics information, etc.

[0039] In the above content, the information of the storage node may include: node identification information, storage medium type, capacity information, I / O performance, read and write speed, operating status (such as normal, faulty, maintenance), network information (such as network interface, bandwidth, data transmission protocol and other network configuration information), etc.

[0040] The information of the storage pool may include: the identification information of the storage pool, the component node information, the total capacity information and the available capacity information, the overall performance indicators (such as throughput, latency, etc.), the redundancy strategy (such as RAID, replication mechanism, erasure code, etc.), the storage protocol support, etc.

[0041] Resource statistics may include: storage resource allocation information of the storage cluster, the total capacity and available capacity of the storage cluster, storage resource utilization trend information of the storage cluster (such as the average storage utilization in the past month, etc.), load balancing status information (such as disk I / O, IOPS and other parameters of each storage node or storage pool), resource migration record information, etc.

[0042] Specifically, the management device may determine, for each storage cluster to be divided, the performance level corresponding to the storage cluster according to the storage resource feature information of the storage cluster to be divided.

[0043] For example, a higher performance level may be set for a storage cluster that adopts an efficient data transmission protocol and uses a solid-state drive as the storage medium.

[0044] For another example, a storage cluster that adopts the NFS data transmission protocol and uses a traditional mechanical hard disk as the storage medium may be set to a lower performance level.

[0045] Furthermore, the management device may divide each storage cluster to be divided into different heterogeneous cluster groups according to the performance level corresponding to each storage cluster to be divided, as shown in the following example: Figure 2 shown.

[0046] Figure 2 This is a schematic diagram of the heterogeneous cluster groupings provided in this specification.

[0047] Combination Figure 2 It can be seen that the management device can determine the performance level corresponding to each storage cluster in advance according to the storage resource characteristic information of different storage clusters, and then divide each storage cluster into heterogeneous cluster groups according to the different corresponding performance levels of each storage cluster. Figure 2In the table, level 0 represents high-performance storage resources, and 0-N represent decreasing performance levels.

[0048] It should be noted that in actual application scenarios, users can select the required performance level of storage resources as demand information through a preset user interface, so that the management device determines the performance level of storage resources required by the user based on the demand information, and determines the target heterogeneous cluster grouping based on the performance level of storage resources required by the user.

[0049] Of course, users can also input the resource type, resource parameters, etc. of the required storage resources as demand information, so that the management device can determine the performance level of the storage resources required by the user based on the user's demand information, and determine the target heterogeneous cluster grouping based on the performance level of the storage resources required by the user.

[0050] For ease of understanding, the following combination Figure 3 The process of adding a newly added storage cluster to the above-mentioned global resource pool for management is described in detail.

[0051] Figure 3 This is a schematic diagram of the process of adding a storage cluster provided in this manual.

[0052] Combination Figure 3 It can be seen that when a storage cluster needs to be added to the global resource pool for management through a management device, the adapter corresponding to the storage cluster can be obtained through the resource management module of the management device (one adapter corresponds to each vendor's storage) and saved for the adapter of the storage resource, thereby obtaining the storage resource characteristic information of the storage cluster, and determining the performance level corresponding to the storage cluster based on the storage resource characteristic information of the storage cluster, and adding the storage cluster to a heterogeneous cluster group that matches the performance level corresponding to the domain storage cluster based on the performance level corresponding to the storage cluster.

[0053] In the process of managing various heterogeneous cluster groups through the global resource pool, the management device can also regularly perform resource calibration with each storage cluster through the resource management module. That is, for each storage cluster, it is detected whether the storage resource characteristic information of the storage cluster stored in the global resource pool is consistent with the actual storage resource characteristic information of the storage cluster. If inconsistent, corrections can be made.

[0054] S103: According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

[0055] In this specification, after determining the target heterogeneous cluster group, the management device may select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and allocate storage resources to the user from the target storage cluster.

[0056] There may be multiple selection strategies, for example: a first selection strategy for selecting a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

[0057] Another example is a second selection strategy for selecting a target storage cluster based on the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group (the storage resource usage distribution characteristics here are used to reflect the usage of storage resources and their changing patterns within a certain period of time, such as average usage, peak usage, usage variance, etc.). By selecting the target storage cluster based on the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group, it is possible to avoid allocating too much data to a single high-load storage cluster, thereby balancing the workload of each storage cluster.

[0058] In actual application scenarios, in order to improve the efficiency of users managing heterogeneous storage clusters through management devices, users can also choose to input the required storage resource type through a preset user interface as the target storage resource type according to actual needs, so that when the management device determines that the user needs to use the storage resource of the target storage resource type based on the acquired demand information, it can select the target storage cluster from the target heterogeneous cluster group according to the preset first selection strategy.

[0059] Among them, the first selection strategy can select a storage cluster with the largest remaining capacity from each storage cluster in the target heterogeneous cluster group according to the remaining capacity of each storage cluster in the target heterogeneous cluster group as the target storage cluster.

[0060] In addition, users can also choose to input the resource parameters of the required storage resources through a preset user interface according to actual needs, so that when the management device determines that the user needs to use storage resources that meet the above resource parameters based on the acquired demand information, it can select the target storage cluster from the target heterogeneous cluster grouping according to the preset second selection strategy.

[0061] Among them, the second selection strategy can be used to select the target storage cluster based on the storage resource utilization distribution characteristics of each storage cluster in the target heterogeneous cluster group. That is, an adaptive algorithm can be used to select the storage cluster with lower resource utilization as the target storage cluster based on the storage resource utilization distribution characteristics of each storage cluster, so as to keep the storage resource utilization of each storage cluster in the target heterogeneous cluster group balanced.

[0062] Furthermore, after determining the target storage cluster, the management device can obtain the adapter of the target storage cluster from the pre-saved adapters, and then send a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources to the user according to the resource request, and return the resource information of the storage resources allocated to the user, and then create a storage instance according to the resource information of the storage resources allocated to the user, so that the user can access the storage resources through the storage instance.

[0063] The storage instance may be a file directory or a bucket of specified capacity.

[0064] It should be noted that after the management device creates a storage instance for the user, the storage instance can be managed by the resource gateway management module according to the operation instructions input by the user through the user interface, as shown in the following example. Figure 4A , Figure 4B shown.

[0065] Figure 4A A schematic diagram of the user interface for file management provided in this manual.

[0066] Combination Figure 4A It can be seen that after a storage instance is created for a user, the user can perform data management operations such as uploading and downloading files for the storage instance through a page component for managing files in the storage instance provided in the user interface.

[0067] Figure 4B A schematic diagram of a user interface for storage instance management provided in this specification.

[0068] Combination Figure 4B It can be seen that after a storage instance is created for a user, the user can perform change operations such as expansion, editing, and deletion on the storage instance through the page components for managing the storage instance provided in the user interface, and synchronize the change operations performed by the user to the global resource pool.

[0069] Among them, the management module can also manage different levels of data stored in the storage resources allocated to the user according to the received data management operation instructions. The data management operation instructions here are determined based on the interactive operations performed by the user in the preset user interface on the different levels of data stored in the storage resources.

[0070] The above-mentioned data at different levels may refer to directory-level data or file-level data. The directory-level data here may refer to the directory structure data contained in the storage instance, and the file-level data here may refer to the content data in the specific file contained in the storage instance.

[0071] From the above content, it can be seen that when users manage storage resources in a multi-heterogeneous storage cluster through management devices, they can perform management operations directly through the user interface without considering the significant differences in technical architecture, interface protocols, management tools, and performance characteristics of storage devices from different storage vendors.

[0072] Furthermore, after creating a storage instance for the user so that the user can access the storage resources allocated to the user through the storage instance, the management device can also obtain metering information of the storage resources allocated to the user, so that when it is determined according to the metering information of the storage resources allocated to the user that the validity period of the storage resources allocated to the user is lower than a preset threshold, the storage resources can be processed according to the expiration management policy corresponding to the storage resources allocated to the user.

[0073] Among them, the above-mentioned expiration management policies include: a first expiration management policy for indicating prohibition of users from at least partial access rights to storage resources, a second expiration management policy for indicating recycling of storage resources, and a third expiration management policy for indicating migration of data stored in storage resources to other storage resources, wherein the performance level corresponding to the above-mentioned other storage resources is lower than the performance level corresponding to the storage resource.

[0074] It should be noted that the method in which the management device manages the storage instance and data of different levels in the storage instance according to the operation instructions input by the user through the user interface can be implemented on the K8s / ACK cloud native infrastructure.

[0075] Specifically, for each storage instance, when the management device creates the storage instance, it can also create a deployment unit Pod for the storage instance in k8s, so that each storage instance can be managed separately through the Pod, thereby setting up separate lifecycle management for each storage instance and isolating the resources of each storage instance from other storage instances.

[0076] Among them, the lifecycle management of each of the above-mentioned storage instances can be set according to actual needs. For example, when a user obtains storage resources for model training, high-performance storage resources can be provided to the computing power cluster used by the user when the user performs model training. When feedback information indicating the end of model training is received from the computing power cluster, it is deemed that the validity period of the storage resources allocated to the user is lower than the preset threshold, and the storage resources can be processed according to the expiration management policy corresponding to the storage resources allocated to the user.

[0077] In addition, when the management device determines that the preset data migration conditions are met, it can also create a migration task through the preset data migration module based on the acquired data source information and destination address, and migrate the source data corresponding to the data source information to the storage resources corresponding to the destination address for storage according to the migration strategy matching the migration task.

[0078] Among them, the above-mentioned data migration conditions can be determined according to actual needs. For example, after the user obtains the storage instance allocated by the management device, the user can migrate the data stored in other storage devices to the above-mentioned storage instance. At this time, the user can configure the data source address and the destination address through the user interface, so that after the management device receives the data migration request sent by the user through the user interface, it can be regarded as meeting the preset data migration conditions. Then, according to the data migration request, the data source address and the destination address configured by the user can be obtained, and a migration task can be created. Then, according to the migration strategy matching the migration task, the source data corresponding to the data source information can be migrated to the storage resource corresponding to the destination address for storage.

[0079] As another example: when the management device determines that the validity period of the storage resources allocated to the user is lower than a preset threshold, it can be deemed that the preset data migration conditions are met, and the data source information corresponding to the storage resources allocated to the user can be obtained, as well as the preset destination address to create a migration task, and then the source data corresponding to the data source information can be migrated to the storage resources corresponding to the destination address for storage according to the migration strategy that matches the migration task.

[0080] In the above content, the number of migration threads used when performing data migration according to different migration strategies and / or the maximum bandwidth used during migration are different, wherein the migration strategy here can be configured in advance by the user according to actual needs.

[0081] It should be noted that in actual application scenarios, management devices often need to migrate data between different types of storage systems, for example: data migration between file systems, data migration between file systems and object systems, data migration between object systems and object systems, and data migration between object systems and file systems.

[0082] Therefore, the above-mentioned data source information may include, in addition to the address of the data to be migrated (which may be the directory or path where the data to be migrated is located) or the storage bucket where the data to be migrated is located, the protocol information required to access the data to be migrated, such as: Network File System (NFS) protocol, Server Message Block (SMB) protocol, object storage protocol S3, etc.

[0083] Furthermore, after obtaining the above-mentioned data source information, the management device can determine the address of the file to be migrated and the destination address based on the data source information, and then can determine the protocol information required for data migration based on the data source information to create a migration task, so as to migrate the source data corresponding to the data source information to the storage resources corresponding to the destination address for storage according to the migration strategy that matches the migration task.

[0084] It is worth noting that when the management device is executing a migration task to migrate data, it can convert the protocol through the preset migration gateway management module to ensure data interoperability between different storage clusters and handle data conversion in different formats and encoding methods to ensure data compatibility between different storage systems.

[0085] In actual application scenarios, users can send storage requests to the management device through the application layer, so that the management device can allocate storage resources to the user according to the storage request sent by the user, and then connect the storage resources allocated to the user to the computing power cluster, so that the user can perform tasks through the computing power cluster (for example: model training task execution).

[0086] In this process, since different computing clusters are usually composed of multiple different types of computing resources, such as CPU, GPU, FPGA, ASIC, etc., these resources have significant differences in architecture, performance characteristics and application scenarios, which leads to different computing clusters having different access methods for storage resources. Therefore, the management device can also select the target access method from the preset access methods for different computing clusters through the resource gateway management module, and connect the storage resources allocated to the user to the computing cluster according to the target access method.

[0087] It can be seen from the above method that all storage resources of multi-heterogeneous storage clusters can be centrally managed and dynamically allocated through the global resource pool of the management device, and seamless data migration across different cloud computing environments and multiple storage clusters can be supported, which enhances the flexibility and adaptability of the system, so that users can perform complete storage resource life cycle management from creation, allocation, use to recycling through the user interface without having to worry about the complex underlying hardware configuration or network topology. This can help reduce the complexity of operation and maintenance, reduce human errors, and significantly reduce the storage operation costs of the computing center.

[0088] For ease of understanding, this specification also provides a multi-heterogeneous storage cluster management system, as shown in the following example: Figure 5 shown.

[0089] Figure 5This is a schematic diagram of a multi-heterogeneous storage cluster management system provided in this specification.

[0090] Combination Figure 5 It can be seen that the above-mentioned multi-heterogeneous storage cluster management system may include: a management device and multiple storage clusters.

[0091] Among them, the above-mentioned management device is used to determine the demand information according to the user's storage request, select the target heterogeneous cluster group matching the demand information from the heterogeneous cluster groups contained in the preset global resource pool, and select the target storage cluster from the target heterogeneous cluster group according to the preset selection strategy, and send a resource request to the target storage cluster through the adapter of the target storage cluster, and create a storage instance according to the resource information returned by the target storage cluster according to the resource request, so that the user can access the storage resource through the storage instance; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster.

[0092] The storage cluster is used to receive resource requests sent by the management device, allocate storage resources to users according to the resource requests, and send resource information of the storage resources allocated to the users to the management device.

[0093] Specifically, the above-mentioned management device integrates a resource management module, a resource gateway management module, a unified storage API gateway, and a data migration module.

[0094] From the above content, it can be seen that all storage resources of multi-heterogeneous storage clusters can be centrally managed and dynamically allocated through the resource management module, resource gateway management module, unified storage API gateway, data migration module and other software program modules contained in the management device, as well as support for seamless data migration across different cloud computing environments and multiple storage clusters, thereby enhancing the flexibility and adaptability of the system, so that users can perform complete storage resource life cycle management from creation, allocation, use to recycling through the user interface without having to worry about the underlying complex hardware configuration or network topology. This can help reduce the complexity of operation and maintenance, reduce human errors, and significantly reduce the storage operation costs of computing centers.

[0095] The above are one or more implementations of the multi-heterogeneous storage cluster management method of this specification. Based on the same idea, this specification also provides a corresponding multi-heterogeneous storage cluster management device, such as Figure 6 shown.

[0096] Figure 6A schematic diagram of a multi-heterogeneous storage cluster management device provided for this specification includes: Determining module 601, used to determine demand information according to a storage request of a user; A selection module 602 is used to select a target heterogeneous cluster group matching the demand information from various heterogeneous cluster groups included in a preset global resource pool; the various heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; The management module 603 is used to select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and allocate storage resources to the user from the target storage cluster.

[0097] Optionally, the storage resource characteristic information includes: at least one of: a network type used by the storage cluster, a storage medium type corresponding to the storage cluster, and a file system type used by the storage cluster.

[0098] Optionally, the selection module 602 is specifically used to, when it is determined that the demand information is the target storage type, select a target storage cluster from the target heterogeneous cluster group according to a preset first selection strategy, and the first selection strategy is used to select the target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

[0099] Optionally, the selection module 602 is specifically used to, when determining that the demand information is a resource parameter, select a target storage cluster from the target heterogeneous cluster group according to a preset second selection strategy, the second selection strategy being used to select the target storage cluster according to the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group, and the resource parameters include: protocol type, throughput, and number of input and output operations per second.

[0100] Optionally, the management module 603 is specifically used to send a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources to the user according to the resource request and return resource information of the storage resources allocated to the user; create a storage instance based on the resource information, so that the user can access the storage resources through the storage instance.

[0101] Optionally, the management module 603 is also used to manage different levels of data stored in the storage resources allocated to the user according to received data management operation instructions, and the data management operation instructions are determined based on the interactive operations performed by the user in a preset user interface on the different levels of data stored in the storage resources.

[0102] Optionally, the management module 603 is also used to, when it is determined that the validity period of the storage resources allocated to the user is lower than a preset threshold, process the storage resources according to the expiration management policy corresponding to the storage resources allocated to the user; the expiration management policy includes: a first expiration management policy for indicating prohibition of the user's at least partial access rights to the storage resource, a second expiration management policy for indicating recycling of the storage resource, and a third expiration management policy for indicating migration of data stored in the storage resource to other storage resources, wherein the performance level corresponding to the other storage resources is lower than the performance level corresponding to the storage resource.

[0103] Optionally, the management module 603 is also used to, when it is determined that the preset data migration conditions are met, create a migration task based on the acquired data source information and destination address, and migrate the source data corresponding to the data source information to the storage resources corresponding to the destination address for storage according to a migration strategy matching the migration task.

[0104] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A multi-heterogeneous storage cluster management method is provided.

[0105] This manual also provides Figure 7 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 7 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0106] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0107] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0108] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0109] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0110] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0116] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0120] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0121] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A multi-heterogeneous storage cluster management method, characterized in that: The method is applied to a management device of a multi-heterogeneous storage cluster, and the method comprises: Determine required information based on user's storage request; A target heterogeneous cluster group matching the demand information is selected from the heterogeneous cluster groups included in the preset global resource pool; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, and storage resources are allocated to the user from the target storage cluster.

2. The method according to claim 1, characterized in that The storage resource characteristic information includes: at least one of: a network type used by the storage cluster, a storage medium type corresponding to the storage cluster, and a file system type used by the storage cluster.

3. The method according to claim 1, characterized in that According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, specifically including: When it is determined that the demand information is a target storage type, a target storage cluster is selected from the target heterogeneous cluster group according to a preset first selection strategy, wherein the first selection strategy is used to select a target storage cluster according to the remaining capacity of each storage cluster in the target heterogeneous cluster group.

4. The method according to claim 1, characterized in that According to a preset selection strategy, a target storage cluster is selected from the target heterogeneous cluster group, specifically including: When it is determined that the demand information is a resource parameter, a target storage cluster is selected from the target heterogeneous cluster group according to a preset second selection strategy, and the second selection strategy is used to select the target storage cluster according to the storage resource usage distribution characteristics of each storage cluster in the target heterogeneous cluster group, and the resource parameters include: protocol type, throughput, and number of input and output operations per second.

5. The method according to claim 1, characterized in that Allocating storage resources to the user from the target storage cluster specifically includes: Sending a resource request to the target storage cluster through the adapter of the target storage cluster, so that the target storage cluster can allocate storage resources to the user according to the resource request, and return resource information of the storage resources allocated to the user; A storage instance is created according to the resource information, so that the user can access the storage resource through the storage instance.

6. The method according to claim 1, characterized in that Also includes: According to the received data management operation instructions, the different levels of data stored in the storage resources allocated to the user are managed, and the data management operation instructions are determined based on the interactive operations performed by the user in a preset user interface on the different levels of data stored in the storage resources.

7. The method according to claim 1, characterized in that The method further comprises: When it is determined that the validity period of the storage resources allocated to the user is lower than a preset threshold, the storage resources are processed according to the expiration management policy corresponding to the storage resources allocated to the user; the expiration management policy includes: a first expiration management policy for indicating that the user is prohibited from having at least part of the access rights to the storage resources, a second expiration management policy for indicating that the storage resources are reclaimed, and a third expiration management policy for indicating that the data stored in the storage resources are migrated to other storage resources, wherein the performance level corresponding to the other storage resources is lower than the performance level corresponding to the storage resource.

8. The method according to claim 1, characterized in that The method further comprises: When it is determined that the preset data migration conditions are met, a migration task is created based on the acquired data source information and destination address, and the source data corresponding to the data source information is migrated to the storage resource corresponding to the destination address for storage according to a migration strategy matching the migration task.

9. A multi-heterogeneous storage cluster management system, characterized in that: The multi-heterogeneous storage cluster management system includes: a management device, a plurality of storage clusters; The management device is used to determine demand information according to the user's storage request, select a target heterogeneous cluster group that matches the demand information from the heterogeneous cluster groups included in the preset global resource pool, and select a target storage cluster from the target heterogeneous cluster group according to a preset selection strategy, and send a resource request to the target storage cluster through the adapter of the target storage cluster, and create a storage instance according to the resource information returned by the target storage cluster according to the resource request, so that the user can access the storage resource through the storage instance; the heterogeneous cluster groups are obtained by dividing different storage clusters in advance according to the storage resource feature information corresponding to different storage clusters, and the storage resource feature information is used to characterize the data access efficiency and data management method of the storage cluster; The storage cluster is used to receive the resource request sent by the management device, allocate storage resources to the user according to the resource request, and send resource information of the storage resources allocated to the user to the management device.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images