Storage resource management method, device, equipment and computer-readable storage medium

Abstract

The present application provides a storage resource management method, apparatus, device and computer-readable storage medium, relating to the field of cloud technology, the method comprising: receiving a resource adjustment request for a target training business; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training business, and the storage resources are used to store the training data used by the target computing power container to execute the target training business; in response to the resource adjustment request, verifying the amount of resources to be adjusted and the permission information carried by the resource adjustment request; if the amount of resources to be adjusted and the permission information are verified, sending a resource adjustment instruction to the network storage device; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container. The storage resource management method provided by the present application can improve the operating efficiency of the training business and improve the stability of the container cluster.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a storage resource management method, apparatus, device, and computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. Graphics Processing Units (GPUs) are also increasingly popular due to their superior computing power. GPUs are often used for computing in various scenarios, such as training neural network models in the field of AI.

[0003] In current model training platforms, it is often necessary to allocate a corresponding computing power container for each training business, and run the corresponding training business through the computing power container. At the same time, corresponding storage resources are also allocated for each training business, and the training data generated by the training business during the operation process is stored through the storage resources.

[0004] In traditional technologies, adjusting the storage resources corresponding to each training business often causes problems such as re-mounting or re-mapping, affecting the normal operation of the training business and reducing the operating efficiency of the training business. Summary of the Invention

[0005] Embodiments of the present application provide a storage resource management method, apparatus, device, and computer-readable storage medium, which can improve the operating efficiency of training services and enhance the stability of container clusters.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] An embodiment of the present application provides a storage resource management method, including:

[0008] Receive a resource adjustment request for a target training business; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training business, and the storage resources are used to store the training data used by the target computing power container to execute the target training business; in response to the resource adjustment request, verify the amount of resources to be adjusted and the permission information carried in the resource adjustment request; if the verification of the amount of resources to be adjusted and the permission information is passed, send a resource adjustment instruction to the network storage device; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

[0009] In some embodiments of the present application, the verification of the amount of resources to be adjusted and the permission information carried in the resource adjustment request includes: obtaining the registration information of the target training service and the resource allocation data of the network storage device; the registration information includes the minimum amount of resources for the target training service; based on the minimum amount of resources and the resource allocation data, verifying the amount of resources to be adjusted carried in the resource adjustment request; if the verification of the amount of resources to be adjusted is passed, verifying the permission information carried in the resource adjustment request.

[0010] In some embodiments of the present application, the verification of the permission information carried in the resource adjustment request includes at least one of the following: obtaining the resource directory corresponding to the resource adjustment request, determining whether the permission requirements of the resource directory are met based on the permission information, and if the permission information meets the permission requirements of the resource directory, determining that the permission information verification is successful; obtaining the registration identifier in the permission information, detecting whether the registration identifier belongs to a network storage device, and if the registration identifier belongs to a network storage device, determining that the permission information verification is successful.

[0011] In some embodiments of the present application, the method further includes: monitoring the container cluster and the network storage device to determine load information; and generating an alarm message based on the load information.

[0012] In some embodiments of the present application, the load information includes traffic information; generating an alarm message based on the load information includes: obtaining the container traffic corresponding to each computing power container in multiple computing power containers in the container cluster; the container traffic is the amount of data transmitted between the computing power container and the network storage device; when the container traffic is higher than a preset traffic threshold, generating a traffic alarm message and limiting the traffic of the computing power container corresponding to the container traffic.

[0013] In some embodiments of the present application, the load information includes hardware information; generating an alarm message based on the load information includes: obtaining the hardware load corresponding to each computing power container in a plurality of computing power containers in a container cluster; the hardware load includes at least one of the following: processor load, memory load, and disk input and output load; based on the hardware load corresponding to each computing power container, determining the hardware load quantization value of each computing power container; when the hardware load quantization value is greater than or equal to a preset hardware load threshold, sending a hardware alarm message to a message queue; the message queue is used to push the hardware alarm message to operation and maintenance personnel.

[0014] In some embodiments of the present application, determining the hardware load quantization value of each computing power container based on the hardware load corresponding to each computing power container includes: obtaining load data of each hardware type in the computing power container based on the hardware load corresponding to the computing power container; determining the load weight of each hardware type according to the business type of the training business running in the computing power container; and determining the hardware load quantization value of the computing power container based on the load data and load weight of each hardware type.

[0015] In some embodiments of the present application, the method also includes: receiving a registration request from the user side for the training service to be registered; the registration request carries registration information of the training service to be registered; the registration information includes the minimum resource amount, the resource amount to be applied for and the permission information to be applied for of the training service to be registered; in response to the registration request, sending a verification request to the network storage device; the network storage device is used to allocate storage resources for the training service to be registered based on the registration information when the registration information is verified, and send the registration identifier of the training service to be registered to the storage management device; storing the registration identifier of the training service to be registered sent by the network storage device, and forwarding the registration identifier of the training service to be registered to the user side, so that the user side carries the registration identifier of the training service to be registered when initiating a resource adjustment request for the training service to be registered.

[0016] An embodiment of the present application provides a storage resource management device, the device comprising:

[0017] A receiving module, configured to receive a resource adjustment request for a target training service; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training service, and the storage resources are used to store training data used by the target computing power container to execute the target training service;

[0018] A verification module, configured to respond to a resource adjustment request and verify the amount of resources to be adjusted and permission information carried in the resource adjustment request;

[0019] The sending module is used to send a resource adjustment instruction to the network storage device when the resource amount to be adjusted and the permission information are verified. The resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

[0020] An embodiment of the present application provides a storage resource management device, including:

[0021] a memory for storing executable instructions;

[0022] The processor is used to implement the storage resource management method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the storage resource management method provided in the embodiment of the present application.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] The embodiment of the present application receives a resource adjustment request for a target training business; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training business, and the storage resources are used to store the training data used by the target computing power container to execute the target training business; in response to the resource adjustment request, the amount of resources to be adjusted and the permission information carried in the resource adjustment request are verified; if the amount of resources to be adjusted and the permission information are verified, a resource adjustment instruction is sent to the network storage device; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container. The embodiment of the present application implements the task of allocating storage resources in the network storage device through an external storage management device. Compared with the solution of allocating storage resources using the master node device in the container cluster in the related art, the embodiment of the present application can not only avoid the problem of computing power resource waste caused by re-mounting or re-mapping in the process of adjusting the storage resources corresponding to the target computing power container, thereby improving the operating efficiency of the training business; it can also avoid the risk of misoperation on the user side when accessing the master node device, thereby improving the stability of the container cluster. At the same time, since the present application also responds to the resource adjustment request and verifies the resource adjustment request, the security of the training data in the network storage device can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an optional architectural diagram of the storage resource management system provided in an embodiment of the present application;

[0027] Figure 2 This is a schematic diagram of the structure of a storage resource management device provided in an embodiment of the present application;

[0028] Figure 3 This is an optional flowchart of the storage resource management method provided in the embodiment of the present application;

[0029] Figure 4 This is an optional flowchart of the storage resource management method provided in the embodiment of the present application;

[0030] Figure 5 This is an optional flowchart of the storage resource management method provided in the embodiment of the present application;

[0031] Figure 6 This is an optional flowchart of the storage resource management method provided in the embodiment of the present application;

[0032] Figure 7 This is a schematic diagram of the system architecture of an optional external center provided in an embodiment of the present application;

[0033] Figure 8 This is a flow chart of an optional scaling process provided in an embodiment of the present application;

[0034] Figure 9 This is a flowchart of an optional monitoring alarm processing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described in this embodiment can be implemented in an order other than that illustrated or described in this embodiment.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] (1) External center: The GPU devices used by users for AI training are uniformly managed and scheduled by the container management system (such as the Kubernetes system). The storage of training data requires the deployment of a storage cluster. For the deployed storage cluster, a management center is built to control the network or storage cluster permissions and resource load. When the cluster load is high, the capacity is expanded through this management center. When the cluster load is low, the capacity is reduced through this management center. In addition, the authentication and monitoring processes are implemented during the expansion and reduction operations. The management center built outside the Kubernetes system architecture is called an external center.

[0041] (2) Kubernetes: It is an open source platform for managing containerized applications on multiple hosts in a cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0042] (3) Container: refers to the smallest deployable unit in a node cluster. A container unit includes one or more containers, and all containers share the same network namespace, that is, have the same address information as the container unit. In the embodiment of the present application, the container unit is used for an internal container. Since the container is used to perform training business, the container is also called a computing power container in the embodiment of the present application.

[0043] (4) Computing power: Also known as hash rate, it is a unit of measurement for network processing power. In some embodiments, computing power can be the speed at which a hash function output is calculated. For example, when a device's computing power reaches a hash rate of 10 Th / s, it indicates that the device can perform 10 trillion calculations per second.

[0044] See also Figure 1 , Figure 1 This is an optional architectural diagram of the storage resource management system 100 provided in an embodiment of the present application. To support a storage resource management application, the terminal 400-1 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. Figure 1 It is also shown that the server 200 can be a server cluster, which includes servers 200-1 to 200-3. Similarly, the servers 200-1 to 200-3 can be physical machines or virtual machines built using virtualization technology (such as container technology and virtual machine technology). This embodiment of the present application is not limited to this. Of course, in this embodiment, a single server can also be used to provide services. The user can access the server 200 through the terminal 400-1 to configure and adjust the storage resource management system 100.

[0045] In some embodiments of the present application, multiple servers in the server cluster of the embodiments of the present application can be combined into a blockchain, and the servers are nodes on the blockchain.

[0046] The methods provided in the embodiments of this application also relate to the field of artificial intelligence. Artificial intelligence is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, machine learning / deep learning, and autonomous driving.

[0048] With the development of artificial intelligence, AI training is required to train AI models in more and more scenarios. When running AI training, computing resources, such as GPU computing resources, need to be delivered. For example, the AI ​​training business can be placed in a computing container with computing resources, and certain network storage resources can be allocated to the computing container so that the computing container can complete the AI ​​training task.

[0049] The method provided in the embodiment of the present application also relates to the field of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the rapid development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0050] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0051] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0052] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0053] See also Figure 2 , Figure 2 is a structural diagram of a storage resource management device 500 provided in an embodiment of the present application, Figure 2 The storage resource management device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the storage resource management device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not shown in FIG. Figure 2 Various buses are labeled as bus system 540 .

[0054] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 550 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices physically remote from the processor 510.

[0057] In some embodiments of the present application, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset of any one of the programs, modules, and data structures, as exemplified below.

[0058] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0059] A network communication module 552 for reaching other computing devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, WiFi, and USB;

[0060] a display module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, speakers, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0061] The input processing module 554 is configured to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

[0062] In some embodiments of the present application, the storage resource management device provided in the embodiments of the present application can be implemented in a combination of software and hardware. As an example, the storage resource management device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the storage resource management method provided in the embodiments of the present application.

[0063] In some embodiments of the present application, the storage resource management device provided in the embodiments of the present application can be implemented in software. Figure 2 A storage resource management device 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a receiving module 5551, a verification module 5552, and a sending module 5553. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.

[0064] The functions of each module will be described below.

[0065] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the storage resource management method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0066] The method provided in the embodiments of the present application can be applied to a storage resource management device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smartwatch, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0067] The following will take the storage resource management device being a server as an example to illustrate the storage resource management method provided by the embodiment of the present application. Figure 3 , Figure 3 This is an optional flow chart of the storage resource management method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0068] In step 301, a resource adjustment request for a target training business is received; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training business, and the storage resources are used to store the training data used by the target computing power container to execute the target training business.

[0069] In some embodiments of the present application, the container cluster is used to provide computing resources for the model training business, and the network storage device is used to provide storage resources for the model training business. For example, the container cluster can be a container cluster such as Kubernetes, and the network storage device can be a network disk device of any storage structure, such as Direct Attached Storage (DAS), Network Attached Storage (NAS), and Storage Area Network (SAN). The container cluster and the network storage device can form a model training platform. Among them, the container cluster can include multiple computing power containers. In the process of each computing power container executing the corresponding training business, the container cluster can allocate corresponding computing power resources to each computing power container. In order to ensure isolation between businesses, different businesses are configured with corresponding containers, and each container is configured with a corresponding computing power resource quota; the network storage device can allocate corresponding storage resources to each computing power container. The embodiments of the present application mainly solve the allocation and management problems of the storage resources corresponding to each computing power container.

[0070] In some embodiments of the present application, before executing step 301, the container cluster has allocated a target computing power container for the target training business, and the target computing power container is used to execute the target training business. In the process of the target computing power container executing the target training business, a certain storage space is required to store the training data generated by the training process. The training data may include the initial neural network model to be trained that is pre-stored before the training process is prepared, and may also include the data set used for training (training set, verification set and test set), and may also include the intermediate model generated during the training process and the trained neural network model generated after the training is completed. This application does not limit this.

[0071] In some embodiments of the present application, due to changes in actual training needs, the user side needs to adjust the storage resources used by each computing container in the current container cluster. For example, when it is necessary to enhance the model generalization capability of the first training business and add different types of training samples to complete model training, it is necessary to increase the storage resources corresponding to the first training business to store the above-mentioned different types of training samples and intermediate training data; when it is necessary to add a second training business to the model training platform, or when the first training business has been completed, it is necessary to reduce the storage resources corresponding to the above-mentioned first training business. The above examples are for the purpose of facilitating understanding of the scenario where the user side initiates a resource adjustment request, and are not limitations on the embodiments of the present application. The embodiments of the present application can also be applied to other scenarios where the storage resources of the training business need to be adjusted.

[0072] In some embodiments of the present application, the resource adjustment request may include an expansion request and a reduction request, wherein the reduction request is used to instruct the network storage device to reduce the storage resources allocated to the target computing power container; the expansion request is used to instruct the network storage device to increase the storage resources allocated to the target computing power container.

[0073] In some embodiments of the present application, the resource adjustment request also carries the service identifier of the target training service. Through the service identifier, the target computing power container corresponding to the target training service can be determined in the container cluster, and the storage resources allocated for the target training service can be determined in the network storage device through the service identifier.

[0074] In step 302, in response to the resource adjustment request, the amount of resources to be adjusted and the permission information carried in the resource adjustment request are verified.

[0075] In some embodiments of the present application, the resource adjustment request also carries the amount of resources to be adjusted. The amount of resources to be adjusted is used to represent the total amount of storage resources currently required by the target training service. For example, the amount of resources to be adjusted can directly represent the total amount of storage resources currently required by the target training service, or the amount of resources to be adjusted can also represent the amount of resources currently required to be adjusted by the target training service. The amount of resources already allocated to the target training service can be obtained using the service identifier of the target training service carried in the resource adjustment request. Based on the amount of resources to be adjusted and the amount of resources already allocated, the total amount of storage resources currently required by the target training service can be obtained. During the verification of the amount of resources to be adjusted, it is necessary to determine whether the amount of resources to be adjusted is reasonable, i.e., whether the total amount of storage resources currently required by the target training service represented by the amount of resources to be adjusted is reasonable. For example, this can be determined by determining whether the total amount of storage resources currently required by the target training service does not meet the actual business needs of the target training service; or by determining whether the total amount of storage resources currently required by the target training service exceeds the maximum amount of storage resources of the network storage device.

[0076] In some embodiments of the present application, the resource adjustment request also carries permission information, wherein the permission information may be user permissions corresponding to the user identifier of the user side, and the user permissions may be used to determine whether the user side has permission to adjust the storage resource.

[0077] In some embodiments of the present application, the resource amount to be adjusted and the permission information need to be verified separately. When both are verified successfully, step 303 is executed.

[0078] In step 303, when the resource amount to be adjusted and the permission information are verified, a resource adjustment instruction is sent to the network storage device; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

[0079] In some embodiments of the present application, if the resource adjustment request is a capacity expansion request, after passing the above verification, a corresponding resource adjustment instruction, i.e., a capacity expansion instruction, will be sent to the network storage device. The capacity expansion instruction is used to instruct the network storage device to increase the resource quota for the target computing capacity container.

[0080] In some embodiments of the present application, if the resource adjustment request is a scaling-down request, after passing the above verification, a corresponding resource adjustment instruction, i.e., a scaling-down instruction, is sent to the network storage device. The scaling-down instruction is used to instruct the network storage device to reduce the resource quota for the target computing capacity container.

[0081] In some embodiments, the relevant technology is to allocate the storage resources of the network storage device through the master node container in the container cluster. For example, the user side can access the master node container and use commands such as kubectl to perform expansion and contraction operations on data volumes or storage directories. In the process of the user side accessing the master node container, all computing power containers in the container cluster are exposed to the operation of the user side. Once an erroneous operation occurs, it will affect the computing power allocation of the entire container cluster and the execution process of the training business. At the same time, after the above-mentioned kubectl and other command operations, the master node container needs to reload the configuration file for allocating storage resources, that is, it needs to be remounted or remapped to make the new configuration file take effect, which seriously affects the normal operation of the training business.

[0082] In some embodiments of the present application, step 303 involves allocating storage resources to the network storage device via a storage management device. Because the storage management device is external to the container cluster and exchanges commands with the network storage device, the user accessing the storage management device and subsequently operating the network storage device will not affect the computing containers in the container cluster, thereby improving the system stability of the container cluster.

[0083] In some embodiments of the present application, a distributed file system can be used to implement resource allocation tasks for storage resources in the network storage device to various computing containers in the container cluster. The distributed file system can allocate a corresponding resource directory for each computing container, and does not set a corresponding resource quota for the resource directory corresponding to each computing container, so as to allocate corresponding storage resources to each computing capacity. The distributed file system can be POSIX.

[0084] Accordingly, the resource adjustment instruction can complete the resource adjustment operation of the target computing power container through the command of the distributed POSIX file system. For example, taking Ceph storage as an example, the resource quota (quota) adjustment is completed through the setattr command in Ceph. Through the solution provided by this embodiment, the storage resource adjustment task of the target training business can be completed by adjusting the resource quota of the resource directory corresponding to the target computing power container in the file system. This solution does not require remounting or remapping, thereby improving the operating efficiency of the training business.

[0085] Through the embodiments of this application Figure 3It can be seen from the above exemplary implementation that the embodiment of the present application realizes the task of allocating storage resources in the network storage device through an external storage management device. Compared with the solution of allocating storage resources using the master node device in the container cluster in the related art, the embodiment of the present application can not only avoid the problem of waste of computing resources caused by re-mounting or re-mapping in the process of adjusting the storage resources corresponding to the target computing power container, thereby improving the operating efficiency of the training business; it can also avoid the risk of misoperation on the user side when accessing the master node device, thereby improving the stability of the container cluster. At the same time, since the present application also responds to the resource adjustment request and verifies the resource adjustment request, it can ensure the security of the training data in the network storage device.

[0086] See also Figure 4 , Figure 4 This is an optional flow chart of the storage resource management method provided in the embodiment of the present application, based on Figure 3 , Figure 3 Step 302 shown can be updated to step 401 to step 403, combining Figure 4 The steps shown are explained.

[0087] In step 401, registration information of a target training service and resource allocation data of a network storage device are obtained; the registration information includes a minimum amount of resources for the target training service.

[0088] In some embodiments of the present application, during the registration process of the target training service, the registration information of the target training service is obtained and saved. The registration information may include the minimum resource amount for the target training service, which is used to represent the minimum amount of storage resources required for the target training service to be operational. That is, when the minimum resource amount is allocated to the target training service, the target training service can be completed within a limited time, and accordingly, the time required to complete the target training service is longer.

[0089] In some embodiments of the present application, the storage management device stores resource allocation data of the network storage device, which includes the current storage resource amount of each registered training service and the total resource amount of the network storage device.

[0090] For example, if the first to fifth training services have been registered, the current storage resource amount corresponding to each training service is 20GB, and the total resource amount of the network storage device is 150GB, then the resource allocation data of the network storage device can be shown in Table 1:

[0091] Table 1

[0092]

[0093] In step 402, the resource amount to be adjusted carried in the resource adjustment request is verified based on the minimum resource amount and the resource allocation data.

[0094] In some embodiments of the present application, the above-mentioned verification of the amount of resources to be adjusted carried in the resource adjustment request based on the minimum amount of resources and the resource allocation data can be achieved in the following manner: when the resource adjustment request is a capacity expansion request, the target total amount of resources is determined based on the amount of resources to be adjusted and the resource allocation data; when the target total amount of resources is greater than the total amount of resources of the network storage device, it is determined that the verification of the amount of resources to be adjusted has failed; when the target total amount of resources is less than or equal to the total amount of resources of the network storage device, it is determined that the verification of the amount of resources to be adjusted has passed.

[0095] For example, based on the example in Table 1, if the resource adjustment request is a capacity expansion request for the first training business, and the amount of resources to be adjusted is the amount of resources that need to be increased by 10GB for the target training business, the total target resource amount of the target training business can be obtained as 110GB based on the 20GB of the first training business in the resource allocation data, the amount of resources to be adjusted (+10GB), and the amount of resources of other training businesses. Since the total target resource amount is less than the total resource amount of the network storage device, it is determined that the verification of the amount of resources to be adjusted is passed.

[0096] In some embodiments of the present application, the above-mentioned verification of the amount of resources to be adjusted carried in the resource adjustment request based on the minimum amount of resources and resource allocation data can be achieved in the following manner: when the resource adjustment request is a scaling-down request, the target amount of resources for the target training service is determined; when the target amount of resources is greater than or equal to the minimum amount of resources, it is determined that the verification of the amount of resources to be adjusted is passed; when the total amount of target resources is less than the total amount of resources of the network storage device, it is determined that the verification of the amount of resources to be adjusted is failed.

[0097] For example, based on the example in Table 1, if the resource adjustment request is a capacity reduction request for the first training business, and the amount of resources to be adjusted is the amount of resources that needs to be reduced by 10GB for the target training business, the target resource amount of the first training business can be determined to be 10GB based on the 20GB of the first training business in the resource allocation data. If the minimum resource amount set for the first training business during the registration process is 15GB, since the target resource amount is less than the minimum resource amount, it is determined that the verification of the resource amount to be adjusted has failed.

[0098] Through the above solution, since the current resource amount corresponding to each training business is stored in the external storage management device, the rationality of the resource amount to be adjusted in the resource adjustment request can be verified without interacting with the container cluster or network storage device. If the verification fails, the verification failure message is directly returned to the user side, saving the information interaction steps between the network storage device.

[0099] In step 403, when the verification of the resource amount to be adjusted is passed, the authority information carried in the resource adjustment request is verified.

[0100] In some embodiments of the present application, step 4031 may be used to implement the above-mentioned verification of the permission information carried in the resource adjustment request.

[0101] In step 4031, the resource directory corresponding to the resource adjustment request is obtained, and whether the permission requirements of the resource directory are met is determined based on the permission information; if the permission information meets the permission requirements of the resource directory, it is determined that the permission information verification is passed.

[0102] Wherein, the permission information may be the user permission of the user side, and each resource directory in the multiple resource directories in the storage management device has a corresponding directory level. The storage management device sets corresponding user permissions for each directory level, and a resource adjustment request with corresponding user permissions can adjust the resource quota of the resource directory corresponding to the directory level. In step 4031, the resource management device can determine the user permission of the user side based on the permission information, and determine the resource directory of the target training business based on the business identifier of the target training business, and then determine the directory level of the resource directory, and determine whether the permission information meets the permission requirements of the resource directory based on the user permission corresponding to the directory level of the resource directory and the user permission of the user side. For example, when the user permission corresponding to the directory level of the resource directory includes the user permission of the user side, or when the user permission of the user side is higher than the user permission corresponding to the directory level of the resource directory, it is determined that the permission information meets the permission requirements of the resource directory.

[0103] Furthermore, when the permission information meets the permission requirement of the resource directory, it is determined that the permission information verification is successful.

[0104] In some embodiments of the present application, step 4032 may also be used to implement the above-mentioned verification of the permission information carried in the resource adjustment request.

[0105] In step 4032, the registration identifier in the permission information is obtained, and it is detected whether the registration identifier belongs to the network storage device; if the registration identifier belongs to the network storage device, it is determined that the permission information verification is successful.

[0106] In some embodiments of the present application, during the registration process of the target training service, the network storage device allocates storage resources for the target training service and also allocates a registration identifier of the network storage device for the target training service. The registration identifier is used to indicate that there are storage resources allocated for the target training service in the network storage device. The storage management device receives the registration identifier allocated by the network storage device for the target training service, and while saving the registration identifier, forwards the registration identifier to the user side. When the user side needs to adjust the amount of storage resources allocated for the target training service in the network storage device, it will send a resource adjustment request corresponding to the registration identifier to the storage management device.

[0107] In some embodiments of the present application, the storage management device can obtain the local registration identifier stored during the registration process of the target training service during the process of verifying the permission information, and by comparing the local registration identifier with the registration identifier in the permission information, if they are the same, it is determined that the registration identifier belongs to the network storage device; if they are not the same, it is determined that the registration identifier does not belong to the network storage device.

[0108] Furthermore, when it is determined that the registration identifier belongs to the network storage device, it is determined that the authority information verification is passed.

[0109] It should be noted that the above-mentioned various authorization information verification methods can be performed simultaneously, or any one of them can be selected to complete the authorization information verification, and this application does not limit this. In the case of selecting multiple verification methods at the same time, it is necessary to ensure that all verification methods pass before determining that the authorization information verification is passed.

[0110] Through the embodiments of this application Figure 4 From the above exemplary implementation, it can be seen that the embodiment of the present application stores the current amount of resources corresponding to each training business in the external storage management device. Therefore, the rationality of the amount of resources to be adjusted in the resource adjustment request can be verified without exchanging information with the container cluster or the network storage device. In addition, if the verification fails, a verification failure message is directly returned to the user side, thus saving the information interaction step with the network storage device. At the same time, since the present application detects the permission information of the resource adjustment request after judging the rationality of the resource adjustment request, it can avoid illegal intrusion of the model training platform by illegal users, thereby ensuring the security of the training data in the network storage device and the stability of the container cluster.

[0111] See also Figure 5 , Figure 5 This is an optional flow chart of the storage resource management method provided in the embodiment of the present application, based on Figure 3 ,exist Figure 3 After step 303 shown, steps 501 to 502 may also be included, combining Figure 5 The steps shown are explained.

[0112] In step 501, the container cluster and the network storage device are monitored to determine load information.

[0113] In some embodiments of the present application, the load information includes the hardware load corresponding to the container cluster, and the hardware load may include the hardware load of the container cluster as a whole and the hardware load of each computing power container, where the hardware load may include the load of hardware such as GPU, processor, memory and hard disk.

[0114] In some embodiments of the present application, the load information includes resource allocation data of the network storage device. The resource allocation data of the network storage device includes the current storage resource amount of each registered training service and the total resource amount of the network storage device.

[0115] In some embodiments of the present application, the load information also includes the traffic load between the container cluster and the network storage device, wherein the traffic load may include the traffic between each computing container in the container cluster and the storage resource corresponding to each computing container.

[0116] In step 502, an alarm message is generated based on the load information.

[0117] In some embodiments of the present application, the above-mentioned generation of an alarm message based on load information can be implemented through step 5021.

[0118] In step 5021, the container flow corresponding to each computing power container in the multiple computing power containers in the container cluster is obtained; the container flow is the amount of data transmitted between the computing power container and the network storage device; when the container flow is higher than the preset flow threshold, a flow alarm message is generated and the flow of the computing power container corresponding to the container flow is restricted.

[0119] Based on the example in Table 1, if the data transmission volume between the computing capacity of the first training service and the network device is P1, that is, the container flow is P1; the container flow of the second training service is P2; the container flow of the third to fifth training services is P3, and the flow threshold is P4, where P1>P2>P4>P3; then a flow alarm message will be generated, indicating that the first training service and the second training service meet the alarm conditions; at the same time, the flow of the computing capacity containers corresponding to the first training service and the second training service will be restricted. In some embodiments, the flow of the computing capacity container corresponding to the first training service can be restricted first, and then the flow of the computing capacity container corresponding to the second training service can be restricted.

[0120] In some embodiments of the present application, step 5022 may also be used to implement the above-mentioned generation of an alarm message based on load information.

[0121] In step 5022, the hardware load corresponding to each computing power container in the multiple computing power containers in the container cluster is obtained; the hardware load includes at least one of the following: processor load, memory load and disk input and output load; based on the hardware load corresponding to each computing power container, the hardware load quantization value of each computing power container is determined; when the hardware load quantization value is greater than or equal to the preset hardware load threshold, a hardware alarm message is sent to the message queue; the message queue is used to push the hardware alarm message to the operation and maintenance personnel.

[0122] Among them, the above-mentioned determination of the hardware load quantization value of each computing power container based on the hardware load corresponding to each computing power container can be achieved in the following ways: based on the hardware load corresponding to the computing power container, the load data of each hardware type in the computing power container is obtained; according to the business type of the training business running in the computing power container, the load weight of each hardware type is determined; according to the load data and load weight of each hardware type, the hardware load quantization value of the computing power container is determined.

[0123] In some embodiments, in the process of calculating the hardware load quantization value corresponding to any computing power container, the load data of the computing power container under different hardware types can be obtained, for example, the CPU utilization, GPU utilization, hard disk read (input and output) utilization and the amount of resources used can be obtained; the business type of the training business running in the computing power container is obtained, and the load weight of each hardware type is determined based on the business type, wherein different business types correspond to different load weights of each hardware type. For example, for the training business of the image recognition model, since a large amount of image sample data and a large number of intermediate feature images need to be processed during the image recognition training process, more space is required. GPU resources and hard disk reading resources are needed, and this is a normal phenomenon. Therefore, it is necessary to select GPU resources and hard disk reading resources with a smaller load weight for the training business, thereby obtaining a lower hardware load quantization value. For example, for the training business of a simple text classification model, since only text training data with a small amount of data needs to be obtained, and during the training process, only a relatively simple fully connected network structure needs to be set, and the weight parameters of the neurons in the fully connected network need to be adjusted, it is not necessary to occupy more GPU resources and hard disk reading resources. Furthermore, it is necessary to select GPU resources and hard disk reading resources with a larger load weight for the training business, thereby obtaining a higher hardware load quantization value.

[0124] It should be noted that the server pre-stores the load weights of each hardware type corresponding to each business type, and the load weight of any hardware type in the business type is inversely proportional to the normal load value of the hardware type used by the business type, that is, during the training task of the business type, the normal load value for a hardware type is high, and the load weight of the hardware type is relatively low. Through the above method, when there are load differences in the training services of different business types during the training process, the same hardware load threshold can be used to perform load detection on the training services of all business types, thereby improving the rationality of the alarm message.

[0125] In some embodiments of the present application, the message queue sets a corresponding subscription topic for the hardware alarm message. The storage management device that currently sets the message queue is equivalent to the publishing end of the subscription topic. After generating the hardware alarm message, the storage management device can generate a message to be published based on the subscription topic corresponding to the hardware alarm message. After the message receiving end that subscribes to the subscription topic goes online (establishes a connection with the storage management device), it can receive the target alarm message sent by the storage management device.

[0126] Through the embodiments of this application Figure 5 As can be seen from the above exemplary implementation, the present application embodiment can monitor the status of the entire model training platform by monitoring the load information of the container cluster and network storage devices, making it easier for users to understand the execution status of each training business. At the same time, triggering corresponding alarm messages based on the above load information can improve the timeliness of operation and maintenance personnel in handling platform abnormal information.

[0127] See also Figure 6 , Figure 6 This is an optional flow chart of the storage resource management method provided in the embodiment of the present application, based on the above embodiment, based on Figure 3 For example, before step 301, steps 601 to 603 may also be included, combining Figure 6 The steps shown are explained.

[0128] In step 601, a registration request for a training service to be registered is received from the user side; the registration request carries registration information of the training service to be registered; the registration information includes the minimum resource amount, the resource amount to be applied for, and the permission information to be applied for of the training service to be registered.

[0129] In some embodiments of the present application, the minimum amount of resources is used to characterize the minimum amount of storage resources required for the target training service to be able to run. That is to say, when the minimum amount of resources is allocated to the target training service, the target training service can be completed within a limited time, and accordingly, the time required to complete the target training service is longer. The amount of resources to be applied for is used to characterize the most appropriate amount of storage resources required for the target training service to be able to run; the amount of resources to be applied for can be obtained from a preset resource amount list based on the business type of the training service to be registered, or it can be set in real time by the user side based on the actual needs of the service to be registered. The permission information to be applied for can be the user authority applied for the user side in the above embodiment.

[0130] In step 602, in response to the registration request, a verification request is sent to the network storage device; the network storage device is used to allocate storage resources for the training service to be registered based on the registration information when the registration information is verified, and send the registration identifier of the training service to be registered to the storage management device.

[0131] In step 603, the registration identifier of the training service to be registered sent by the storage network storage device is stored and forwarded to the user side so that the user side carries the registration identifier of the training service to be registered in the resource adjustment request initiated for the training service to be registered.

[0132] In some embodiments of the present application, the storage resource management device saves the registration identifier of the training service to be registered locally, and at the same time forwards the registration identifier of the training service to be registered to the user side that sent the registration request. Afterwards, when the user side needs to adjust the storage resources of the training service to be registered, it is necessary to send a resource adjustment request carrying the registration identifier to the storage resource management device. The storage resource management device verifies the registration identifier carried in the resource adjustment request based on the registration identifier previously saved locally. Through the above method, the security of the training data corresponding to the training service to be registered can be guaranteed.

[0133] Through the embodiments of this application Figure 6It can be seen from the above exemplary implementation that the embodiment of the present application receives the registration request for the training service to be registered from the user side through the storage management device, and through the verification operation between the storage management device and the network storage device, the storage management device can obtain the configuration data of all storage resources of the training service from the beginning of registration, so that in the subsequent process of receiving the resource adjustment request, the rationality of the resource adjustment request can be verified based on the storage resource configuration data of the training service that has been stored, thereby improving the verification efficiency; eliminating the information interaction step between the storage management device and the network storage device, reducing the bandwidth demand of the network storage device, and indirectly improving the utilization rate of the traffic bandwidth between the network storage device and the container cluster, thereby improving the execution efficiency of the training service.

[0134] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0135] In the current training business scenario, GPU computing power needs to be allocated to the training business so that the training business can complete the corresponding training business. Among them, when the GPU computing power is allocated to the training business, the training data needs to use network storage. The embodiment of this application is aimed at the network storage management solution for training data in the training business. In the current traditional solution, the Kubernetes system architecture supports the use of plug-in mode, which is compatible with the storage solutions of the open source community or cloud vendors. The GPU computing power container delivered to the user performs read and write operations on the network storage through data volumes or mounted directories.

[0136] After research, the applicant found that the above-mentioned traditional solution has the following technical problems: (1) If storage expansion is required, the expansion operation needs to be implemented according to the kubectl command on the master (master node device) of Kubernetes. Even some network storage needs to be remounted or mapped after expansion to take effect, which provides a poor user experience; (2) There is a high risk of operating the Kubernetes cluster control node through commands; (3) The cluster control node on the Kubernetes side has weak stability and control over cluster permissions and storage clusters.

[0137] Based on the above issues, this application focuses on network storage management issues during the network storage process of user training data when GPU computing containers are delivered to users. During the training process, it is necessary to ensure data permission control and cluster stability. When scaling network storage, it is also necessary to reduce the risk of scaling operations to the system architecture and improve the user experience.

[0138] To scale the network storage of computing containers used by businesses, the traditional approach is to access the Kubernetes cluster control node and perform scaling operations on data volumes or storage directories using commands such as kubectl. After scaling is complete, the cluster control node needs to reload the configuration file. If other configuration items are mistakenly operated during the process of changing the scaling value, it is likely to cause unnecessary problems.

[0139] The embodiments of the present application add an external storage control center to centralize the expansion and contraction operations of network storage to the storage control center, enabling expansion or contraction of storage space without logging into the Kubernetes cluster control node (control center). Furthermore, the expansion and contraction operations in the external center use a quota-reducing directory approach, eliminating the need for remounting or remapping after expansion or contraction. By adding permission control and monitoring the load flow of the storage cluster in the external center, the shortcomings of training data security are addressed.

[0140] In some embodiments of the present application, through policies based on an external storage configuration center, the configuration center focuses on the management of network storage. Scaling operations can be initiated from the control plane of the center to verify user permissions. Moreover, the scaling operations are only for the directory (quota) of the network storage and will not change other configuration items in the system. The above-mentioned traditional technical solutions are optimized as follows:

[0141] (1) An independent network storage configuration center is used to handle the management operations of the network disk. Scaling operations do not require the operation of the Kubernetes cluster control node, which reduces the risk to the architecture and is beneficial to the operability of the system.

[0142] (2) The configuration center counts the usage rate of each storage network disk, which facilitates the monitoring of the storage network disk status and capacity warning, and is beneficial to improving user experience.

[0143] (3) After the GPU computing power container is destroyed, users can query the space usage, directory, file number and other information of the training data through the external configuration center, which makes it easier for users to evaluate the data loading during the next training.

[0144] (4) If the user has not submitted a training task but needs to read the network disk data for analysis, the external configuration center can easily add permission control and other policies to ensure the security of user data.

[0145] (5) When users load network disk data, the external configuration center can monitor the load of the network storage cluster and dynamically adjust the data reading and writing thresholds to ensure the stability of the cluster.

[0146] In some embodiments of this application, please refer to Figure 7 , Figure 7A system architecture diagram of an external center is shown, wherein the system architecture diagram shows the overall architecture of the external center 710. The external center 710 manages and controls the network disk during the training process (also called the training network disk, corresponding to the network storage device in the above embodiment), thereby realizing the expansion and contraction operations of the network disk, storage permissions and cluster stability. The external center is equivalent to the storage management device in the above embodiment.

[0147] exist Figure 7 As can be seen, the external center 710 includes a configuration center 711, an authentication center 712, and a monitoring center 713.

[0148] Configuration center 711 is used to configure the size of the user's training network disk. When the user needs to expand or shrink the size of the network disk, a scaling request will be sent from the configuration center. The configuration center will first determine the capacity in the request. If the expanded capacity exceeds the cluster capacity value, a failure message will be returned. If the shrinkage value is lower than the threshold required for business expansion, a failure message will be returned to the user. If the request is judged to be reasonable, it will be packaged and sent to the authentication center.

[0149] In some embodiments, for a training business, the training business corresponds to a network disk and a container. The network disk is used to store training data, and the container is used to obtain training data from the network disk and execute the training business. When a training business is registered, it is necessary to set the minimum capacity value of the training business and the capacity value that needs to be applied for. The Kubernetes cluster 715 will allocate the corresponding network disk quota to the training business based on the capacity value that needs to be applied for, and run the training business. During operation, in response to the user's expansion and contraction request, when the sum of the expansion capacity and the current network disk capacity carried in the expansion request exceeds the maximum network disk capacity of the network disk storage 714, a failure message is returned; when the contraction capacity carried in the contraction request is less than the minimum capacity value set previously, a failure message is also returned. The authentication center 712 is used to implement the permission verification operation, wherein the verification operation may include at least one of the following: (1) verifying whether the user has write permission to the network disk directory being operated, and returning failure if there is no write permission; (2) verifying whether the key information carried in the request (corresponding to the registration identifier in the above embodiment) is issued by the storage cluster. The key issuance process occurs when the user registers the network disk directory. If the registration is successful, the network disk cluster will issue the key information back to the user, and the user operation directory needs to carry the key.

[0150] The monitoring center 713 is used to monitor the traffic and resource load of the network disk storage 714 and generate corresponding alarm messages based on real-time conditions. If the monitored traffic exceeds a threshold, an alarm is triggered and an automatic throttling strategy is implemented, prioritizing bandwidth restrictions on high-traffic network sessions to quickly reduce the cluster's traffic load. If the monitored load on other resources is high, an alarm operation is executed, requiring the operation and maintenance personnel who receive the alarm to manually analyze and resolve the cause of the high load. The other resources include, but are not limited to, at least one of the following: CPU, memory, and disk I / O.

[0151] In a real-world scenario, for example, a new training service requires a storage registration request from the configuration center. This request includes the service name, required storage resources, and requested permissions. The configuration center sends this information to the storage cluster. After the storage cluster determines the legitimacy of the service access, it issues a key and returns it to the authentication center and the service. When the service requires network disk storage expansion during training, it initiates an expansion request with the key. This request is authenticated by the authentication center and sent to the storage cluster. After the storage cluster finds the service's expansion directory, it executes the expansion operation. The expansion process runs in parallel with the monitoring and alarm processes; triggering an alarm does not interrupt the expansion process.

[0152] The following will be combined Figure 8 and Figure 9 Explain the scaling process and monitoring alarm processing process respectively.

[0153] See also Figure 8 , Figure 8 This is a flow chart of an optional scaling process provided by the embodiment of the present application. Figure 8 The steps shown are explained.

[0154] In step 801, the configuration center receives a scaling request for a target service.

[0155] In step 802, the configuration center determines whether the scaling request is reasonable based on the capacity threshold.

[0156] If it is determined that the scaling request is reasonable, step 803 is executed; if it is determined that the scaling request is unreasonable, step 806 is executed.

[0157] In some embodiments, during the registration of the target business, it is necessary to set a minimum capacity value (corresponding to the minimum amount of resources in the above embodiment) and a capacity value that needs to be applied for (corresponding to the amount of resources to be applied for in the above embodiment). The storage cluster will allocate the corresponding network disk quota to the business based on the capacity value that needs to be applied for, and run the target business. During the operation of the target business, the above-mentioned scaling request is received. In the case that the scaling request is a capacity expansion request, if the sum of the capacity expansion carried in the capacity expansion request and the current real-time capacity of the storage cluster exceeds the maximum capacity of the storage cluster, the scaling request is judged to be unreasonable; in the case that the scaling request is a capacity reduction request, if the capacity reduction carried in the capacity reduction request is less than the minimum capacity value set previously, the scaling request is judged to be unreasonable.

[0158] In step 803, the configuration center initiates an authentication request to the authentication center.

[0159] In step 804, the authentication center determines whether the authentication is successful. If the authentication is successful, step 805 is executed; if the authentication is not successful, step 806 is executed.

[0160] In some embodiments, the authentication center will verify the key information carried in the request (corresponding to the identification information in the above embodiment) and the user's permission information. If the verification fails, the process ends. After the verification succeeds, the request will be sent to the storage cluster to complete the expansion and contraction operation of the user directory.

[0161] In step 805, the scaling operation corresponding to the scaling request is performed.

[0162] In step 806, end.

[0163] In some embodiments, the authentication center or the configuration center may also send a corresponding request failure message to the user side that sent the scaling request.

[0164] See also Figure 9 , Figure 9 This is a flow chart of an optional monitoring alarm processing process provided by the embodiment of the present application. Figure 9 The steps shown are explained.

[0165] In step 901 , the monitoring center monitors the traffic load and other loads of the storage cluster.

[0166] In some embodiments, the monitoring center periodically polls the traffic load and other loads of the storage cluster, where the other loads refer to the loads of other resources, including other resource loads within the container, such as CPU, memory, and disk IO.

[0167] In step 902, the monitoring center determines whether the alarm threshold is exceeded.

[0168] If the value exceeds the alarm threshold, step 903 is executed; if the value does not exceed the alarm threshold, step 901 is continued to be executed.

[0169] In some embodiments, a corresponding traffic threshold can be set for the traffic load, and a corresponding hardware load can be set for other loads. When at least one of the following events occurs, it is determined that the alarm threshold is exceeded: (1) the traffic load exceeds the preset traffic threshold; (2) the other load exceeds the preset hardware threshold.

[0170] In step 903, the monitoring center triggers an alarm operation.

[0171] In step 904, the monitoring center determines whether it is a flow threshold alarm.

[0172] If the alarm operation is a flow threshold alarm operation, step 905 is executed; if the alarm operation is not a flow threshold alarm operation, step 906 is executed.

[0173] In step 905, the automatic current limiting strategy is triggered.

[0174] In some embodiments, the automatic traffic limiting strategy may be to prioritize limiting the traffic bandwidth of network sessions with higher traffic, so as to quickly reduce the above-mentioned traffic load.

[0175] In step 906 , a manual intervention strategy is triggered.

[0176] In some embodiments, the manual intervention strategy can be to send a hardware alarm message to a message queue; the message queue is used to push the hardware alarm message to the operation and maintenance personnel, so that the operation and maintenance personnel can complete manual intervention, analyze the cause of the high load and solve the above problem.

[0177] Through the above-mentioned embodiments of the present application, the technical problem of how to realize network storage of business through an external central control can be solved, and the expansion and contraction operations of the user network disk directory can be completed without operating the main node device of Kubernetes, and the security and stability of the operation can be guaranteed during the operation; the security and stability of network storage in the current Kubernetes architecture are weak, and by adding an authentication center and a monitoring center, the shortcomings of security and stability operations are made up; the expansion and contraction operations controlled by the configuration center can be achieved by requesting to modify the directory quota of the network disk storage without accessing the main node device of Kubernetes, which reduces system risks while facilitating user operations and improving the experience.

[0178] The following continues to describe the exemplary structure of the storage resource management device 555 provided in the embodiment of the present application as a software module. In some embodiments of the present application, such as Figure 2 As shown, the software modules stored in the storage resource management device 555 of the memory 550 may include:

[0179] Receiving module 5551, configured to receive a resource adjustment request for a target training service; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training service, and the storage resources are used to store training data used by the target computing power container to execute the target training service;

[0180] Verification module 5552, configured to respond to a resource adjustment request and verify the amount of resources to be adjusted and permission information carried in the resource adjustment request;

[0181] The sending module 5553 is used to send a resource adjustment instruction to the network storage device when the resource amount to be adjusted and the permission information are verified. The resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

[0182] In some embodiments of the present application, the verification module 5552 is also used to: obtain the registration information of the target training service and the resource allocation data of the network storage device; the registration information includes the minimum resource amount of the target training service; based on the minimum resource amount and the resource allocation data, verify the resource amount to be adjusted carried in the resource adjustment request; if the verification of the resource amount to be adjusted is passed, verify the permission information carried in the resource adjustment request.

[0183] In some embodiments of the present application, the verification module 5552 is also used to perform at least one of the following: obtaining a resource directory corresponding to the resource adjustment request, determining whether the permission requirements of the resource directory are met based on the permission information, and if the permission information meets the permission requirements of the resource directory, determining that the permission information verification is successful; obtaining a registration identifier in the permission information, detecting whether the registration identifier belongs to a network storage device, and if the registration identifier belongs to a network storage device, determining that the permission information verification is successful.

[0184] In some embodiments of the present application, the storage resource management device 555 further includes a monitoring module, wherein;

[0185] The monitoring module is used to monitor the container cluster and network storage devices, determine the load information, and generate alarm messages based on the load information.

[0186] In some embodiments of the present application, the load information includes traffic information; the monitoring module is also used to: obtain the container traffic corresponding to each computing power container in multiple computing power containers in the container cluster; the container traffic is the amount of data transmitted between the computing power container and the network storage device; when the container traffic is higher than the preset traffic threshold, a traffic alarm message is generated and the traffic of the computing power container corresponding to the container traffic is restricted.

[0187] In some embodiments of the present application, the load information includes hardware information; the monitoring module is also used to: obtain the hardware load corresponding to each computing power container in the multiple computing power containers in the container cluster; the hardware load includes at least one of the following: processor load, memory load and disk input and output load; based on the hardware load corresponding to each computing power container, determine the hardware load quantification value of each computing power container; when the hardware load quantification value is greater than or equal to the preset hardware load threshold, send a hardware alarm message to the message queue; the message queue is used to push the hardware alarm message to the operation and maintenance personnel.

[0188] In some embodiments of the present application, the monitoring module is also used to: obtain load data of each hardware type in the computing power container based on the hardware load corresponding to the computing power container; determine the load weight of each hardware type according to the business type of the training business running in the computing power container; determine the hardware load quantification value of the computing power container based on the load data and load weight of each hardware type.

[0189] In some embodiments of the present application, the receiving module 5551 is also used to: receive a registration request from the user side for the training service to be registered; the registration request carries the registration information of the training service to be registered; the registration information includes the minimum amount of resources for the training service to be registered, the amount of resources to be applied for, and the permission information to be applied for; in response to the registration request, send a verification request to the network storage device; the network storage device is used to allocate storage resources for the training service to be registered based on the registration information when the registration information is verified, and send the registration identifier of the training service to be registered to the storage management device; store the registration identifier of the training service to be registered sent by the network storage device, and forward the registration identifier of the training service to be registered to the user side, so that the user side carries the registration identifier of the training service to be registered when initiating a resource adjustment request for the training service to be registered.

[0190] The present disclosure provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the storage resource management method described above in the present disclosure.

[0191] The embodiment of the present disclosure provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the storage resource management method provided by the embodiment of the present application, for example, Figures 3 to 6 The method shown.

[0192] In some embodiments of the present application, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an electronic programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM; or it may be various devices including one or any combination of the above memories.

[0193] In some embodiments of the present application, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0194] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0195] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0196] In summary, the following technical effects can be achieved through the embodiments of the present application:

[0197] (1) The embodiment of the present application uses an external storage management device to implement the task of allocating storage resources in the network storage device. Compared with the solution of allocating storage resources using the master node device in the container cluster in the related art, the embodiment of the present application can not only avoid the problem of waste of computing resources caused by re-mounting or re-mapping in the process of adjusting the storage resources corresponding to the target computing power container, thereby improving the operating efficiency of the training business; it can also avoid the risk of misoperation when the user side accesses the master node device, thereby improving the stability of the container cluster. At the same time, since the present application also responds to the resource adjustment request and verifies the resource adjustment request, it can ensure the security of the training data in the network storage device.

[0198] (2) Since the embodiment of the present application stores the current resource amount corresponding to each training business in the external storage management device, it can still verify the rationality of the resource amount to be adjusted in the resource adjustment request without exchanging information with the container cluster or network storage device. If the verification fails, a verification failure message is directly returned to the user side, saving the information interaction step between the network storage device. At the same time, since the present application also detects the permission information of the resource adjustment request after judging the rationality of the resource adjustment request, it can avoid illegal intrusion of the model training platform by illegal users, thereby ensuring the security of the training data in the network storage device and the stability of the container cluster.

[0199] (3) By monitoring the load information of the container cluster and network storage devices, the embodiment of the present application can monitor the status of the entire model training platform, making it easier for users to understand the execution status of each training business. At the same time, triggering corresponding alarm messages based on the above load information can improve the timeliness of operation and maintenance personnel in handling platform abnormal information.

[0200] (4) In the embodiment of the present application, a storage management device receives a registration request from a user for a training service to be registered, and through a verification operation between the storage management device and the network storage device, the storage management device can obtain configuration data of all storage resources of the training service from the start of registration, so that when a resource adjustment request is subsequently received, the rationality of the resource adjustment request can be verified based on the storage resource configuration data of the training service that has been stored, thereby improving verification efficiency; eliminating the information interaction step between the storage management device and the network storage device, reducing the bandwidth demand of the network storage device, and indirectly improving the utilization rate of the traffic bandwidth between the network storage device and the container cluster, thereby improving the execution efficiency of the training service.

[0201] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A storage resource management method, characterized in that: Applied to a storage management device, the storage management device is used to manage storage resources allocated by a network storage device to a container cluster, the method comprising: Receive a resource adjustment request for a target training service; the resource adjustment request is used to adjust the storage resources allocated by the network storage device to a target computing power container, the target computing power container is used to execute the target training service, and the storage resources are used to store training data used by the target computing power container to execute the target training service; In response to the resource adjustment request, obtaining registration information of the target training service and resource allocation data of the network storage device; the registration information includes a minimum resource amount of the target training service, the minimum resource amount representing a minimum amount of storage resources required when the target training service is running, and the resource allocation data is stored in the storage management device; If the amount of resources to be adjusted carried in the resource adjustment request is verified to be successful based on the minimum amount of resources and the resource allocation data, verify the permission information carried in the resource adjustment request; wherein the verification of the amount of resources to be adjusted is successful including: when the resource adjustment request is a scale-down request, the target amount of resources requested by the target training service is greater than or equal to the minimum amount of resources; When the amount of resources to be adjusted and the permission information are verified, a resource adjustment instruction is sent to the network storage device; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

2. The method according to claim 1, characterized in that The verifying the authority information carried in the resource adjustment request includes at least one of the following: Obtaining a resource directory corresponding to the resource adjustment request, determining whether permission requirements of the resource directory are met based on the permission information, and determining that verification of the permission information is successful if the permission information meets the permission requirements of the resource directory; The registration identifier in the permission information is obtained, and it is detected whether the registration identifier belongs to the network storage device. If the permission information meets the permission requirement of the resource directory, it is determined that the permission information is verified successfully.

3. The method according to any one of claims 1 to 2, characterized in that The method further comprises: Monitoring the container cluster and the network storage device to determine load information; An alarm message is generated based on the load information.

4. The method according to claim 3, characterized in that The load information includes flow information; and generating an alarm message based on the load information includes: Obtaining the container flow corresponding to each of the plurality of computing power containers in the container cluster; the container flow is the amount of data transmitted between the computing power container and the network storage device; When the container traffic exceeds the preset traffic threshold, a traffic alarm message is generated and the traffic of the computing power container corresponding to the container traffic is restricted.

5. The method according to claim 3, characterized in that The load information includes hardware information; and generating an alarm message based on the load information includes: Obtaining a hardware load corresponding to each of the plurality of computing containers in the container cluster; the hardware load includes at least one of the following: processor load, memory load, and disk input and output load; Determine a quantized hardware load value for each computing power container based on the hardware load corresponding to each computing power container; When the hardware load quantization value is greater than or equal to a preset hardware load threshold, a hardware alarm message is sent to a message queue; the message queue is used to push the hardware alarm message to operation and maintenance personnel.

6. The method according to any one of claims 1 to 2, characterized in that The method further comprises: Receive a registration request from a user side for a training service to be registered; the registration request carries registration information of the training service to be registered; the registration information includes a minimum resource amount, a resource amount to be applied for, and permission information to be applied for the training service to be registered; In response to the registration request, a verification request is sent to the network storage device; the network storage device is used to allocate storage resources to the training service to be registered based on the registration information when the registration information is verified, and send the registration identifier of the training service to be registered to the storage management device; The registration identifier of the training service to be registered sent by the network storage device is stored, and the registration identifier of the training service to be registered is forwarded to the user side, so that the user side carries the registration identifier of the training service to be registered in the resource adjustment request initiated for the training service to be registered.

7. A storage resource management device, characterized in that: include: A receiving module, configured to receive a resource adjustment request for a target training service; The resource adjustment request is used to adjust the storage resources allocated by the network storage device to the target computing power container, the target computing power container is used to execute the target training service, and the storage resources are used to store the training data used by the target computing power container to execute the target training service; a verification module, configured to obtain registration information of the target training service and resource allocation data of the network storage device in response to the resource adjustment request; The registration information includes the minimum resource amount of the target training business, the minimum resource amount represents the minimum storage resource amount required by the target training business when running, the resource allocation data is stored in a storage management device, and the storage management device is used to manage the storage resources allocated by the network storage device to the container cluster; if the resource amount to be adjusted carried in the resource adjustment request is verified based on the minimum resource amount and the resource allocation data, verify the permission information carried in the resource adjustment request; wherein, the case where the resource amount to be adjusted is verified includes: when the resource adjustment request is a shrinking request, the target resource amount requested by the target training business is greater than or equal to the minimum resource amount; A sending module is used to send a resource adjustment instruction to the network storage device when the resource amount to be adjusted and the permission information are verified; the resource adjustment instruction is used to adjust the resource quota of the network storage device for the target computing power container.

8. The device according to claim 7, characterized in that The verification module is further configured to perform at least one of the following: Obtaining a resource directory corresponding to the resource adjustment request, determining whether permission requirements of the resource directory are met based on the permission information, and determining that verification of the permission information is successful if the permission information meets the permission requirements of the resource directory; The registration identifier in the permission information is obtained, and it is detected whether the registration identifier belongs to the network storage device. If the permission information meets the permission requirement of the resource directory, it is determined that the permission information is verified successfully.

9. The device according to any one of claims 7 to 8, characterized in that The device further comprises: A monitoring module is configured to monitor the container cluster and the network storage device, determine load information, and generate an alarm message based on the load information.

10. The device according to claim 9, characterized in that The load information includes flow information; the monitoring module is further configured to: Obtaining the container flow corresponding to each of the plurality of computing power containers in the container cluster; the container flow is the amount of data transmitted between the computing power container and the network storage device; When the container traffic exceeds the preset traffic threshold, a traffic alarm message is generated and the traffic of the computing power container corresponding to the container traffic is restricted.

11. The device according to claim 9, characterized in that The load information includes hardware information; the monitoring module is further configured to: Obtaining a hardware load corresponding to each of the plurality of computing containers in the container cluster; the hardware load includes at least one of the following: processor load, memory load, and disk input and output load; Determine a quantized hardware load value for each computing power container based on the hardware load corresponding to each computing power container; When the hardware load quantization value is greater than or equal to a preset hardware load threshold, a hardware alarm message is sent to a message queue; the message queue is used to push the hardware alarm message to operation and maintenance personnel.

12. A storage resource management device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 6 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 6.

14. A computer program product, characterized in that The computer program product comprises computer instructions for implementing the method according to any one of claims 1 to 6 when executed by a processor.

Images