Visual intelligent storage management method and system for domestic GPU management platform

Abstract

The invention provides a visual intelligent storage management method and system for a domestic GPU management platform. The method comprises the following steps: creating and initializing file storage resources; the intelligent data storage interface is integrated with storage resources; the domestic GPU management platform manages and monitors storage resources; and GPU training task data sharing and access. According to the system, the method is realized when the processor executes the computer program stored in the memory. According to the method, application and use of file storage resources by GPU management platform training and reasoning tasks can be realized through a visual interface, intelligent management and application of the storage resources are realized through the GPU management platform, and visual data sharing management and storage integration are realized for a domestic GPU management platform; life cycle management of storage resources is realized according to dynamic resource allocation and static resource allocation, and binding of the storage resources and storage requirements of tenants is realized through a GPU management platform, so that complexity of storage configuration management by training and reasoning application is reduced; and the problems of data sharing and data governance in the training task are solved based on a domestic GPU environment.

Description

Technical Field

[0001] The present invention belongs to the field of data storage of a domestic GPU management platform, and more specifically, relates to a visualized intelligent storage management method and system for a domestic GPU management platform. Background Art

[0002] With the rapid development of AI technology, the management of multiple machines and multiple GPU cards has become a challenge, especially in large models and fine-tuning tasks, which require multi-card collaborative training. GPU collaborative management technology is becoming increasingly important in high-performance computing and deep learning. At present, Kubernetes cluster technology has become the mainstream solution for AI training, supporting flexible scheduling of multiple GPU cards and providing a variety of workload strategies to maximize resource utilization and provide powerful computing power for model training. However, in the process of artificial intelligence training, it is often necessary to use a large number of data sets and generate multiple model files, and batch training relies on shared storage. Therefore, it is particularly important to provide an intelligent storage solution for Kubernetes clusters. Existing storage solutions can no longer meet the needs of intelligent storage, and efficient and intelligent storage technologies are urgently needed to support the stable operation and efficient management of GPU management platforms.

[0003] There are three main ways for existing GPU management platforms to use storage resources: Kubernetes cluster container internal storage, container host disk storage, and container remote volume storage. Container internal storage is similar to data in memory, which is temporary and will be cleared when the container stops; container host disk storage is a mapping of the local disk, but its disadvantage is that it cannot achieve the drift of stored data between different container nodes. Existing GPU management platforms usually use container remote volume storage to achieve configuration and integration with storage systems. Although this method allows containers to save data sets or generated model files required for training through remote volumes, there are still some shortcomings. On the one hand, existing GPU management platforms usually lack fine-grained control in the allocation and management of storage resources, resulting in low utilization of storage resources; on the other hand, in terms of visual management, existing GPU management platforms are insufficient, and it is difficult for users to intuitively monitor and manage GPU training tasks and their shared storage resources. These problems limit the demand for intelligent storage in GPU management platforms in complex application scenarios. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing domestic GPU management platform has deficiencies in intelligent storage management and visualization.

[0005] In order to achieve the above objectives, the present invention provides a visual intelligent storage management method and system for a domestic GPU management platform.

[0006] According to a first aspect of the present invention, a visual intelligent storage management method for a domestic GPU management platform is provided, the method comprising the following steps:

[0007] File storage resource creation and initialization;

[0008] Intelligent data storage interface and storage resource integration;

[0009] Domestic GPU management platform storage resource management and monitoring;

[0010] GPU training task data sharing and access.

[0011] Optionally, the file storage resource creation and initialization further includes the following steps:

[0012] Distributed storage pool creation and configuration;

[0013] File directory creation and management;

[0014] Creation and configuration of shared storage services.

[0015] Optionally, the distributed storage pool creation and configuration further includes the following steps:

[0016] Before creating a distributed storage pool, set the hard disk to JBOD mode based on the domestic Kylin system, and add the solid-state disk and mechanical disk to different hard disk groups;

[0017] Create a distributed storage pool, select the storage pool type as file storage or block storage, select the data protection mechanism, select the pre-built disk group, fill in the storage pool name, and call the Python script to the corresponding storage node through JAVA HTTP request to execute the command to create the storage pool.

[0018] Optionally, the file directory creation and management are specifically as follows:

[0019] Based on the operation of multiple domestic storage servers, multiple copies or erasure code data protection mechanisms are selectively set up, and multiple logical volumes are built based on the storage pool;

[0020] The logical volumes include distributed volumes, stripe volumes, copy volumes, distributed stripe volumes and distributed copy volumes.

[0021] Optionally, the integration of the intelligent data storage interface and the storage resource further comprises the following steps:

[0022] Container cluster deployment and configuration: Deploy container clusters based on the domestic Kylin operating system, interconnect management nodes and computing nodes through the Calico network, and deploy basic images;

[0023] Deploy the intelligent storage VGCSI plug-in: install the VGCSI controller component and VGCSI node component in the computing cluster;

[0024] Operation and coordination of the VGCSI plug-in: Run the VGCSI controller component in the computing cluster as a Deployment or StatefulSet to create, delete, and expand storage resources, and deploy the VGCSI node component as a DaemonSet and run it on each computing node to mount and unmount storage volumes.

[0025] Optionally, the domestic GPU management platform storage resource management and monitoring further includes the following steps:

[0026] Visual configuration of storage management: storage resource creation, resource binding, resource usage, resource release and resource recycling are realized through a visual interface;

[0027] Shared use of storage management: multiple containers can mount a storage together to achieve storage sharing;

[0028] Monitoring and alerting of storage resources: View the usage rate, remaining amount, associated containers, storage status, and storage type of container storage resources through a visual interface.

[0029] Optionally, the GPU training task data sharing and access further includes the following steps:

[0030] Create a domestic GPU basic image: Use the Kylin basic container image as a template to create a training container template containing the driver of the domestic GPU card, and use docker commit to create a basic image and import it into the image template;

[0031] Create a training task through the GPU management interface: Create a GPU training task through the GPU visual management interface, select the pre-created domestic GPU image template, set the GPU resources and mounted storage volume to create a training task;

[0032] Tenant training result data sharing and access: The data generated by domestic GPU training under the same tenant is saved in the mounted storage volume, and mounted for use by other training tasks through visualization operations.

[0033] According to a second aspect of the present invention, a visual intelligent storage management system for a domestic GPU management platform is provided, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above-mentioned visual intelligent storage management methods for a domestic GPU management platform when executing the computer program.

[0034] The beneficial effects of the present invention are:

[0035] The visualized intelligent storage management method for the domestic GPU management platform of the present invention realizes the intelligent management of storage for training tasks through the GPU management platform. At the same time, it also provides an intelligent storage plug-in based on the domestic Feiteng platform to realize the visualized integration and configuration management of the GPU management platform and distributed storage. It realizes the application of distributed storage resources through the training tasks of the GPU platform and manages and uses the storage resources according to dynamic resource allocation and static resource allocation. Then, it realizes the shared management and refined allocation of data through the GPU management platform to ensure the security and efficiency of data, and proposes a new innovative solution for the use of domestic GPU management platforms and storage resources.

[0036] The storage system and GPU management platform involved in the present invention are both based on the domestic Feiteng server platform, providing a demonstration reference for the connection between other domestic GPU management platforms and storage systems; the present invention proposes a binding method for the GPU management platform to use distributed storage to perform dynamic storage resource allocation and static storage resource allocation, and matches the corresponding storage resources according to the minimum capacity according to the requirements of the container training task; the present invention performs refined management and operation of storage resources through resource monitoring and resource management of the GPU management platform, which facilitates operation and maintenance administrators to timely grasp the utilization of storage resources in GPU training tasks.

[0037] The visual intelligent storage management system for domestic GPU management platform of the present invention and the visual intelligent storage management method for domestic GPU management platform mentioned above belong to a general inventive concept, and have at least the same beneficial effects as the visual intelligent storage management method for domestic GPU management platform mentioned above, and its beneficial effects will not be repeated here.

[0038] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to represent the same or similar components.

[0040] Figure 1 A flowchart of implementing a visualized intelligent storage management method for a domestic GPU management platform according to an embodiment of the present invention is shown;

[0041] Figure 2 A flowchart of creating and initializing file storage resources according to an embodiment of the present invention is shown;

[0042] Figure 3A flowchart for implementing the integration of an intelligent data storage interface and storage resources according to an embodiment of the present invention is shown;

[0043] Figure 4 A flowchart for implementing storage resource management and monitoring of a domestic GPU management platform according to an embodiment of the present invention is shown;

[0044] Figure 5 A flow chart of GPU training task data sharing and access according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to more fully understand the technical solution of the present invention, the exemplary embodiments of the present invention will be described in more comprehensive and detailed in conjunction with the accompanying drawings below. Obviously, the one or more embodiments of the present invention described below are only one or more of the specific ways in which the technical solution of the present invention can be implemented, and are not exhaustive. It should be understood that other ways belonging to a general inventive concept can be used to implement the technical solution of the present invention, and should not be limited by the exemplary embodiments described. Based on one or more embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0046] Example: Figure 1 The following is a flowchart showing the implementation of a visual intelligent storage management method for a domestic GPU management platform according to an embodiment of the present invention. Figure 1 The visual intelligent storage management method for the domestic GPU management platform of the embodiment of the present invention includes the following steps:

[0047] Step S100, creating and initializing file storage resources;

[0048] Step S200, integrating the intelligent data storage interface with storage resources;

[0049] Step S300, domestic GPU management platform storage resource management and monitoring;

[0050] Step S400: GPU training task data sharing and access.

[0051] Specifically, in an embodiment of the present invention, steps S100 to S400 can effectively manage and allocate shared storage issues in GPU training. This not only reduces the complexity of the GPU management platform in terms of storage capacity planning, but also improves the utilization rate of distributed storage resources. In addition, in the training tasks of the GPU management platform, if access hotspots or failures are encountered, they can be quickly switched and restored to ensure the continuity and stability of training and reasoning tasks. This efficient management mechanism will further optimize resource allocation, improve the performance and reliability of the overall system, and thus better meet the intelligent storage needs in complex application scenarios.

[0052] Further, Figure 2 The following is a flowchart showing the implementation of file storage resource creation and initialization according to an embodiment of the present invention. Figure 2 In the embodiment of the present invention, the creation and initialization of the file storage resource described in step S100 further includes the following steps:

[0053] S101, distributed storage pool creation and configuration;

[0054] S102, file directory creation and management;

[0055] S103: Creation and configuration of shared storage services.

[0056] Specifically, in the embodiment of the present invention, file storage resource creation and initialization are implemented based on the Springboot architecture of the J2EE platform, providing a Restful interface to the outside world, and shielding the differences of various block storage basic systems through the platform layer internally. The view layer View adopts popular Vue, Jquery, and Bootstrap technologies. The control layer Controller calls Shell scripts and Python interfaces through the Java language to push commands to the server for execution and return corresponding results. Persistent data control is achieved through the domestic DAMO database, supporting the separation of static resources and dynamic resources of microservices, supporting the separate deployment of applications and databases, and having distributed operation and maintenance and elastic expansion characteristics.

[0057] Specifically, in the embodiment of the present invention, the specific process of creating and initializing the file storage resource described in step S100 is:

[0058] Add all storage servers to a storage cluster to form a storage cluster. Create a new storage pool based on the storage cluster, enter the storage pool name, select the multi-copy or erasure code data protection strategy, disk group, number of placement groups and other page information, and then send the form request to the Springboot control layer Action via http. The controller then calls the Python script to create the storage pool and returns the result to the page.

[0059] In the creation and management of file directories, to create a file target, you need to fill in the directory name, directory capacity, storage pool and other information based on the view form. After filling in, submit it to the agent layer through the control layer for the operation of adding a new directory. In directory management, you can set the number of files and capacity quota of the directory, and you can also modify and delete directory operations.

[0060] The creation and configuration of shared storage services mainly involves setting the file directory to NFS, SMB, FTP and other shared services, and using the newly created file directory as a shared directory for the GPU management platform. To add a shared storage service, you need to fill in the shared service name, select the shared service, set the shared path, and then use the file storage gateway to provide the newly created file directory or shared directory to the GPU platform.

[0061] Before adding a storage pool, add the hard disks of all storage servers to the disk group according to the needs of the visual storage management system. The capacity of the storage pool cannot be greater than the total capacity of the disk group. The process of creating and initializing file storage resources includes View visualization layer components, logical processing layer components, and proxy layer components:

[0062] a)View visualization layer components

[0063] The visual component generation mechanism uses html, jquery and bootstrap components, and uses the modal-dialog pop-up box to implement the settings of the visual storage pool form, file directory form and shared service form:

[0064] <form method＝"post"id＝"storpoolForm"onsubmit＝"return false；">

[0065] <div class＝"modal-body">

[0066] <div class＝"form-group"ms-if＝"isadd">

[0067] <label class＝"control-label label"> <em class＝"em_position">*< / em> Storage Pool Name:< / label>

[0068]

[0069] <input type="text"class="col-xs-11form-control input-sm width_97radius_0"name="depotName"id="depotName"

[0070] ms-duplex="depot.name"placeholder="Please enter the storage pool name"maxlength="20">

[0071]

[0072]

[0073] <div class＝"form-group"ms-if＝"isadd">

[0074] <label class＝"control-label label"> <em class＝"em_position">*< / em> Disk Group:< / label>

[0075]

[0076] <select name＝"diskgroupselect"id＝"diskgroupselect"class＝"col-xs-11form-control input-sm width_97radius_0">

[0077] <option ms-repeat＝"domainlist"ms-attr-value＝"el.name"> {{el.name}}< / option>

[0078] < / select>

[0079]

[0080]

[0081] <div class＝"form-group">

[0082] <label class＝"control-label label"> <em class＝"em_position">*< / em> Number of placement groups:< / label>

[0083]

[0084] <input type="text"class="col-xs-11form-control input-sm width_97radius_0"name="pgNum"id="pgNum"ms-duplex="depot.pgNum"

[0085] placeholder="Please enter the number of placement groups"maxlength="8"autocomplete="off">

[0086]

[0087]

[0088] <div class＝"form-group"ms-if＝"isadd">

[0089] <label class＝"control-label label"> <em class＝"em_position">*< / em> Strategy Type:< / label>

[0090]

[0091] <select class＝"col-xs-11form-control input-sm width_97"ms-duplex＝"ruleset.type"id＝"rulesettype">

[0092] <option value＝"mc"> Multiple copies< / option>

[0093] <option value＝"ec"> Erasure Coding< / option>

[0094] < / select>

[0095]

[0096]

[0097]

[0098] <form method＝"post"id＝"addDirForm"onsubmit＝"return false；">

[0099] <div class＝"form-group mb10">

[0100] <label class＝"control-label label"> <em class＝"em_position">*< / em> Directory Name:< / label>

[0101]

[0102]

[0103] <option ms-repeat＝"directorylist"ms-attr-value＝"el.id">{{el.directoryName}}

[0104] < / option>

[0105]

[0106]

[0107]

[0108] <div ms-if＝"！isaddnfsshare">

[0109] <label class＝"control-label label"> <em class＝"em_position">*< / em> Access path:< / label>

[0110]

[0111] <input type="text" style="margin-left:8px;width:97%;"

[0112] class="form-control input-sm radius_0" readonly

[0113] ms-attr-value=" / {{selectNfsShare.directoryName}}"

[0114] id="nfsPath" name="nfsPath" placeholder="Access path">

[0115]

[0116]

[0117] < / form>

[0118] B) Logical processing layer components

[0119] The JAVA controller processes the parameters received from the view layer, submits them to the proxy layer for processing through MAP data, receives the return from the proxy layer and then feeds back to the front-end page. The specific implementation is as follows:

[0120] @SysLog("Add storage pool")

[0121] @ResponseBody

[0122] @RequestMapping(value="{""},method="{org.springframework.web.bind.annotation.RequestMethod.POST})

[0123] @ApiOperation(value="Add storage pool",notes="Add storage pool")

[0124] public String save(@RequestBody DepotDTO depotDTO)throwsVgsRestException,JsonProcessingException{if((depotDTO.getName()==null)||("".equals(depotDTO.getName()))){

[0125] throw new VgsRestException("");

[0126] }

[0127] if(this.depotService.checkNameExist(depotDTO.getName())){

[0128] throw new VgsRestException(StrUtil.format("{}This name already exists, please change another name",new Object[]{depotDTO.getName()}));

[0129] }

[0130] long allCapacity=licenseInfo.getCapacity().longValue();

[0131] double existCapacity=vgStorLicenseService.calTotalGBForAllClusters() / 1024.0D;

[0132] try{

[0133] VgStorDepot depot=VgStorDepot.form(depotDTO);

[0134] this.depotService.saveDepot(depot,depotDTO.getRuleset());

[0135] }catch(VgsRestException e){

[0136] log.error("Storage pool save failed",e);

[0137] throw new VgsRestException(e);

[0138] }

[0139] return this.mapper.writeValueAsString(new ResponseResult(this.depotService.selectList(new EntityWrapper())));}

[0140] @SysLog("Add directory")

[0141] @RequestMapping(value = {"",}, method = {org.springframework.web.bind.annotation.RequestMethod.POST}, produces = {"text / html;charset=UTF-8"})

[0142] @ResponseBody

[0143] @ApiOperation(value = "Add directory", notes = "Add directory")

[0144] public String addDir(@RequestBody Directory directory)

[0145] throws VgsRestException

[0146] {

[0147] try

[0148] {

[0149] this.vgStorDirectoryConfigService.addDir(directory);

[0150] return this.mapper.writeValueAsString(new ResponseResult());

[0151] } catch (Exception e) {

[0152] throw new VgsRestException(e.getMessage());

[0153] }

[0154] }

[0155] @SysLog("Add nfs share")

[0156] @RequestMapping(value = {" / {dirId}"},

[0157] method = {org.springframework.web.bind.annotation.RequestMethod.POST},

[0158] produces="{"text / html;charset=UTF-8"})

[0159] @ResponseBody

[0160] @ApiOperation(value="Add nfs share",notes="Add nfs share")

[0161] public String addDirNfsShare(@PathVariable("dirId")String dirId,@RequestBody List <directorynfsacl>directoryNfsAcls)

[0162] throws VgsRestException

[0163] {

[0164] try

[0165] {

[0166] this.vgStorDirectoryNfsService.addDirNfsShare(dirId,directoryNfsAcls);

[0167] return this.mapper.writeValueAsString(new ResponseResult());

[0168] }catch(Exception e){

[0169] throw new VgsRestException(e.getMessage());

[0170] }

[0171] }

[0172] C) Proxy layer components

[0173] The proxy layer component is written in Python scripts, which mainly completes the submission and return of the underlying commands, updates the OSDMap of the cluster through the monitor, and adds new storage pool information to it. At the same time, the monitor will also send a command to create a PG to the main storage unit. After receiving the command to create a PG, the main storage unit will calculate the initial position of the PG according to the CRUSH algorithm and allocate the corresponding storage space for it. When creating a new directory, the metadata server will create a new directory entry in the metadata structure maintained internally. The metadata server node will listen to the NFS protocol request and convert the access to the distributed file system into the NFS protocol format, thereby allowing the GPU management platform to access the file directory in the distributed file system through the network. The relevant processing process can be described as:

[0174] def post(self):

[0175] sys_logger.debug('handle create new depot')

[0176] result = {'code':'SUCCESS'}

[0177] size='3'

[0178] depot=request.json

[0179] rule_name=str(depot['name'])+"_rule"

[0180] ruleset_info=self._create_rule_set(rule_name,str(depot['domainName']),str(depot['domainType']),str(depot['type']),str(depot['config']))

[0181] rule_id=ruleset_info['ruleId']

[0182] depot_name=str(depot['name'])

[0183] pattern=str(depot['type'])

[0184] if pattern=='mc':

[0185] size=str(depot['config'])

[0186] pg_num=int(depot['pgNum'])

[0187] result['code']=local_ceph.create_pool(depot_name,pg_num,size,pattern,rule_id,rule_name)

[0188] if result['code']=='SUCCESS':

[0189] result['poolId']=local_ceph.get_pool_id(depot_name)

[0190] else:

[0191] self._delete_rule_set(depot_name,pattern)

[0192] return jsonify(result)

[0193] def put(self):

[0194] executor_class=request.headers.get('X-Cmd-Name')

[0195] executor=eval('executors.%s()'%executor_class)

[0196] return executor.execute()

[0197] Further, Figure 3 FIG. 1 shows a flowchart of the implementation of the intelligent data storage interface and storage resource integration of an embodiment of the present invention. Figure 3 In the embodiment of the present invention, the integration of the intelligent data storage interface and the storage resource described in step S200 further includes the following steps:

[0198] S201, container cluster deployment and configuration:

[0199] S202, deploy the intelligent storage VGCSI plug-in;

[0200] S203. Operation and coordination of VGCSI plug-in.

[0201] Specifically, in an embodiment of the present invention, the integration of the intelligent data storage interface and storage resources is implemented through the independently developed VGCSI plug-in. The VGCSI plug-in provides a unified interface to achieve decoupling and integration with various storage systems, and provide flexible storage options for training task containers. The VGCSI plug-in includes a VGCSI controller, a VGCSI node, and external components. The plug-in and driver calls are through the grpc protocol, which is rich in functions and supports dynamic provision, fast, dynamic expansion of storage volumes, etc. The VGCSI agent running on the Kubernetes node is responsible for forwarding storage operations in the container to the CSI driver; the driver is responsible for implementing all operations defined in the CSI interface. The driver can be a plug-in based on the file system, block storage, object storage, or other storage types. It runs on the Kubernetes control plane to handle management operations related to storage volumes.

[0202] Specifically, in the embodiment of the present invention, the container cluster deployment and configuration process is based on the domestic Kylin operating system environment. First, through automated scripts or manual configuration, ensure that all nodes (including management nodes and computing nodes) have installed compatible versions of Docker or other container runtime environments. Subsequently, using container orchestration tools such as Kubernetes, the Kubernetes cluster is deployed on the Kylin operating system. In the Kubernetes cluster, Pod, Service and other resources are defined through YAML files, and the network interconnection between the management node and the computing node is realized through the Calico network plug-in.

[0203] The deployment of the smart storage VGCSI plug-in is a key step in integrating the data storage interface with storage resources. First, obtain the installation package or source code of the VGCSI plug-in, and build and package it according to the documentation or guide. Then, create the corresponding resource definition files (such as Deployment, DaemonSet, etc.) in the Kubernetes cluster and deploy the VGCSI controller and node components to the cluster.

[0204] Among them, VGCSI controller: As the management center of storage resources, VGCSI controller usually runs in the computing cluster in the form of Deployment or StatefulSet. It is responsible for processing operations such as creation, deletion, and expansion of storage resources, and interacts with the underlying storage system to achieve dynamic management of storage resources.

[0205] VGCSI node component: Deployed on each computing node in the form of DaemonSet, it is responsible for performing tasks such as mounting and unmounting storage volumes. When a Pod needs to mount a storage volume, the node component communicates with the controller to obtain the storage volume information and performs the mounting operation on the local node.

[0206] The operation and collaboration of the VGCSI plug-in communicate and work together through the Kubernetes API. The controller is responsible for global storage resource management, while the node component is responsible for performing specific storage operations on the local node. When an application needs to access storage resources, it will initiate a request to the VGCSI plug-in through mechanisms such as Kubernetes Service or Persistent Volume Claim (PVC). The VGCSI plug-in calls the corresponding controller and node components to complete operations such as allocation, mounting, and access of storage resources based on the type and parameters of the request.

[0207] The VGCSI plug-in workflow includes storage volume request, volume provisioning, volume binding, volume mounting, container application access to storage, and volume assistance and deletion. The storage services provided by the VGCSI plug-in are mainly VGCSI identity service, VGCSI controller service, and VGCSI node service. The VGCSI identity service is mainly responsible for exposing the plug-in's own information to the outside world so that the GPU management platform can identify the plug-in. The protocol code is defined as follows:

[0208] service Identity

[0209] / / return the version and name of the plugin

[0210] rpc GetPluginInfo(GetPluginInfoRequest)returns(GetPluginInfoResponse){}

[0211] / / Report whether the plugin has the ability to provide services for the controller interface

[0212] rpc GetPluginCapabilities(GetPluginCapabilitiesRequest)returns(GetPluginCapabilitiesResponse){}

[0213] / / Called by CO just to check if the plugin is running

[0214] rpc Probe(ProbeRequest)returns(ProbeResponse){}

[0215] }

[0216] The VGCSI controller service defines the operations of the volume plug-in management interface, which is also the core logic part. The interface definition code is as follows:

[0217] CreateVolume(context.Context,*CreateVolumeRequest)(*CreateVolumeResponse,error)

[0218] DeleteVolume(context.Context,*DeleteVolumeRequest)(*DeleteVolumeResponse,error)

[0219] ControllerPublishVolume(context.Context,*ControllerPublishVolumeRequest)(*ControllerPublishVolumeResponse,error)

[0220] ControllerUnpublishVolume(context.Context,*ControllerUnpublishVolumeRequest)(*ControllerUnpublishVolumeResponse,error)

[0221] ValidateVolumeCapabilities(context.Context,*ValidateVolumeCapabilitiesRequest)(*ValidateVolumeCapabilitiesResponse,error)

[0222] ListVolumes(context.Context,*ListVolumesRequest)(*ListVolumesResponse,error)

[0223] GetCapacity(context.Context,*GetCapacityRequest)(*GetCapacityResponse,error)

[0224] ControllerGetCapabilities(context.Context,*ControllerGetCapabilitiesRequest)(*ControllerGetCapabilitiesResponse,error)

[0225] CreateSnapshot(context.Context,*CreateSnapshotRequest)(*CreateSnapshotResponse,error)

[0226] DeleteSnapshot(context.Context,*DeleteSnapshotRequest)(*DeleteSnapshotResponse,error)

[0227] ListSnapshots(context.Context, *ListSnapshotsRequest) (*ListSnapshotsResponse, error)

[0228] ControllerExpandVolume(context.Context, *ControllerExpandVolumeRequest) (*ControllerExpandVolumeResponse, error)

[0229] ControllerGetVolume(context.Context, *ControllerGetVolumeRequest) (*ControllerGetVolumeResponse, error)

[0230] }

[0231] The VGCSI node service defines that these three methods will be called by the kubelet through the node - driver - registrar container during the mount stage: NodePublishVolume, NodeUnpublishVolume, and NodeGetCapabilities. The code definition is as follows:

[0232] type NodeServer interface{

[0233] NodeStageVolume(context.Context, *NodeStageVolumeRequest) (*NodeStageVolumeResponse, error)

[0234] NodeUnstageVolume(context.Context, *NodeUnstageVolumeRequest) (*NodeUnstageVolumeResponse, error)

[0235] NodePublishVolume(context.Context, *NodePublishVolumeRequest) (*NodePublishVolumeResponse, error)

[0236] NodeUnpublishVolume(context.Context,*NodeUnpublishVolumeRequest)(*NodeUnpublishVolumeResponse,error)

[0237] NodeGetVolumeStats(context.Context,*NodeGetVolumeStatsRequest)(*NodeGetVolumeStatsResponse,error)

[0238] NodeExpandVolume(context.Context,*NodeExpandVolumeRequest)(*NodeExpandVolumeResponse,error)

[0239] NodeGetCapabilities(context.Context,*NodeGetCapabilitiesRequest)(*NodeGetCapabilitiesResponse,error)

[0240] NodeGetInfo(context.Context,*NodeGetInfoRequest)(*NodeGetInfoResponse,error)

[0241] }

[0242] In actual operation, the domestic GPU management platform achieves efficient utilization and management of storage resources through sophisticated storage volume management and permission control mechanisms. Specifically, the platform first uses the distributed storage resources created by S100 to allocate independent storage volumes for each training task to ensure data security and privacy. Then, with the help of the VGCSI plug-in of the Kubernetes cluster, these storage volumes can be dynamically mounted to the required containers, providing flexible and elastic storage support for training tasks.

[0243] In order to solve the problem of storage resource competition that may exist among multiple training tasks, the platform adopts advanced resource scheduling algorithms and quota management mechanisms. By monitoring the usage of storage resources in real time, the platform can intelligently adjust the storage quota of each task to ensure that key tasks can obtain sufficient resource support while avoiding idleness and waste of resources. In addition, the platform also supports storage resource reservation and preemption strategies, further improving the flexibility and reliability of resource scheduling.

[0244] In the process of using storage resources, the domestic GPU management platform also pays attention to the persistence and availability of data. Through the copy redundancy mechanism and the multi-ReplicaSet mechanism to ensure the security of data and applications, the platform can effectively respond to emergencies such as hardware failures and ensure the integrity and recoverability of training tasks. At the same time, the platform also supports data version management and monitoring management functions, which makes it convenient for users to trace back and view historical data at any time, providing strong support for the iteration and optimization of training tasks.

[0245] Further, Figure 4 The following is a flowchart showing the implementation of storage resource management and monitoring of a domestic GPU management platform according to an embodiment of the present invention. Figure 4 In the embodiment of the present invention, the storage resource management and monitoring of the domestic GPU management platform described in step S300 further includes the following steps:

[0246] S301, visual configuration of storage management: storage resource creation, resource binding, resource use, resource release and resource recycling are realized through a visual interface;

[0247] S302, shared use of storage management: multiple containers are mounted together with a storage to achieve storage sharing;

[0248] S303, monitoring and alarming of storage resources: The usage rate, remaining amount, associated containers, storage status and storage type of container storage resources can be viewed through a visual interface. Alarm reminders can also be issued when storage resources are insufficient, thereby improving operation and maintenance efficiency.

[0249] Specifically, in an embodiment of the present invention, the use of distributed storage by the GPU management platform includes three categories: storage class, persistent volume and NAS storage. The storage class defines how to dynamically create and manage persistent volumes. The storage class contains information about the type of storage volume, access mode, recycling policy, etc. Through the storage class, the GPU management platform can automatically create and manage storage space according to the user's storage needs without manually creating and configuring each storage volume; the persistent volume is pre-configured by the GPU management platform, and the persistent volume has attributes such as capacity, access mode and recycling policy, which are used to meet the persistent storage requirements of container training tasks; NAS storage is integrated with distributed storage through NFS sharing, FTP sharing, and SMB sharing, and the GPU management platform mainly uses standard storage protocols to realize the use of distributed storage and management of storage resources.

[0250] Specifically, in an embodiment of the present invention, in the process of visual configuration of storage management, when selecting a storage class, the storage pool of distributed storage can be docked through the VGCSI plug-in. The GPU management platform provides two mechanisms, YAML or visual configuration form, to integrate with distributed storage. The configuration form needs to fill in the monitor address, storage pool name, access mode, and storage system administrator key of the distributed storage cluster. It can also be integrated with the storage system through YAML. During the writing, the VGCSI plug-in interface must be formulated, and then the kubectl apply command is used to apply the storage class to the storage module of the GPU management platform. When selecting a persistent volume, the process is basically the same as selecting a storage class, but the configuration options are different in configuring the visual form or writing YAML, because the persistent volume type pre-allocated space needs to fill in the capacity information. When selecting NAS storage, it can be docked with distributed storage through NFS, SMB or FTP, and the GPU management platform uses storage resources through the standard NAS storage protocol interface. When selecting NAS storage, you only need to fill in the gateway address and sharing path. If it is SMB or FTP sharing, you also need to fill in the user name and password related to the storage resource permissions.

[0251] Specifically, in the embodiment of the present invention, in actual production, the same training task often uses multiple containers to share the Pod-level storage volume Volume. The storage volume Volume can be defined as various types, and multiple containers perform mounting operations respectively, and a Volume can be mounted as the directory required inside the container. For each container, the access capability of the shared persistent volume can be set, and the single-host read-write, multi-host read-only and multi-host read-write access modes can be set. The GPU management platform specifies the persistent volume to be used through the volumes field in the configuration of the training task container, and specifies the mount path of the persistent volume in the container in the volumeMounts field of each container. In this way, all containers in the Pod can realize data sharing by mounting the same persistent volume. The GPU management platform supports data sharing between multiple containers through the Pod and Volume mechanism, supports multiple training pods to access a PV resource at the same time; supports pods to access the original data on the new node when reconstructing across nodes, and can also create persistent volumes and persistent volume declarations to manage storage resources, and users can mount these persistent volumes in the Pod to realize data sharing between containers. This mechanism enables Kubernetes to support more complex and flexible application deployment scenarios.

[0252] Specifically, in an embodiment of the present invention, the GPU management platform not only fully supports the management, monitoring and alarm analysis functions of storage resources, but also through the intuitive data screen, users can easily view the various types of plug-in storage, the current storage usage, the remaining available amount, and the estimated number of days of use based on the current usage speed. In addition, in the detailed statistics of storage resources, users can obtain key information such as the name of the storage resource, the specific amount of allocated storage, the remaining storage, the current usage rate, the number of associated PVCs, and can clearly see the corresponding relationship between each PVC and the training task container, the access method, the current status, the cluster to which it belongs, and the user to which it belongs. Detailed information. Furthermore, the alarm list function allows users to customize the threshold of storage resources. Once the storage usage of a user or the entire cluster exceeds the preset threshold, the system will automatically trigger an alarm to promptly remind users to perform capacity expansion operations or delete unnecessary PVC snapshots and storage resources to optimize storage usage efficiency. This series of storage resource monitoring and management functions provided by the GPU management platform provides system administrators and users with powerful tools to ensure the effective use of storage resources and respond to potential resource shortages in a timely manner.

[0253] Further, Figure 5 The following is a flowchart of GPU training task data sharing and access in accordance with an embodiment of the present invention. Figure 5 In an embodiment of the present invention, the GPU training task data sharing and access described in step S400 further includes the following steps:

[0254] S401. Create a domestic GPU basic image: When the platform determines that there is a domestic GPU training card, the system uses the Kylin basic container image as a template to create a basic container containing the driver of the domestic GPU card, and uses docker commit to create a basic image and import it into the image template;

[0255] S402. Create a training task through the GPU management interface: Create a GPU training task through the GPU visualization management interface, select the domestic GPU basic image created in the previous step, set GPU resources and mounted storage volumes to create a training task;

[0256] S403, tenant training result data sharing and access: The data generated after GPU training under the same tenant is stored in the mounted storage volume and can be mounted for use by other training tasks through visualization operations.

[0257] Specifically, in the embodiment of the present invention, the domestic GPU training task needs to first create a basic image based on the Kubernetes container management platform. Because the embodiment of the present invention is built based on the domestic Feiteng 2000+ / 64 processor and the domestic Kirin V10 operating system, when there is a domestic GPU card, it is necessary to first use the domestic Kirin minimized operating system image as the basis, deploy domestic GPU-related SDK drivers and vGPU-related tasks, management, and scheduling-related plug-ins, and then import the libcuda and libixml runtime library configurations. Finally, use docker commit to create a basic domestic GPU container image and import it into the GPU image list through the GPU management platform.

[0258] The specific implementation process of creating a training task through the GPU management interface is as follows:

[0259] In the model training, select the training task, select the domestic GPU card training container image, fill in the task name, the number of workers, the number of GPUs, the amount of memory, the storage volume, the data set and other form contents, and start the training task to realize the execution of training tasks such as images, texts, videos, and large models. The storage involved in artificial intelligence training includes the data set and the model data generated in the training task. For the shared data set, the GPU management platform uses the NAS storage in the persistent configuration of the storage management. The platform administrator can put the commonly used COCO, WordNet, TNO and other data sets into the NAS storage. Based on the GPU management platform, the data set directory can be managed, and folders can be created, modified, deleted, and moved. The data set can also be set as a private and shared data set; the storage volume in the specific training task container can be used to place the materials and result data generated in the training task. The platform provides model version management and batch management of training materials. After the training is completed, a version is generated. After multiple trainings, the versions can be compared to provide a reference for the use of the best model file for the inference task.

[0260] Specifically, in the embodiment of the present invention, the core of the tenant training result data sharing and access mechanism is to achieve efficient and flexible data management and utilization. Specifically, when GPU training is performed in the same tenant environment, the generated data will be properly stored in the mounted storage volume. This design not only ensures the immediate storage of data, but also facilitates its subsequent use. Through an intuitive visual interface, users can easily mount these storage volumes to other training tasks to achieve data sharing. This mechanism supports multiple pods (i.e., containerized application instances) to access the same persistent volume (PV) resource at the same time, so that these mounted storage resources will be automatically released when the container is destroyed, and the status will change from "Bound" to "Available" for the next container to bind and use, which greatly improves the concurrency and efficiency of data processing. Furthermore, when the training task container needs to be migrated or rebuilt between different nodes, this mechanism can ensure that the original data can still be seamlessly accessed on the new node. This cross-node data access capability provides tenants with a highly flexible and reliable data usage environment, effectively meeting the storage requirements of large-scale training tasks.

[0261] The visualized intelligent storage management method for the domestic GPU management platform of the embodiment of the present invention pre-allocates intelligent storage space based on the domestic platform, and then realizes the demand for intelligent storage of GPU training tasks through the visualized intelligent storage plug-in technology of the GPU management platform. This solution not only improves the flexibility and scalability of storage resources, but also significantly improves the stability and efficiency of training tasks, and provides strong storage support for the application of GPU in complex storage tasks. Through the deep docking of intelligent storage plug-ins with distributed storage, this technology realizes the seamless integration of intelligent storage and GPU management platform, supports multi-machine distributed cross-node automatic allocation of computing resources and automatic scheduling of training tasks, allows GPU training tasks in the cluster to share external distributed storage space, realizes storage sharing of multiple training tasks, and GPU training tasks can dynamically apply for and release storage resources according to actual needs. At the same time, this technology also provides rich visual management functions, and users can monitor and manage GPU training tasks and their storage resources in real time through an intuitive interface, thereby greatly improving management efficiency and user experience. The emergence of this technology provides a more efficient and flexible solution for the application of domestic GPU management platforms in fields such as artificial intelligence and AI storage.

[0262] The visualized intelligent storage management method for the domestic GPU management platform of the embodiment of the present invention can realize the application and use of file storage resources for GPU management platform training and reasoning tasks through a visualized interface, realize intelligent management and application of storage resources through the GPU management platform, realize visualized data sharing management and storage integration for the domestic GPU management platform, realize life cycle management of storage resources according to dynamic resource allocation and static resource allocation, and realize the bundling of storage resources and tenant storage requirements through the GPU management platform, which not only reduces the complexity of storage configuration management for training and reasoning applications, but also solves the problem of data sharing and data governance in training tasks based on the domestic GPU environment. The embodiment of the present invention divides the storage pool of the distributed storage system through a Web interface, creates multiple file storage directories based on the storage pool, and then uses the resources of the storage pool or storage directory as storage units for the GPU management platform to mount through a shared gateway. The GPU management platform reallocates the storage resources through an intelligent storage plug-in, and automatically mounts the storage volume to the container of the GPU management platform according to the configuration, meeting the training task container's demand for persistent storage.

[0263] Correspondingly, based on the visualized intelligent storage management method for domestic GPU management platform in the embodiment of the present invention, the embodiment of the present invention also proposes a visualized intelligent storage management system for domestic GPU management platform, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the visualized intelligent storage management method for domestic GPU management platform in the embodiment of the present invention is implemented.

[0264] Although one or more embodiments of the present invention are described above, it should be known to those skilled in the art that the present invention can be implemented in any other form without departing from its subject matter and scope. Therefore, the embodiments described above are illustrative rather than restrictive, and many modifications and substitutions are obvious to those skilled in the art without departing from the spirit and scope of the present invention as defined in the appended claims.< / directorynfsacl> < / form>

Claims

1. A visual intelligent storage management method for a domestic GPU management platform, characterized in that: include: File storage resource creation and initialization; Intelligent data storage interface and storage resource integration; Domestic GPU management platform storage resource management and monitoring; GPU training task data sharing and access.

2. According to claim 1, the visual intelligent storage management method for domestic GPU management platform is characterized in that: The file storage resource creation and initialization further includes: Distributed storage pool creation and configuration; File directory creation and management; Creation and configuration of shared storage services.

3. The visual intelligent storage management method for domestic GPU management platform according to claim 2 is characterized in that: The distributed storage pool creation and configuration further includes: Before creating a distributed storage pool, set the hard disk to JBOD mode based on the domestic Kylin system, and add the solid-state disk and mechanical disk to different hard disk groups; Create a distributed storage pool, select the storage pool type as file storage or block storage, select the data protection mechanism, select the pre-built disk group, fill in the storage pool name, and call the Python script to the corresponding storage node through JAVA HTTP request to execute the command to create the storage pool.

4. According to claim 3, the visual intelligent storage management method for domestic GPU management platform is characterized in that: The file directory creation and management are specifically as follows: Based on the operation of multiple domestic storage servers, multiple copies or erasure code data protection mechanisms are selectively set up, and multiple logical volumes are built based on the storage pool; The logical volumes include distributed volumes, stripe volumes, copy volumes, distributed stripe volumes and distributed copy volumes.

5. The visual intelligent storage management method for domestic GPU management platform according to claim 4 is characterized in that: The intelligent data storage interface and storage resource integration further include: Container cluster deployment and configuration: Deploy container clusters based on the domestic Kylin operating system, interconnect management nodes and computing nodes through the Calico network, and deploy basic images; Deploy the intelligent storage VGCSI plug-in: install the VGCSI controller component and VGCSI node component in the computing cluster; Operation and coordination of the VGCSI plug-in: Run the VGCSI controller component in the computing cluster as a Deployment or StatefulSet to create, delete, and expand storage resources, and deploy the VGCSI node component as a DaemonSet and run it on each computing node to mount and unmount storage volumes.

6. The visual intelligent storage management method for domestic GPU management platform according to claim 5 is characterized in that: The domestic GPU management platform storage resource management and monitoring further includes: Visual configuration of storage management: storage resource creation, resource binding, resource usage, resource release and resource recycling are realized through a visual interface; Shared use of storage management: multiple containers can mount a storage together to achieve storage sharing; Monitoring and alerting of storage resources: View the usage rate, remaining amount, associated containers, storage status, and storage type of container storage resources through a visual interface.

7. The visual intelligent storage management method for domestic GPU management platform according to claim 6 is characterized in that: The GPU training task data sharing and access further includes: Create a domestic GPU basic image: Use the Kylin basic container image as a template to create a training container template containing the driver of the domestic GPU card, and use docker commit to create a basic image and import it into the image template; Create a training task through the GPU management interface: Create a GPU training task through the GPU visual management interface, select the pre-created domestic GPU image template, set the GPU resources and mounted storage volume to create a training task; Tenant training result data sharing and access: The data generated by domestic GPU training under the same tenant is saved in the mounted storage volume, and mounted for use by other training tasks through visualization operations.

8. A visual intelligent storage management system for domestic GPU management platform, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the visualized intelligent storage management method for a domestic GPU management platform as described in any one of claims 1 to 7.

Images