Development machine operation task processing method, device, equipment and storage medium

Abstract

The present application discloses a processing method, device, equipment and storage medium for development machine operation tasks, and relates to the field of deep learning of artificial intelligence. The specific implementation scheme is: receiving a task creation request initiated by a client; generating a development machine operation task according to the task creation request; allocating a target image processor GPU required to execute the development machine operation task to the development machine operation task; sending a development machine operation task request to the master node in the cluster node, and the task request is used to request to execute the development machine operation task on the target GPU. Compared with the prior art, the present application uses a docker container to execute the development machine operation task on the GPU, so that the operating system of the local host can be directly used, thereby improving the utilization rate of the hardware of the physical machine.

Description

Technical Field

[0001] The present application relates to the field of deep learning of artificial intelligence in data processing, and in particular to a processing method, device, equipment and storage medium for development machine operation tasks. Background Art

[0002] Since the concept of deep learning was proposed, deep learning has made great progress in both theory and application. Existing deep learning training tasks are all run on high-performance graphics processing unit (GPU) clusters. In order to achieve the goal of consistent development and training environments, most developers also use GPU development machines for development and debugging.

[0003] The current mainstream way to develop machines is to use platform virtualization technology to establish an abstract virtualization platform between computer, storage and network hardware, so that all the hardware of the physical machine is unified into a virtualization layer. A virtual machine with the same hardware structure as the physical machine is created on top of the virtualization platform, and developers can perform development machine operation tasks on the virtual machine. Since virtual machines do not interfere with each other, system resources can be protected.

[0004] However, the virtual machine needs to encapsulate the real hardware layer of the physical machine, and virtualization will inevitably occupy a part of the physical machine's resources, resulting in a loss of some of the physical machine's performance, thereby making the utilization rate of the physical machine's hardware low. Summary of the invention

[0005] The present application provides a processing method, apparatus, device and storage medium for developing machine operation tasks.

[0006] According to one aspect of the present application, a method for processing a development machine operation task is provided, receiving a task creation request initiated by a client;

[0007] Generate a development machine operation task according to the task creation request;

[0008] Allocating a target graphics processor GPU required to execute the development machine operation task to the development machine operation task;

[0009] The development machine operation task request is sent to a master node in the cluster nodes, where the task request is used to request execution of the development machine operation task on the target GPU.

[0010] According to a second aspect of the present application, a method for processing a development machine operation task is provided, comprising:

[0011] A development machine operation task request received from a task management server, wherein the task request is used to request execution of the development machine operation task on a target GPU;

[0012] Determine the target working node according to the running status of multiple working nodes in the cluster node;

[0013] The Docker container of the target working node is scheduled to execute the development machine operation task on the target GPU.

[0014] According to a third aspect of the present application, a device for processing a development machine operation task is provided, comprising:

[0015] A receiving module, used to receive a task creation request initiated by a client;

[0016] A processing module, configured to generate a development machine operation task according to the task creation request; and to allocate a target image processor GPU required to execute the development machine operation task to the development machine operation task;

[0017] The sending module is used to send the development machine operation task request to the master node in the cluster node, and the task request is used to request to execute the development machine operation task on the target GPU.

[0018] According to a fourth aspect of the present application, there is provided a device for processing a development machine operation task, comprising:

[0019] A receiving module, used for receiving a development machine operation task request sent by a task management server, wherein the task request is used for requesting execution of the development machine operation task on a target GPU;

[0020] The processing module is used to determine the target working node according to the running status of multiple working nodes in the cluster node; and schedule the Docker container of the target working node to execute the development machine operation task on the target GPU.

[0021] According to a fifth aspect of the present application, there is provided an electronic device, including:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0025] According to a sixth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect above.

[0026] According to the seventh aspect of the present application, a computer program product is provided, the program product comprising: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0027] The technology of this application solves the problem of low utilization of physical machine hardware. Compared with the prior art, this application uses Docker containers to perform development machine operation tasks on the graphics processor GPU, so that the operating system of the local host can be directly used, thereby improving the utilization of the physical machine hardware.

[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0030] Figure 1 A schematic diagram of a scenario of a method for processing a development machine operation task provided in an embodiment of the present application;

[0031] Figure 2 A system architecture diagram of a development machine operation task provided in an embodiment of the present application;

[0032] Figure 3 A signaling interaction diagram of a method for processing a development machine operation task provided in an embodiment of the present application;

[0033] Figure 4 A flowchart of a method for processing a development machine operation task provided in an embodiment of the present application;

[0034] Figure 5 A flowchart of another method for processing development machine operation tasks provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the structure of a device for processing development machine operation tasks provided in an embodiment of the present application;

[0036] Figure 7A schematic diagram of the structure of another device for processing development machine operation tasks provided in an embodiment of the present application;

[0037] Figure 8 It is a scene diagram of a method for processing development machine operation tasks that can implement the embodiment of the present application. DETAILED DESCRIPTION

[0038] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0039] The current mainstream way to develop machines is to use platform virtualization technology to establish an abstract virtualization platform between computer, storage and network hardware, so that all the hardware of the physical machine is unified into a virtualization layer. A virtual machine with the same hardware structure as the physical machine is created on top of the virtualization platform, and developers can perform development machine operation tasks on the virtual machine. Since virtual machines do not interfere with each other, system resources can be protected.

[0040] However, the virtual machine needs to encapsulate the real hardware layer of the physical machine, and virtualization will inevitably occupy a part of the physical machine's resources, resulting in a loss of some of the physical machine's performance, thereby making the utilization rate of the physical machine's hardware low.

[0041] The present application provides a method and device for processing development machine operation tasks, which are applied to the field of deep learning of artificial intelligence in data processing to solve the technical problem of low utilization of physical machine hardware and achieve the effect of improving the utilization of physical machine hardware. The inventive concept of the present application is: by allocating the target GPU required to execute the development machine operation task to the development machine operation task, and then sending the development machine operation task request to the master node in the cluster node, the master node schedules the docker container of the target working node to execute the development machine operation task on the target GPU.

[0042] In order to clearly understand the technical solution of this application, the terms involved in this application are explained below:

[0043] Development machine: A software program provided to developers to obtain software code through their own code during the software development process, and to compile and debug the obtained code.

[0044] Docker container: An open source application container engine that allows developers to package applications and dependent packages into a portable container in a unified way, and then publish it to any server with the Docker engine installed.

[0045] Snapshot: A fully usable copy of a specified data set that includes an image of the corresponding data at a certain point in time.

[0046] Block device: A type of input / output (I / O) device that stores information in blocks of fixed size.

[0047] The following describes the usage scenarios of this application.

[0048] Figure 1 A schematic diagram of a scenario of a method for processing a development machine operation task provided in an embodiment of the present application. Figure 1 As shown, when a user needs to perform development machine operation tasks such as development machine creation and development machine deletion, a task creation request can be sent to the task management server 102 of the development machine task processing system through the client 101. After receiving the task creation request sent by the client 101, the task management server 102 allocates the GPU required to execute the development machine operation task to the development machine operation task in the task creation request, and then sends the development machine operation task request to the master node 103 in the cluster node, and the master node 103 schedules the docker container of the working node 104 to execute the development machine operation task on the target GPU.

[0049] The client 101 may include but is not limited to: a tablet computer, a personal computer (PC), a laptop computer, a personal digital assistant (PDA), a mobile phone and other devices.

[0050] It should be noted that the application scenarios of the technical solution of this application can be Figure 1 The present invention is a processing scenario of the development machine operation task in the present invention, but is not limited to this and can also be applied to other related scenarios.

[0051] Figure 2 This is a system architecture diagram of a development machine operation task provided by an embodiment of the present application. Figure 2As shown, the client, task management server, cluster node, GPU and task database. The above-mentioned client includes a UI interface and a platform layer. The user triggers the module in the platform layer to send a task creation request to the task database through the Open API by operating on the UI interface. After receiving the task creation request, the task database sends the task creation request to the task management server. The task management server includes multiple service units. The task management server is used to process the task creation request and send a development machine operation task request to the master node in the cluster node. After the master node in the cluster node receives the development machine operation task request, it schedules the docker container of the target working node to execute the development machine operation task on the target GPU.

[0052] It can be understood that the above-mentioned method for processing development machine operation tasks can be implemented by the processing device for development machine operation tasks provided in the embodiment of the present application. The processing device for development machine operation tasks can be part or all of a certain device, for example, it can be the above-mentioned task management server and cluster master node.

[0053] The following takes the task management server and cluster master node integrated or installed with relevant execution code as an example to describe the technical solution of the embodiment of the present application in detail with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0054] Figure 3 A signaling interaction diagram of a method for processing a development machine operation task provided in an embodiment of the present application. The present application relates to how to process a development machine operation task, such as Figure 3 As shown, the method includes:

[0055] S201: The task management server receives a task creation request initiated by a client.

[0056] The development machine operation task includes at least one of the following: creating a development machine, deleting a development machine, restarting a development machine, and reinstalling a development machine.

[0057] In the present application, when a user needs to operate a development machine, the user can operate on the client so that the client sends a task creation request. In some embodiments, the client can directly send a task creation request to the task management server. In other embodiments, the client can first send a task creation request to the task database. Subsequently, the task database sends the task creation request to the task management server.

[0058] S202: The task management server generates a development machine operation task according to the task creation request.

[0059] In this step, after receiving the task creation request initiated by the client, the task management server can generate a development machine operation task according to the task creation request.

[0060] The embodiment of the present application does not limit how to generate the development machine operation task. For example, the task creation request may include task requirement data input by the user. According to the task requirement data input by the user, the task management server may generate the development machine operation task.

[0061] In the present application, after the task management server generates a development machine operation task, the development machine operation task can be added to the task queue.

[0062] It should be understood that the embodiments of the present application do not limit how to add a development machine operation task to a task queue. In some embodiments, a task scheduler service unit in a task management server can schedule the development machine operation task, and then add the development machine operation task to a corresponding task queue based on the type of the development machine operation task.

[0063] S203: The task management server allocates a target GPU required to execute the development machine operation task to the development machine operation task.

[0064] In some embodiments, the task management server may allocate a target GPU required to perform the operation task based on the resources required by the development machine to perform the operation task.

[0065] In other embodiments, the operating status of the GPUs in the cluster can also be used as a basis for determining the target GPU. For a GPU that is executing a task or a GPU that is malfunctioning, the task management server can avoid using it as a target GPU.

[0066] It should be understood that in the process of determining the target GPU, the task management server can also verify the user's authority. Exemplarily, the task management server can determine the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions. Subsequently, according to the resource usage permissions corresponding to the user group to which it belongs and the resources required for the development machine operation task, the task management server can allocate the target GPU required to execute the operation task.

[0067] It should be understood that a user group is not directly bound to a user, that is, a user group will not be granted permissions to a user so that the user included in the user group has permissions. In the present application, the system management module can determine the user's permissions by searching the pre-set entity table and association table. Among them, the entity table may include a permission table, a role table, a user table, and a user group table, and the association table may include a user-user group association table, a role-user group association table, and a permission-role association table.

[0068] In this application, by setting different resource usage permissions for different user groups, the target GPU required for the development machine operation task can be developed according to the resource usage permissions corresponding to the user groups, thereby achieving reasonable management and control of the resources available to the user groups.

[0069] In other embodiments, the task creation request also includes the resource quota required to execute the development machine operation task. Accordingly, after determining the resource usage quota of the user group to which the development machine operation task belongs, the task management server can compare the resource quota required for the development machine operation task with the resource usage quota of the user group. If the resource usage quota of the user group is greater than or equal to the amount of resources required for the development machine operation task, the target GPU required to execute the operation task is allocated. If the resource usage quota of the user group is less than the amount of resources required for the development machine operation task, an error prompt is issued to the client. Accordingly, after completing the development machine operation task, the task management server can subtract the amount of resources required for the development machine operation task from the resource usage quota of the user group.

[0070] In this application, by setting a resource usage quota for a user group, the user group can only use an amount of resources less than or equal to the resource usage quota within a period of time to perform development machine operation tasks, thereby avoiding excessive resource usage by the user group.

[0071] It should be understood that the user group administrator can also call an open application programming interface (Open API) to determine the resource quota of the user group, thereby limiting the resources that the user group can use.

[0072] In some embodiments, for a development machine with low GPU utilization, the system management module may also make notifications or even release resources based on the GPU utilization.

[0073] Exemplarily, the task management server may query the resource utilization rate of the target GPU by the development machine operation task in the task database. If the resource utilization rate of the GPU by the development machine operation task is lower than the first threshold, the task management server sends a task release instruction to the master node, which releases the development machine operation task on the target GPU.

[0074] In some embodiments, for a development machine with a high GPU utilization rate, the task management server may also reassign a target GPU for the development machine operation task.

[0075] Exemplarily, the task management server may query the resource utilization of the target GPU in the task database. If the resource utilization of the target GPU exceeds the second threshold, the target GPU is reallocated to the development machine operation task. A development machine operation task request is sent to the master node based on the reallocated GPU.

[0076] In this application, through the above method, the task management server can achieve efficient management of development machine operation tasks, user groups, etc., thereby eliminating the need for developers to manually handle development machine operation and maintenance issues.

[0077] S204. The task management server sends a development machine operation task request to the master node in the cluster nodes. The task request is used to request execution of the development machine operation task on the target GPU.

[0078] In the present application, after the task management server allocates the target image processor GPU required to execute the development machine operation task to the development machine operation task, it can send a development machine operation task request to the master node in the cluster node, so as to execute the development machine operation task on the target GPU.

[0079] It should be understood that the embodiment of the present application does not limit how to send the development machine operation task request to the master node in the cluster node. In some embodiments, the development machine operation task can be sent to the master node through the task worker service unit.

[0080] Specifically, the cluster nodes may be a Kubernetes (K8S) architecture.

[0081] The following is an explanation of the K8S architecture. The K8S architecture can divide the GPU into a master node (K8S Master) and a group of worker nodes. The master node is responsible for maintaining the target state of the cluster and running a set of processes related to cluster management, such as kube-apiserver, controller-manager, and scheduler. The above processes can implement cluster resource management and Pod (a programming language) scheduling on the worker nodes. Among them, the worker nodes run real applications, the smallest operating unit pod managed by K8S on the worker nodes, and the kubelet and kube-proxy processes. The Kubelet and kube-proxy processes are responsible for pod creation, startup, monitoring, restart, destruction, and discovery and load balancing of services within the cluster.

[0082] In some optional implementations, after sending a development machine operation task request to a master node in a cluster node, the task management server may also update a snapshot of the development machine corresponding to the development machine operation task, where the snapshot is a logical relationship between data of the development machine.

[0083] It should be noted that the update of the snapshot of the development machine may include the creation of the snapshot of the development machine and the deletion of the snapshot of the development machine. The update of the snapshot of the development machine may be completed by the task worker service unit.

[0084] In some optional implementations, after sending the development machine operation task request to the master node in the cluster nodes, the task management server may also determine the block device required for the development machine operation task, where the block device is used to request storage resources for the development machine operation task.

[0085] The updating of the block device required for the development machine operation task can also be completed through the task status synchronization (task status sync) service unit in the task management module.

[0086] In addition, the task status sync service unit can also monitor cluster nodes.

[0087] S205. The master node determines a target working node according to the operating status of multiple working nodes in the cluster node.

[0088] The embodiment of the present application does not limit how the master node determines the target working node based on the operating status of multiple working nodes in the cluster node.

[0089] Exemplarily, the master node may first determine the operating status of the working nodes that meet the requirements, and then select the target working node from them. Exemplarily, the master node may first determine the working node that has failed, and then determine the target working node from the working nodes other than the working node that has failed.

[0090] S206. The master node schedules the docker container of the target working node to execute the development machine operation task on the target GPU.

[0091] In this step, after the master node determines the target working node according to the running status of multiple working nodes in the cluster node, it can schedule the Docker container of the target working node to execute the development machine operation task on the target GPU.

[0092] In some embodiments, the master node can also monitor the execution progress of the development machine operation task of the target working node and the status of the development machine corresponding to the development machine operation task, and send the execution progress of the development machine operation task and the status of the development machine corresponding to the development machine operation task to the task database.

[0093] In some embodiments, the master node may also monitor the resource utilization of the target GPU by the development machine operation tasks, and send the resource utilization of the target GPU to the task database.

[0094] In some embodiments, the master node also stores the operating environment and operating data of the development machine corresponding to the development machine operation task on the backup server by remote mounting. When the target GPU fails, the development machine can be quickly restored on other GPUs through the operating environment and operating data of the development machine stored in the backup server to continue to execute the development machine operation task.

[0095] It should be understood that when the master node schedules the docker container of the target working node to perform the development machine operation task, it can directly use the operating system of the local host, making it more efficient in utilizing system resources, faster in application execution, lower in memory loss, and faster in file storage. At the same time, the use of docker containers only occupies MB-level disks, which occupies less physical machine resources than the GB-level disks of virtual machines, and can support up to thousands of containers on a single machine.

[0096] It should be understood that since the docker container application runs directly on the host kernel, there is no need to start a complete operating system. Therefore, compared with the virtual machine in the prior art, the container management module using the docker container can greatly save the operating time of the development machine, and its operating time can be achieved in seconds or even milliseconds.

[0097] It should be understood that the docker image in the snapshot of the development machine can provide a complete runtime environment except the kernel, thereby ensuring environmental consistency. At the same time, the docker image of the customized application can also solve the problem of complex and difficult development machine environment deployment.

[0098] It should be understood that while executing the development machine operation task, the containerized management module can also store the operating environment and operating data of the development machine corresponding to the development machine operation task on the backup server by remote mounting. Through the backup server, if the physical machine in the development machine task processing system crashes or fails, the development machine instance can be quickly migrated to other physical machines, which ensures data security while reducing the waiting time for developers due to machine failures.

[0099] The processing method of the development machine operation task provided by the embodiment of the present application is that the task management server receives the task creation request initiated by the client, and then generates the development machine operation task according to the task creation request. Secondly, the task management server allocates the target image processor GPU required to execute the development machine operation task to the development machine operation task, and sends the development machine operation task request to the master node in the cluster node, and the task request is used to request to execute the development machine operation task on the target GPU. Compared with the prior art, the present application executes the development machine operation task on the GPU by using the Docker container, so that the operating system of the local host can be directly used, thereby improving the utilization rate of the hardware of the physical machine.

[0100] Based on the above embodiment, the following describes how to allocate a target graphics processor GPU required to execute the development machine operation task to the development machine operation task. Figure 4 A flowchart of a method for processing a development machine operation task provided in an embodiment of the present application, the method comprising:

[0101] S301: The task management server receives a task creation request initiated by a client.

[0102] S302: The task management server generates a development machine operation task according to the task creation request.

[0103] The technical terms, technical effects, technical features, and optional implementation methods of S301-S302 can be found in Figure 3 The S201-S202 shown are understood, and the repeated contents will not be repeated here.

[0104] S303: The task management server determines the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions.

[0105] Exemplarily, the task management server may determine the user group to which the development machine operation task belongs based on the user information logged in by the client.

[0106] It should be understood that a user group is not directly bound to a user, that is, a user group will not be granted permissions to a user so that the user included in the user group has permissions. In the present application, the system management module can determine the user's permissions by searching the pre-set entity table and association table. Among them, the entity table may include a permission table, a role table, a user table, and a user group table, and the association table may include a user-user group association table, a role-user group association table, and a permission-role association table.

[0107] S304: The task management server allocates a target GPU required to execute the operation task according to the resource usage rights corresponding to the user group and the resources required by the development machine for the operation task.

[0108] In this step, different user groups have different resource usage permissions, and the task management server can determine the target GPU required for the operation task from the GPUs with resource usage permissions.

[0109] S305. The task management server sends a development machine operation task request to the master node in the cluster node, where the task request is used to request execution of the development machine operation task on the target GPU.

[0110] The technical terms, technical effects, technical features, and optional implementation methods of S305 can be found in Figure 3 As shown in S204, repeated contents will not be described again here.

[0111] Based on the above embodiments, Figure 5 A flowchart of another method for processing a development machine operation task provided in an embodiment of the present application, the method comprising:

[0112] S401: The task management server receives a task creation request initiated by a client.

[0113] S402: The task management server generates a development machine operation task according to the task creation request.

[0114] S403: The task management server determines the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions.

[0115] The technical terms, technical effects, technical features, and optional implementation methods of S401-S402 can be found in Figure 4 It is understood that S301-S303 shown in the figure will not be repeated here for the repeated contents.

[0116] S404: The task management server determines the resource usage quota of the user group to which the development machine operation task belongs.

[0117] The resource usage quota of a user group can be applied for by the user group and then determined after the administrator agrees. After determining the resource usage quota of the user group, each time the user group uses resources, the task management server will deduct the amount of resources used from the resource usage quota of the user group.

[0118] S405: If the resource usage quota of the user group is greater than or equal to the resource amount required by the development machine to perform the operation task, the task management server allocates the target GPU required to perform the operation task.

[0119] In this application, after determining the resource usage quota of the user group to which the development machine operation task belongs, the task management server can compare the resource usage quota required for the development machine operation task with the resource usage quota of the user group. If the resource usage quota of the user group is greater than or equal to the resource amount required for the development machine operation task, the target GPU required to execute the operation task is allocated. If the resource usage quota of the user group is less than the resource amount required for the development machine operation task, an error prompt is issued to the client.

[0120] S406: The task management server deducts the amount of resources required for the development machine to operate the task from the resource usage quota of the user group.

[0121] The processing method of the development machine operation task provided in the embodiment of the present application is that the task management server receives the task creation request initiated by the client, and then generates the development machine operation task according to the task creation request. Secondly, the task management server allocates the target image processor GPU required to execute the development machine operation task to the development machine operation task, and sends the development machine operation task request to the master node in the cluster node, and the task request is used to request to execute the development machine operation task on the target GPU. Compared with the prior art, the present application executes the development machine operation task on the GPU by using the docker container, so that the operating system of the local host can be directly used, thereby improving the utilization rate of the hardware of the physical machine.

[0122] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program information, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0123] Figure 6 The present invention provides a schematic diagram of a device for processing development machine operation tasks in an embodiment of the present invention. The device for processing development machine operation tasks can be implemented by software, hardware, or a combination of both, such as the task management server or a chip in the task management server, for executing the method for processing development machine operation tasks. Figure 6 As shown, the processing device 500 of the development machine operation task includes:

[0124] Receiving module 501, used to receive a task creation request initiated by a client;

[0125] The processing module 502 is used to generate a development machine operation task according to the task creation request; allocate a target graphics processor GPU required to execute the development machine operation task to the development machine operation task;

[0126] The sending module 503 is used to send a development machine operation task request to the master node in the cluster node, where the task request is used to request to execute the development machine operation task on the target GPU.

[0127] In an optional implementation, the processing module 502 is specifically used to determine the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions; according to the resource usage permissions corresponding to the user group and the resources required for the development machine operation task, the target GPU required to execute the operation task is allocated.

[0128] In an optional implementation, processing module 502 is also used to determine the resource usage quota of the user group to which the development machine operation task belongs; if the resource usage quota of the user group is greater than or equal to the amount of resources required for the development machine operation task, the target GPU required to execute the operation task is allocated.

[0129] In an optional implementation, the processing module 502 is further configured to deduct the amount of resources required for the development machine operation task from the resource usage quota of the user group.

[0130] In an optional implementation, the processing module 502 is also used to query the resource utilization of the target GPU by the development machine operation task in the task database; if the resource utilization of the GPU by the development machine operation task is lower than a first threshold, a task release indication is sent to the master node, and the task release indication releases the development machine operation task on the target GPU.

[0131] In an optional implementation, the processing module 502 is also used to query the resource utilization of the target GPU in the task database; if the resource utilization of the target GPU exceeds a second threshold, the target GPU is reallocated to the development machine operation task; and a development machine operation task request is sent to the master node based on the reallocated GPU.

[0132] In an optional implementation, the processing module 502 is further used to update the snapshot of the development machine corresponding to the development machine operation task, where the snapshot is the logical relationship between the data of the development machine.

[0133] In an optional implementation, the processing module 502 is further used to determine a block device required for the development machine operation task, and the block device is used to request storage resources for the development machine operation task.

[0134] In an optional implementation manner, the development machine operation task includes at least one of the following: creating a development machine, deleting a development machine, restarting a development machine, and reinstalling a development machine.

[0135] The processing device for development machine operation tasks provided in the embodiment of the present application can execute the actions on the task management server side in the processing method for development machine operation tasks in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0136] Figure 7 A schematic diagram of the structure of another device for processing development machine operation tasks provided in an embodiment of the present application. The device for processing development machine operation tasks can be implemented by software, hardware, or a combination of both, such as the above-mentioned master node or a chip in the master node, for executing the above-mentioned method for processing development machine operation tasks. Figure 7 As shown, the processing device 600 of the development machine operation task includes:

[0137] The receiving module 601 is used to receive a development machine operation task request sent by the task management server, where the task request is used to request to execute the development machine operation task on the target GPU;

[0138] The processing module 602 is used to determine the target working node according to the running status of multiple working nodes in the cluster node; and schedule the Docker container of the target working node to execute the development machine operation task on the target GPU.

[0139] In an optional implementation, the processing module 602 is further used to monitor the execution progress of the development machine operation task of the target working node and the state of the development machine corresponding to the development machine operation task;

[0140] The device also includes a sending module 603, which is used to send the execution progress of the development machine operation task and the state of the development machine corresponding to the development machine operation task to the task database.

[0141] In an optional implementation, the processing module 602 is further used to monitor the resource utilization of the target GPU by the development machine operation task;

[0142] The sending module 603 is further configured to send the resource utilization of the target GPU to the task database.

[0143] In an optional implementation manner, the development machine operation task includes at least one of the following: creating a development machine, deleting a development machine, restarting a development machine, and reinstalling a development machine.

[0144] The processing device for development machine operation tasks provided in the embodiment of the present application can execute the actions on the main node side in the processing method for development machine operation tasks in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0145] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0146] According to an embodiment of the present application, the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0147] like Figure 8, is a block diagram of an electronic device according to a method for processing a development machine operation task according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0148] like Figure 8 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 701 is taken as an example.

[0149] The memory 702 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the processing method for the development machine operation task provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the processing method for the development machine operation task provided in the present application.

[0150] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the processing method of the development machine operation task in the embodiment of the present application (for example, the attached Figure 5 and Figure 6 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the processing method of the development machine operation task in the above method embodiment is implemented.

[0151] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the processing electronic device of the development machine operation task, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the processing electronic device of the development machine operation task via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The electronic device for developing a method for processing machine operation tasks may also include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0153] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing electronic device for the development machine operation task, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0154] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0157] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0158] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0159] The present application also provides a chip, including a processor and an interface. The interface is used to input and output data or instructions processed by the processor. The processor is used to execute the method provided in the above method embodiment. The chip can be applied to a server.

[0160] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program information, and the program information is used for the above method.

[0161] An embodiment of the present application also provides a program, which, when executed by a processor, is used to execute the method provided in the above method embodiment.

[0162] The embodiment of the present application also provides a program product, such as a computer-readable storage medium, in which instructions are stored. When the program product is run on a computer, the computer executes the method provided by the above method embodiment.

[0163] According to the technical solution of the embodiment of the present application, the problem of low utilization rate of the hardware of the physical machine is solved. Compared with the prior art, the present application uses a docker container to perform development machine operation tasks on a graphics processor GPU, thereby directly utilizing the operating system of the local host, thereby improving the utilization rate of the hardware of the physical machine.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0165] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for processing a development machine operation task, wherein: The method comprises: Receive task creation request initiated by the client; Generate a development machine operation task according to the task creation request, wherein the development machine operation task includes at least one of the following: creating the development machine, deleting the development machine, restarting the development machine, and reinstalling the development machine; Determine the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions; Allocate a target graphics processor GPU required to execute the development machine operation task to the development machine operation task according to the resource usage authority corresponding to the user group to which it belongs and the resources required for the development machine operation task; Sending the development machine operation task request to the master node in the cluster node, wherein the task request is used to request execution of the development machine operation task on the target GPU; the master node is used to determine the target working node according to the running status of multiple working nodes in the cluster node, and schedule the docker container of the target working node to execute the development machine operation task on the target GPU; The method further comprises: Querying the resource utilization rate of the target GPU by the development machine operation task in the task database; If the resource utilization rate of the target GPU by the development machine operation task is lower than a first threshold, a task release instruction is sent to the master node, wherein the task release instruction releases the development machine operation task on the target GPU.

2. The method according to claim 1, after determining the user group to which the development machine operation task belongs, further comprises: Determine the resource usage quota of the user group to which the development machine operation task belongs; The step of allocating a target image processor GPU required for executing the development machine operation task to the development machine operation task includes: If the resource usage quota of the user group is greater than or equal to the resource amount required by the development machine operation task, a target GPU required to execute the development machine operation task is allocated.

3. The method according to claim 2, after allocating the target GPU required to execute the development machine operation task, the method further comprises: The amount of resources required for the development machine operation task is subtracted from the resource usage quota of the user group.

4. The method according to claim 1, wherein: The method further comprises: If the resource utilization rate of the target GPU by the development machine operation task exceeds a second threshold, reallocating the target GPU to the development machine operation task; Sending the development machine operation task request to the master node based on the reallocated GPU.

5. The method according to claim 1, after sending the development machine operation task request to the master node in the cluster node, the method further comprises: The process takes a snapshot of the development machine corresponding to the development machine operation task, where the snapshot is the logical relationship between the data of the development machine.

6. The method according to claim 1, after sending the development machine operation task request to the master node in the cluster node, the method further comprises: Determine a block device required for the development machine operation task, where the block device is used to request storage resources for the development machine operation task.

7. A method for processing a development machine operation task, wherein: The method comprises: Receive a development machine operation task request sent by a task management server, the task request is used to request execution of the development machine operation task on a target graphics processing unit (GPU), and the development machine operation task includes at least one of the following: creating the development machine, deleting the development machine, restarting the development machine, and reinstalling the development machine; the task management server is used to receive the task creation request initiated by the client, generate the development machine operation task according to the task creation request; and allocate a target GPU required for executing the development machine operation task to the development machine operation task according to the resource usage rights corresponding to the user group to which the development machine operation task belongs and the resources required for the development machine operation task; Determine the target working node according to the running status of multiple working nodes in the cluster node; Scheduling the Docker container of the target working node to execute the development machine operation task on the target GPU; Monitoring resource utilization of the target GPU by the development machine operation tasks; The resource utilization of the target GPU is sent to the task database; the task manager is also used to query the resource utilization of the target GPU in the task database, and if the resource utilization of the target GPU is lower than a first threshold, a task release indication is sent to the master node in the cluster node, and the task release indication releases the development machine operation task on the target GPU.

8. The method according to claim 7, after scheduling the Docker container of the target working node to execute the development machine operation task on the target GPU, the method further comprises: Monitor the execution progress of the development machine operation task of the target working node and the status of the development machine corresponding to the development machine operation task; The execution progress of the development machine operation task and the state of the development machine corresponding to the development machine operation task are sent to the task database.

9. A processing device for developing machine operation tasks, wherein: The device comprises: A receiving module, used to receive a task creation request initiated by a client; A processing module, configured to generate a development machine operation task according to the task creation request, wherein the development machine operation task includes at least one of the following: creating the development machine, deleting the development machine, restarting the development machine, and reinstalling the development machine; Determine the user group to which the development machine operation task belongs, and different user groups correspond to different resource usage permissions; Allocate a target graphics processor GPU required to execute the development machine operation task to the development machine operation task according to the resource usage authority corresponding to the user group to which it belongs and the resources required for the development machine operation task; A sending module is used to send the development machine operation task request to the master node in the cluster node, wherein the task request is used to request to execute the development machine operation task on the target GPU; the master node is used to determine the target working node according to the running status of multiple working nodes in the cluster node, and schedule the docker container of the target working node to execute the development machine operation task on the target GPU; The processing module is further used to query the resource utilization rate of the target GPU by the development machine operation task in the task database; If the resource utilization rate of the target GPU by the development machine operation task is lower than a first threshold, a task release instruction is sent to the master node, wherein the task release instruction releases the development machine operation task on the target GPU.

10. According to the device according to claim 9, the processing module is also used to determine the resource usage quota of the user group to which the development machine operation task belongs; if the resource usage quota of the user group is greater than or equal to the amount of resources required by the development machine operation task, then the target GPU required to execute the development machine operation task is allocated.

11. According to the device of claim 10, the processing module is further used to subtract the amount of resources required for the development machine operation task from the resource usage quota of the user group.

12. The device according to claim 9, wherein: The processing module is also used to query the resource utilization of the target GPU in the task database; if the resource utilization of the target GPU exceeds a second threshold, reallocate the target GPU to the development machine operation task; and send the development machine operation task request to the master node based on the reallocated GPU.

13. The device according to claim 9, wherein the processing module is further used to update a snapshot of the development machine corresponding to the development machine operation task, wherein the snapshot is a logical relationship between data of the development machine.

14. The apparatus according to claim 13, wherein the processing module is further used to determine a block device required for the development machine operation task, and the block device is used to request storage resources for the development machine operation task.

15. A processing device for developing machine operation tasks, wherein: The device comprises: A receiving module, used for receiving a development machine operation task request sent by a task management server, wherein the task request is used for requesting execution of the development machine operation task on a target graphics processing unit (GPU), and the development machine operation task includes at least one of the following: creating the development machine, deleting the development machine, restarting the development machine, and reinstalling the development machine; the task management server is used for receiving a task creation request initiated by a client, generating the development machine operation task according to the task creation request; and allocating a target GPU required for executing the development machine operation task to the development machine operation task according to the resource usage rights corresponding to the user group to which the development machine operation task belongs and the resources required for the development machine operation task; A processing module is used to determine a target working node according to the running status of multiple working nodes in the cluster node; and schedule the Docker container of the target working node to execute the development machine operation task on the target GPU; Monitoring resource utilization of the target GPU by the development machine operation tasks; The sending module is also used to send the resource utilization of the target GPU to the task database; the task manager is also used to query the resource utilization of the target GPU in the task database, and if the resource utilization of the target GPU is lower than the first threshold, send a task release indication to the master node in the cluster node, and the task release indication releases the development machine operation task on the target GPU.

16. The device according to claim 15, wherein the processing module is further used to monitor the execution progress of the development machine operation task of the target working node and the state of the development machine corresponding to the development machine operation task; The device also includes a sending module, which is used to send the execution progress of the development machine operation task and the state of the development machine corresponding to the development machine operation task to the task database.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images