Quick application switching method and system applied to GPU (Graphics Processing Unit) computing power scheduling

Abstract

The invention relates to an application rapid switching method and system applied to GPU computing power scheduling, and the method comprises the steps: dividing a task, running on a GPU, of an AI program into a plurality of execution subtasks, and dividing a task preparation stage into a first preparation stage and a second preparation stage based on a preset preparation rule; in the task preparation stage process of the current task, running the first preparation stage of the next task; when the task execution stage of the current task reaches a preset process, running a second preparation stage of the first execution subtask in the next task; after the operation of the second preparation stage is finished, the first execution subtask is operated; in the running process of the first execution subtask, running the second preparation stage of the next execution subtask, and running the task cleaning stage at the same time; and after all the execution sub-tasks are operated, the task cleaning stage is operated. The method has the advantages that the task switching overhead of the AI program running on the GPU is reduced, and the utilization rate of the GPU is increased.

Description

Technical Field

[0001] The present application relates to the technical field of GPUs, and more particularly to a method and system for fast switching applications for GPU computing power scheduling. Background Art

[0002] Currently, for the main workloads of AI applications, including throughput-intensive training tasks and latency-sensitive inference tasks of AI programs, the mainstream approach is to provide dedicated GPU clusters for training and inference respectively. However, in large model architectures, both inference tasks and training tasks are difficult to serve using training clusters.

[0003] Reference Figure 1 ,The task running of AI programs includes three stages: task preparation, task execution, and task cleaning.,Since the GPU has a high overhead when switching between new and old tasks, the current switching method is at the second level, and most inference tasks have strict service,indicators (SLOs) that need to be controlled within the range of tens to hundreds of milliseconds.,Thus, the switching efficiency is low, and it is difficult to pack multiple deep learning (DL) programs onto the same GPU server.

[0004] The existing solution is to allow multiple different models to share GPU memory in space, including Figure 2 NVIDIA Multi-Process Sharing (MPS) is shown in the figure. The MPS method is to prepare the tasks of each model in advance, that is, both new and old tasks are prepared first, and the switching speed is fast. However, the problem is that multiple models occupy a large video memory space, resulting in low GPU utilization, so it needs to be improved. Summary of the invention

[0005] In order to reduce the task switching overhead of AI programs running on the GPU and improve the utilization of the GPU, the present application provides a method and system for fast application switching for GPU computing power scheduling.

[0006] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A method for fast switching applications for GPU computing power scheduling, comprising the steps of: Divide a task of an AI program running on a GPU into several execution subtasks, and divide the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; Based on the preset operation rules, during the task preparation phase of the current task, the first preparation phase of the next task is executed; When the task execution phase of the current task reaches a preset progress, the second preparation phase of the first execution subtask in the next task is executed; When the second preparation phase of the first execution subtask in the next task is completed, the first execution subtask is executed; During the running of the first execution subtask, the second preparation phase of the next execution subtask is run, and at the same time, the task cleaning phase is run to clear the tasks that have been completed; When all execution subtasks have finished running, the task cleaning phase is run.

[0007] By adopting the above technical solution, in order to effectively time-share the GPU and minimize the overhead of task switching, the tasks running on the GPU of the AI ​​program are divided into multiple execution subtasks, and the task preparation stage is divided. During the preparation of the current task, the first preparation stage of the next task is started, that is, when the old task starts, the preparation of part of the new task is started first, and the second preparation stage of the execution subtask is run simultaneously with the execution of the previous task, and the task cleaning stage is run after the execution subtask is run, so as to keep enough video memory on the GPU to prepare and run each execution subtask, and also improve the utilization rate. Therefore, the above-mentioned pipelined task context switching setting unifies the video memory management and the operation of the main and standby working molecular tasks, greatly reduces the switching overhead, and improves the utilization rate of the GPU.

[0008] Optionally, the task preparation phase is divided into a first preparation phase and a second preparation phase based on a preset preparation rule, including: The first preparation phase includes an environment initialization phase, which is used for process startup and loading of the PyTorch CUDA runtime and CUDA context initialization; The second preparation stage includes a video memory allocation stage and a data transmission stage, wherein the video memory allocation stage is used to apply for matching video memory for the execution subtask in the AI ​​program, and the AI ​​program is stored in a preset host memory; The data transmission stage is used to copy the model data of the AI ​​program model and the tasks that need to be run on the GPU from the host memory to the video memory.

[0009] By adopting the above technical solution, since only the runtime and CUDA context initialization are performed in the environment initialization stage, the video memory occupied by the operation is small, and multiple tasks or subtasks can be initialized, thereby improving GPU utilization.

[0010] Optionally, the first preparation phase of the next task is run during the task preparation phase of the current task based on a preset operation rule; including: Based on the preset running rules, when the current task starts running the environment initialization phase, the environment initialization phase of the next task is run synchronously; When the video memory occupied by all tasks in the synchronous running environment initialization phase reaches a first preset value, the environment initialization phase of running new tasks is suspended; After the task cleaning phase is executed, when the video memory occupied by all tasks in the synchronously executed environment initialization phase is less than the first preset value again, the environment initialization phase of the next task is executed again.

[0011] By adopting the above technical solution, considering the influence of video memory size, the advance preparation of the environment initialization phase of each task also needs to be controlled, so a first preset value is set. When the video memory occupied by the task for environment initialization preparation or the execution of the subtask reaches the first preset value, the advance preparation of the environment initialization phase is first suspended, and the advance preparation of the environment initialization is performed again after the video memory is released in the task cleaning phase. This is the time to reduce the switching overhead while maintaining high GPU utilization.

[0012] Optionally, when the task execution phase of the current task reaches a preset progress, the second preparation phase of the first execution subtask in the next task is run; including: Get the number of subtasks executed in the current task, and identify the number of subtasks that have not been executed; When the quantity information is less than the preset quantity, the corresponding video memory application information of all the execution subtasks that have not been run is obtained; Based on the video memory application information, when the video memory applied by any of the unexecuted execution subtasks is greater than a second preset value, marking the unexecuted execution subtask; When the marked execution subtask is completed, the second preparation phase of the first execution subtask in the next task is executed.

[0013] By adopting the above technical solution, in order to meet the low-overhead task switching while maintaining high GPU utilization, for the operation of the second preparation stage, that is, the operation settings of the video memory allocation stage and the data transmission stage, it is necessary to identify the number and video memory of the currently unfinished execution subtasks. If there are still many currently unrunning execution subtasks, the second preparation stage of the execution subtask in the next task will not be run first. After the currently unrunning execution subtasks are less than the preset number, it is still necessary to identify the video memory of the unrunning execution subtasks. If its video memory is greater than the second preset value, the execution subtask is marked, and it is determined that there is still a risk of high switching overhead. Therefore, after waiting for the marked execution subtask to complete, the second preparation stage of the next execution subtask is run, which can reduce the context switching overhead.

[0014] Optionally, after the second preparation phase of the first execution subtask in the next task is completed, the first execution subtask is executed; comprising: When the second preparation phase of the first execution subtask in the next task is finished, determining whether the task execution phase of the current task is finished; If the task execution phase of the current task is completed, the first execution subtask is immediately run; If the task execution phase of the current task has not ended, determine whether the sum of the video memory requested by the currently running execution subtask and the first execution subtask to be run exceeds a second preset value. If so, wait until the task execution phase of the current task ends and run the first execution subtask.

[0015] By adopting the above technical solution, after the second preparation stage of the execution subtask is completed, that is, after the video memory application and data copying are completed, it is still necessary to determine whether the current execution subtask can be run immediately to reduce the context switching overhead. Therefore, after the second preparation stage is completed, it is necessary to determine whether the video memory of the remaining unfinished execution subtasks and the execution subtask to be run plus the video memory of the execution subtask to be run exceeds the second preset value, and selectively determine whether the next execution subtask should be run immediately or delayed, so as to effectively reduce the context switching overhead in each running link.

[0016] Optionally, the video memory allocation stage is used to apply for matching video memory for the execution subtask in the AI ​​program, including: Obtain all execution subtasks in the AI ​​program to be executed, and input the execution subtasks into the pre-trained video memory matching model in sequence; When the video memory matching model receives the execution subtask, it identifies the type information and task division information of the task to which the execution subtask belongs; Based on the type information and the task division information, a corresponding video memory application value is matched for the currently executed subtask; and when the video memory allocation phase of the currently executed subtask is running, a video memory application instruction is generated and sent based on the video memory application value.

[0017] By adopting the above technical solution, in order to improve the efficiency of the video memory application stage of each execution subtask in the AI ​​program, the type of task and the task division are identified through the preset video memory matching model, so that each task to be run or execution subtask can be accurately matched to the required video memory size in advance, prompting the efficiency of the second preparation stage.

[0018] Optionally, the matching of a corresponding video memory application value for a currently executed subtask based on the type information and the task division information includes: Based on the type information, a video memory interval corresponding to the type of the task is selected from a preset database of a video memory matching model; The task division number of the task to which the currently executed subtask belongs is identified based on the task division information, and a video memory matching the task division number is selected from the video memory interval as the video memory application value of the currently executed subtask.

[0019] By adopting the above technical solution, in order to further improve the efficiency of video memory matching, the task type information can be used to quickly determine the approximate video memory matching range, and then the task division information can be used to identify how many execution subtasks the current task is divided into, so that the required video memory application value can be quickly matched within the filtered video memory range.

[0020] The second object of the invention is achieved by the following technical solutions: An application fast switching system for GPU computing power scheduling, comprising: A division module, used to divide a task of the AI ​​program running on the GPU into a plurality of execution subtasks, and divide the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; A first operation module, used for executing a first preparation phase of a next task during the task preparation phase of a current task based on a preset operation rule; A second running module, used for running the second preparation phase of the first execution subtask in the next task when the task execution phase of the current task reaches a preset process; A third running module is used to run the first execution subtask in the next task after the second preparation phase of the first execution subtask is completed; A fourth running module is used to run the second preparation phase of the next execution subtask during the execution of the first execution subtask, and to run the task cleaning phase to clear the tasks that have been completed; The fifth running module is used to run the task cleaning phase after all execution subtasks are completed.

[0021] The third objective of the present application is achieved through the following technical solutions: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for rapid application switching applied to GPU computing power scheduling are implemented.

[0022] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for fast switching of applications applied to GPU computing power scheduling.

[0023] In summary, the present application includes at least the following beneficial technical effects: 1. During the preparation of the current task, that is, when the first preparation phase of the next task begins, that is, when the old task starts, the preparation of part of the new task begins first, and the second preparation phase of the execution subtask is run simultaneously with the execution of the previous task, and the task cleaning phase is run after the execution subtask is run, so as to keep enough video memory on the GPU to prepare and run each execution subtask, and at the same time improve the utilization rate; therefore, the above-mentioned pipelined task context switching setting unifies the video memory management and the operation of the main and standby working subtasks, greatly reduces the switching overhead, and improves the utilization rate of the GPU; 2. Since only the runtime and CUDA context initialization are performed during the environment initialization phase, the video memory occupied by the operation is small, and multiple tasks or subtasks can be initialized, thereby improving GPU utilization; 3. Considering the impact of video memory size, the amount of advance preparation for the environment initialization phase of each task also needs to be controlled. Therefore, a first preset value is set. When the video memory occupied by the task for environment initialization preparation or the execution of the subtask reaches the first preset value, the advance preparation for the environment initialization phase is suspended first, and the advance preparation for environment initialization is performed again after the video memory is released in the task cleaning phase. This is the time to reduce the switching overhead while maintaining high GPU utilization; 4. In order to improve the efficiency of the video memory application stage of each execution subtask in the AI ​​program, the type of task and the task division are identified through the preset video memory matching model, which can accurately match the required video memory size for each task or subtask to be run in advance, prompting the efficiency of the second preparation stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a prior art figure of the background technology of a method for fast switching of applications applied to GPU computing power scheduling in the present application; Figure 2 This is a schematic diagram of MPS switching in the background technology of a method for fast switching of applications for GPU computing power scheduling in this application; Figure 3 This is a flowchart of an implementation of an embodiment of a method for fast switching of applications applied to GPU computing power scheduling in the present application; Figure 4 This is a schematic diagram of a task execution phase in an embodiment of a method for fast switching of applications applied to GPU computing power scheduling of the present application; Figure 5 This is a task switching diagram of an embodiment of a method for fast switching of applications applied to GPU computing power scheduling of the present application; Figure 6 This is a schematic diagram comparing switching modes of an embodiment of a method for fast switching applications for GPU computing power scheduling in the present application; Figure 7 This is a throughput comparison diagram of an embodiment of an application fast switching method for GPU computing power scheduling in the present application. DETAILED DESCRIPTION

[0025] The most popular workloads include throughput-intensive training tasks and latency-sensitive inference tasks. The current mainstream approach is to provide dedicated GPU clusters for training and inference respectively. The main reason is that in large model architectures, inference tasks cannot be served by training clusters, and when the inference load is low, training tasks cannot use inference clusters.

[0026] Ideally, multiple DL applications should be able to be packed onto the same GPU server, maximizing GPU utilization through time-sharing. This is exactly how operating systems achieve high CPU utilization through task scheduling and context switching. However, this is not possible on GPUs because CPU switching is at the microsecond level, while GPUs have high overhead when switching between tasks, and the current switching method is at the second level. Current inference has strict SLOs in the range of tens to hundreds of milliseconds, so it cannot be packed onto the same GPU server.

[0027] The existing solution is to allow different models to share the GPU memory in space, such as NVIDIA's multi-process sharing (MPS). Tasks are prepared in advance and the switching speed is fast enough. However, the problem is that multiple models occupy a large amount of memory space, resulting in low GPU utilization.

[0028] The following is combined with Figure 3-7 This application is described in further detail.

[0029] In one embodiment, if Figure 3 As shown, the present application discloses a method for fast switching of applications applied to GPU computing power scheduling, which specifically includes the following steps: S10: dividing a task of the AI ​​program running on the GPU into a plurality of execution subtasks, and dividing the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; In this embodiment, the AI ​​program model is stored in the host memory, which is larger and more economical than the GPU memory, and the GPU can quickly switch between contexts. Figure 4 , which is a schematic diagram of task execution in an AI program. It is a hierarchical execution method, including input, encoding encoder, decoding decoder and output steps, so the task can be split.

[0030] The first preparation phase includes the environment initialization phase, which is used to start the process and load the PyTorchCUDA runtime and initialize the CUDA context. The environment initialization only performs runtime loading and context initialization, and occupies very little video memory, so that multiple tasks or subtasks can be initialized.

[0031] The second preparation phase includes a video memory allocation phase and a data transmission phase. The video memory allocation phase is used to apply for matching video memory for the execution subtasks in the AI ​​program. The data transfer stage is used to copy the model data of the AI ​​program model and the tasks that need to be run on the GPU from the host memory to the video memory.

[0032] Furthermore, when applying for matching video memory for executing subtasks in the AI ​​program, the steps include: S11: Obtain all execution subtasks in the AI ​​program to be executed, and input the execution subtasks into the pre-trained video memory matching model in sequence; S12: When the video memory matching model receives the execution subtask, it identifies the type information and task division information of the task to which the execution subtask belongs; S13: matching a corresponding video memory application value for the currently executing subtask based on the type information and the task division information; and generating and sending a video memory application instruction based on the video memory application value when the video memory allocation phase of the currently executing subtask is running.

[0033] In this embodiment, the video memory matching model is a model pre-trained through a neural network for matching video memories for different execution subtasks. Different types of information represent tasks with different video memory requirements. Task division information refers to the number of execution subtasks into which the task is divided.

[0034] The video memory request value refers to the video memory value that is sufficient to execute the subtask.

[0035] Wherein, step S13 comprises: S131: based on the type information, filter out a video memory interval corresponding to the type of the task from a preset database of a video memory matching model; S132: Identify the task division number of the task to which the currently executed subtask belongs based on the task division information, and select a video memory matching the task division number from the video memory interval as the video memory application value of the currently executed subtask.

[0036] Specifically, based on the type information of the task, the video memory interval for the task to run is screened out from the preset database of the video memory model, and the video memory interval includes the video memory values ​​corresponding to the different division quantities of the task. Then, by identifying the number of execution subtasks into which the task is divided, the video memory value that meets the division quantity is selected from the video memory interval as the video memory request value for the execution subtask.

[0037] Furthermore, there are still differences in the video memory values ​​required by different execution subtasks in the same task. By using the field feature information of the execution subtasks, the video memory application values ​​of different execution subtasks in the same task can be further accurately matched.

[0038] S20: Based on a preset operation rule, during the task preparation phase of the current task, the first preparation phase of the next task is executed; Specifically, the steps include: S21: Based on the preset operation rules, when the current task starts to run the environment initialization phase, the environment initialization phase of the next task is synchronously run; S22: When the video memory occupied by all tasks in the synchronous running environment initialization phase reaches a first preset value, pausing the running of the environment initialization phase of new tasks; S23: After the task cleaning phase is executed, when the video memory occupied by all tasks in the synchronous execution environment initialization phase is less than the first preset value again, the environment initialization phase of the next task is executed again.

[0039] In this embodiment, the first preset value is a custom setting. The task cleaning phase is used to delete the environment and the data of the executed subtasks that have been run, and release the video memory.

[0040] S30: When the task execution phase of the current task reaches a preset progress, the second preparation phase of the first execution subtask in the next task is executed; Specifically, the steps include: S31: Obtain the number of executed subtasks in the current task, and identify the number of executed subtasks that have not been run; S32: when the quantity information is less than the preset quantity, obtaining the video memory application information corresponding to each of the execution subtasks that have not been executed; S33: Based on the video memory application information, when the video memory applied by any of the unrunning execution subtasks is greater than a second preset value, the unrunning execution subtask is marked; S34: When the marked execution subtask is completed, the second preparation phase of the first execution subtask in the next task is executed.

[0041] In this embodiment, the number of unexecuted subtasks in the current task and their corresponding memory application values, ie, memory application information, can be obtained from the memory application model; the preset number and the second preset value can be customized.

[0042] When the marked execution subtasks include two or more, the second preparation phase of the first execution subtask in the next task will be executed only after all the execution subtasks are completed.

[0043] S40: after the second preparation phase of the first execution subtask in the next task is completed, the first execution subtask is executed; Specifically, the steps include: S41: when the second preparation phase of the first execution subtask in the next task is finished, determining whether the task execution phase of the current task is finished; S42: If the task execution phase of the current task is completed, immediately run the first execution subtask; S43: If the task execution phase of the current task has not ended, determine whether the sum of the video memory requested by the currently running execution subtask and the first execution subtask to be run exceeds a second preset value. If so, wait until the task execution phase of the current task ends and run the first execution subtask.

[0044] In this embodiment, when the task execution phase of the current task has not yet ended, there is no marked self-subtask, so only the unmarked execution subtask is still in the running process. It is necessary to consider whether the sum of the video memory value of the current execution subtask and the video memory value of the first execution subtask to be run exceeds the second preset value. To determine whether two execution subtasks can be run at the same time, to minimize the context switching overhead.

[0045] S50: During the execution of the first execution subtask, the second preparation phase of the next execution subtask is executed, and at the same time, a task cleaning phase is executed to clear the tasks that have been completed; In this embodiment, the operation switching between different execution subtasks in the same task also needs to consider the second preset value.

[0046] S60: After all execution subtasks are completed, the task cleaning phase is executed.

[0047] In one embodiment, after all execution subtasks in the same task are completed, the task cleaning phase will clean the environment information of the previous task. If the environment information of the execution subtasks in the same task is different, the environment information is cleared immediately after the execution of one execution subtask is completed.

[0048] In one embodiment, referring to Figure 5 As shown, the task is divided into three execution subtasks G1, G2, and G3. When the old task is in the preparation stage, the first preparation stage of the new task begins, namely, task preparation 1; when the task execution stage of the old task reaches the preset process, the second preparation stage of the first execution subtask G1 in the new task begins, namely, task preparation 2; when the execution stage of the old task ends, the execution subtask G1 is run, namely, task execution 1, and the second preparation stage of the execution subtask G2 is carried out at the same time, namely, task preparation 2; further, the execution task G2 is run, and the second preparation stage of the execution subtask G3 and the task cleaning stage are carried out at the same time.

[0049] refer to Figure 6 , the time consumption of switching modes such as NVIDIA MPS switching, hard switching, and pipeline switching of this application is compared with the ready model on three general models, among which the throughput comparison of the three switching modes is as follows Figure 7 As shown. Figure 7 It can be seen that after adopting the pipeline switching mode, the time for switching AI programs on the GPU is greatly shortened, and the throughput can be maintained at a high level. Existing methods such as the hard switching mode pause for more than 6 seconds, such as the long switching time of MPS, and occupy too much video memory.

[0050] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0051] In one embodiment, a system for fast switching applications for GPU computing power scheduling is provided, and the method for fast switching applications for GPU computing power scheduling corresponds one-to-one to the system for fast switching applications for GPU computing power scheduling in the above embodiment. The system for fast switching applications for GPU computing power scheduling includes: A division module, used to divide a task of the AI ​​program running on the GPU into a plurality of execution subtasks, and divide the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; A first operation module, used for executing a first preparation phase of a next task during the task preparation phase of a current task based on a preset operation rule; A second running module, used for running the second preparation phase of the first execution subtask in the next task when the task execution phase of the current task reaches a preset process; A third running module is used to run the first execution subtask in the next task after the second preparation phase of the first execution subtask is completed; A fourth running module is used to run the second preparation phase of the next execution subtask during the execution of the first execution subtask, and to run the task cleaning phase to clear the tasks that have been completed; The fifth running module is used to run the task cleaning phase after all execution subtasks are completed.

[0052] For the specific definition of an application fast switching system applied to GPU computing power scheduling, please refer to the definition of an application fast switching system applied to GPU computing power scheduling above, which will not be repeated here. Each module in the above-mentioned application fast switching system applied to GPU computing power scheduling can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0053] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for fast switching of applications applied to GPU computing power scheduling is implemented.

[0054] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for fast switching of applications for GPU computing power scheduling is implemented.

[0055] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for fast switching of applications applied to GPU computing power scheduling is implemented.

[0056] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0057] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0058] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for fast application switching for GPU computing power scheduling, characterized in that: Divide a task of an AI program running on a GPU into several execution subtasks, and divide the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; Based on the preset operation rules, during the task preparation phase of the current task, the first preparation phase of the next task is executed; When the task execution phase of the current task reaches a preset progress, the second preparation phase of the first execution subtask in the next task is executed; When the second preparation phase of the first execution subtask in the next task is completed, the first execution subtask is executed; During the running of the first execution subtask, the second preparation phase of the next execution subtask is run, and at the same time, the task cleaning phase is run to clear the tasks that have been completed; When all execution subtasks have finished running, the task cleaning phase is run.

2. The method for fast application switching for GPU computing power scheduling according to claim 1, characterized in that: The task preparation phase is divided into a first preparation phase and a second preparation phase based on the preset preparation rules, including: The first preparation phase includes an environment initialization phase, which is used for process startup and loading of the PyTorch CUDA runtime and CUDA context initialization; The second preparation stage includes a video memory allocation stage and a data transmission stage, wherein the video memory allocation stage is used to apply for matching video memory for the execution subtask in the AI ​​program, and the AI ​​program is stored in a preset host memory; The data transmission stage is used to copy the model data of the AI ​​program model and the tasks that need to be run on the GPU from the host memory to the video memory.

3. The method for fast application switching for GPU computing power scheduling according to claim 2, characterized in that: Based on the preset operation rules, during the task preparation phase of the current task, the first preparation phase of the next task is executed; include: Based on the preset running rules, when the current task starts running the environment initialization phase, the environment initialization phase of the next task is run synchronously; When the video memory occupied by all tasks in the synchronous running environment initialization phase reaches a first preset value, the environment initialization phase of running new tasks is suspended; After the task cleaning phase is executed, when the video memory occupied by all tasks in the synchronously executed environment initialization phase is less than the first preset value again, the environment initialization phase of the next task is executed again.

4. The method for fast application switching for GPU computing power scheduling according to claim 1, characterized in that: When the task execution phase of the current task reaches a preset progress, the second preparation phase of the first execution subtask in the next task is run; comprising: Get the number of subtasks executed in the current task, and identify the number of subtasks that have not been executed; When the quantity information is less than the preset quantity, the corresponding video memory application information of all the execution subtasks that have not been run is obtained; Based on the video memory application information, when the video memory applied by any of the unexecuted execution subtasks is greater than a second preset value, marking the unexecuted execution subtask; When the marked execution subtask is completed, the second preparation phase of the first execution subtask in the next task is executed.

5. The method for fast application switching for GPU computing power scheduling according to claim 4, characterized in that: After the second preparation phase of the first execution subtask in the next task is completed, the first execution subtask is executed; comprising: When the second preparation phase of the first execution subtask in the next task is finished, determining whether the task execution phase of the current task is finished; If the task execution phase of the current task is completed, the first execution subtask is immediately run; If the task execution phase of the current task has not ended, determine whether the sum of the video memory requested by the currently running execution subtask and the first execution subtask to be run exceeds a second preset value. If so, wait until the task execution phase of the current task ends and run the first execution subtask.

6. The method for fast application switching for GPU computing power scheduling according to claim 2, characterized in that: The video memory allocation stage is used to apply for matching video memory for the execution subtask in the AI ​​program, including: Obtain all execution subtasks in the AI ​​program to be executed, and input the execution subtasks into the pre-trained video memory matching model in sequence; When the video memory matching model receives the execution subtask, it identifies the type information and task division information of the task to which the execution subtask belongs; Based on the type information and the task division information, a corresponding video memory application value is matched for the currently executed subtask; and when the video memory allocation phase of the currently executed subtask is running, a video memory application instruction is generated and sent based on the video memory application value.

7. The method for fast application switching for GPU computing power scheduling according to claim 6, characterized in that: The method of matching the corresponding video memory application value for the currently executed subtask based on the type information and the task division information includes: Based on the type information, a video memory interval corresponding to the type of the task is selected from a preset database of a video memory matching model; The task division number of the task to which the currently executed subtask belongs is identified based on the task division information, and a video memory matching the task division number is selected from the video memory interval as the video memory application value of the currently executed subtask.

8. A fast application switching system for GPU computing power scheduling, characterized by: A division module, used to divide a task of the AI ​​program running on the GPU into a plurality of execution subtasks, and divide the task preparation phase into a first preparation phase and a second preparation phase based on a preset preparation rule; A first operation module, used for executing a first preparation phase of a next task during the task preparation phase of a current task based on a preset operation rule; A second running module, used for running the second preparation phase of the first execution subtask in the next task when the task execution phase of the current task reaches a preset process; A third running module is used to run the first execution subtask in the next task after the second preparation phase of the first execution subtask is completed; A fourth running module is used to run the second preparation phase of the next execution subtask during the execution of the first execution subtask, and to run the task cleaning phase to clear the tasks that have been completed; The fifth running module is used to run the task cleaning phase after all execution subtasks are completed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a method for fast switching of applications applied to GPU computing power scheduling as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of a method for fast switching of applications applied to GPU computing power scheduling as described in any one of claims 1 to 7 are implemented.

Images