Task allocation method of GPU equipment and related device thereof
By selecting the appropriate GPU device based on the parameters of the inference task and assigning tasks, the problem of idle and waste of computing power of GPU devices is solved, and the processing efficiency of AI tasks is improved.
Patent Information
- Application Number
- CN202411951769.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-29
AI Technical Summary
In the prior art, the computing power scheduling strategy for GPU devices is not perfect enough, resulting in idle and wasted computing power of GPU devices, and the processing speed of AI tasks is slow.
By obtaining the minimum number of devices of the required GPU devices based on the parameters of the inference task, selecting preselected GPU devices, and inputting the task parameters and parameters of the preselected GPU devices into the prediction model to obtain the expected response time, and finally selecting the target GPU device based on the expected and target response time for task allocation.
The computing power resources of GPU equipment are reasonably allocated, the processing efficiency of computing power tasks is improved, and the waste of computing power resources is reduced.
Smart Images
Figure CN119902881A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a task allocation method for a GPU device and related devices thereof. Background Art
[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.
[0003] At present, countries around the world are accelerating their layout of artificial intelligence and regard it as one of the strategic technologies. Artificial intelligence applications will penetrate into various industries. GPU (Graphics Processing Unit) as the computing power infrastructure of artificial intelligence will inevitably play a key and decisive role.
[0004] In related technologies, AI tasks are generally accelerated by building GPU clusters for inference. Some AI tasks need to be deployed on multiple GPU devices. However, the current computing power scheduling strategy for GPU devices is not perfect, which can easily cause the computing power of GPU devices to be idle and wasted, and can also lead to slow processing of AI tasks. Summary of the invention
[0005] In order to solve or partially solve the problems existing in the related technology, the present application provides a task allocation method for a GPU device and a related device, which can reasonably allocate the computing power resources of the GPU device, improve the processing efficiency of computing power tasks, and reduce the waste of computing power resources.
[0006] In a first aspect, the present application provides a method for assigning tasks to a GPU device, comprising obtaining a minimum number of required GPU devices according to a first parameter of an inference task, wherein the first parameter at least includes a target video memory value of the inference task; obtaining a preselected GPU device according to the minimum number of devices and a current video memory remaining value of the GPU device; inputting the first parameter of the inference task and a second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; obtaining a target response time of the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; and assigning the inference task to the target GPU device for processing.
[0007] In combination with the first aspect, in a possible implementation of the first aspect, obtaining the minimum number of required GPU devices based on the first parameter of the reasoning task includes inputting the first parameter of the reasoning task into a second prediction model to obtain a target video memory value of the reasoning task; and obtaining the minimum number of GPU devices based on the target video memory value of the reasoning task and the maximum video memory value of the GPU device.
[0008] In combination with the first aspect, in a possible implementation of the first aspect, the preselected GPU device is obtained based on the minimum number of devices and the current remaining video memory value of the GPU device, including: if the minimum number of devices is equal to 1, then the GPU device whose current remaining video memory value is greater than the target video memory value is used as the preselected GPU device; if the minimum number of devices is greater than 1, then according to the target video memory value of the inference task and the current remaining video memory value of each of the GPU devices, a plurality of GPU device combinations are obtained, and the GPU devices in the GPU device combinations are used as the preselected GPU devices, wherein the sum of the current remaining video memory values of the preselected GPU devices in the GPU device combination is greater than the target video memory value of the inference task.
[0009] In combination with the first aspect, in a possible implementation of the first aspect, the first parameter of the reasoning task and the second parameter of the preselected GPU device are input into the first prediction model to obtain the expected response time of the reasoning task, including: if the minimum number of devices is equal to 1, according to the running status of the preselected GPU device, the first parameter of the reasoning task and the second parameter of the preselected GPU device are input into the first prediction model to obtain the expected response time of the reasoning task; if the minimum number of devices is greater than 1, the first parameter of the reasoning task and the second parameter of each preselected GPU device in the GPU device combination are input into the first prediction model to obtain the expected response time of the reasoning task.
[0010] In combination with the first aspect, in a possible implementation of the first aspect, the target response time of the inference task is obtained, and at least one target GPU device is obtained from the preselected GPU devices based on the expected response time and the target response time, including: obtaining the preset GPU device or the GPU device combination when the expected response time is less than the target response time; calculating the load capacity of each preselected GPU device or the load capacity of each GPU device combination, and taking the preselected GPU device whose load capacity is greater than a preset threshold or the preselected GPU device in the GPU device combination as the target GPU device.
[0011] In combination with the first aspect, in a possible implementation of the first aspect, obtaining the target response time of the reasoning task includes: obtaining the target response time of the reasoning task from a database; and / or inputting the first parameter of the reasoning task into a third prediction model to obtain the target response time.
[0012] According to a second aspect of the present application, there is provided a task allocation device for a GPU device, comprising a first acquisition module, for acquiring a minimum number of required GPU devices according to a first parameter of an inference task, wherein the first parameter at least includes a target video memory value of the inference task; a second acquisition module, for obtaining a preselected GPU device according to the minimum number of devices and a current video memory remaining value of the GPU device; a calculation module, for inputting the first parameter of the inference task and the second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; a processing module, for acquiring a target response time of the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; and an allocation module, for allocating the inference task to the target GPU device for processing.
[0013] In combination with the second aspect, in a possible implementation of the second aspect, the first acquisition module is also used to input the first parameter of the reasoning task into the second prediction model to obtain the target video memory value of the reasoning task; based on the target video memory value of the reasoning task and the maximum video memory value of the GPU device, the minimum device quantity of the GPU device is obtained.
[0014] In combination with the second aspect, in a possible implementation of the second aspect, the second acquisition module is also used to, if the minimum number of devices is equal to 1, use the GPU device whose current video memory remaining value is greater than the target video memory value as the preselected GPU device; if the minimum number of devices is greater than 1, obtain several GPU device combinations based on the target video memory value of the reasoning task and the current video memory remaining value of each of the GPU devices, and use the GPU devices in the GPU device combinations as the preselected GPU devices, wherein the sum of the current video memory remaining values of the preselected GPU devices in the GPU device combination is greater than the target video memory value of the reasoning task.
[0015] In combination with the second aspect, in a possible implementation of the second aspect, the computing module is also used to, if the minimum number of devices is equal to 1, input the first parameter of the reasoning task and the second parameter of the preselected GPU device into the first prediction model according to the running status of the preselected GPU device to obtain the expected response time of the reasoning task; if the minimum number of devices is greater than 1, input the first parameter of the reasoning task and the second parameter of each preselected GPU device in the GPU device combination into the first prediction model to obtain the expected response time of the reasoning task.
[0016] In combination with the second aspect, in a possible implementation of the second aspect, the processing module is also used to obtain the preset GPU device or the GPU device combination when the expected response time is less than the target response time; calculate the load capacity of each of the preselected GPU devices or the load capacity of each of the GPU device combinations, and take the preselected GPU device whose load capacity is greater than a preset threshold or the preselected GPU device in the GPU device combination as the target GPU device.
[0017] In combination with the second aspect, in a possible implementation of the second aspect, the processing module is also used to obtain the target response time of the reasoning task from a database; and / or, input the first parameter of the reasoning task into a third prediction model to obtain the target response time.
[0018] A third aspect of the present application provides an electronic device, including:
[0019] Processor; and
[0020] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.
[0021] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.
[0022] A fifth aspect of the present application provides a computer program product, comprising: a computer program / instruction that implements the above-mentioned method when executed by a processor.
[0023] The technical solution provided by this application may have the following beneficial effects:
[0024] The present application provides a method for allocating tasks to a GPU device and a related device thereof, comprising obtaining a minimum number of required GPU devices according to a first parameter of an inference task, wherein the first parameter at least includes a target video memory value of the inference task; obtaining a preselected GPU device according to the minimum number of devices and a current video memory remaining value of the GPU device; inputting the first parameter of the inference task and a second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; obtaining a target response time of the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; and allocating the inference task to the target GPU device for processing, thereby being able to reasonably allocate computing power resources of the GPU device, improve the processing efficiency of computing power tasks, and reduce the waste of computing power resources.
[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.
[0027] Figure 1 It is a flowchart of a method for allocating tasks to a GPU device according to an embodiment of the present application;
[0028] Figure 2 is a structural diagram of a task allocation device for a GPU device shown in an embodiment of the present application;
[0029] Figure 3 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION
[0030] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.
[0031] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "an", and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.
[0032] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0033] At present, countries around the world are accelerating their layout of artificial intelligence and regard it as one of the strategic technologies. Artificial intelligence applications will penetrate into various industries. GPU (Graphics Processing Unit) as the computing power infrastructure of artificial intelligence will inevitably play a key and decisive role.
[0034] As AI technology continues to be used in enterprise business applications, enterprises are accelerating the reasoning of AI tasks by building GPU clusters to meet the demand for computing resources for a large number of natural language processing and video image algorithm tasks. Some AI tasks need to be deployed on multiple GPU cards to solve the problem of insufficient video memory on a single GPU card and the inability to deploy large AI models.
[0035] However, the relevant technologies lack a reasonable allocation strategy for the GPU resources of GPU devices, which easily leads to idleness and waste of computing resources, and an inability to reasonably allocate GPU devices to process AI tasks according to the parameters of different GPU devices, resulting in slow processing of AI tasks and reduced processing efficiency.
[0036] In response to the above problems, an embodiment of the present application provides a task allocation method for a GPU device, which can reasonably allocate the computing resources of the GPU device, improve the processing efficiency of computing tasks, and reduce the waste of computing resources.
[0037] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.
[0038] Figure 1 It is a flowchart of a method for allocating tasks to a GPU device according to an embodiment of the present application.
[0039] See also Figure 1 , a task allocation method for a GPU device, comprising:
[0040] S110: Obtain the minimum number of GPU devices required according to the first parameter of the inference task.
[0041] Specifically, the inference task may be an AI inference task, which requires the computing power of a GPU device (GPU video memory) to be processed. The first parameter of the inference task may include the task type of the inference task, such as a text recognition inference task, an image classification inference task, a speech recognition inference task, etc., a target video memory value, etc. The target video memory value may be the amount of video memory required for the inference task, that is, the GPU device requires a video memory of the target video memory value to process the inference task well. The GPU device may be a GPU device in a GPU cluster.
[0042] Specifically, the minimum number of GPU devices required can be calculated based on the task type of the inference task, the predicted required video memory value and other parameters, that is, the minimum number of GPU devices required to process the inference task, such as 1 GPU device, 2 GPU devices, etc.
[0043] In one possible implementation, the first parameter of the reasoning task is input into the second prediction model to obtain the target video memory value of the reasoning task; based on the target video memory value of the reasoning task and the maximum video memory value of the GPU device, the minimum device quantity of the GPU device is obtained.
[0044] Specifically, the second prediction model can be a Transformer neural network model, a CNN neural network model, an RNN neural network model and other models. The second prediction model can have parameters such as network layer data. The first parameters such as the task type of the reasoning task and the video memory value required for prediction can be input into the second prediction model to obtain the target video memory value of the reasoning task. The target video memory value can be the minimum video memory value required for the reasoning model output by the second prediction model.
[0045] Specifically, the target video memory value can be recorded as M (unit: GB), the maximum video memory value of a single GPU device can be recorded as m, and the maximum video memory value of each GPU device can be recorded as m. The maximum video memory value of a GPU device is the maximum video memory value that a single GPU can process at the same time. The minimum number of GPU devices can be calculated by the following formula:
[0046]
[0047] Among them, Qg is the minimum number of devices, ceil is the round-up function, and the minimum number of devices for inference tasks can be calculated through this formula. Furthermore, there may be GPU devices with different maximum video memory values in the GPU cluster, and the maximum video memory value of a single GPU device can be m1, m2, m3, and so on. The minimum number of GPU devices can be calculated by the following formula:
[0048]
[0049] The value of Qg may be a range, which may be the minimum number of devices required for GPU devices with different maximum video memory values.
[0050] S120: Obtain a pre-selected GPU device according to the minimum number of devices and the current remaining value of the GPU memory.
[0051] Specifically, after obtaining the minimum number of required GPU devices, the current remaining video memory value of each GPU device can be obtained. The current remaining video memory value can be the remaining video memory value of the GPU device. The remaining video memory value can be obtained by subtracting the used video memory value from the maximum video memory value. According to the current remaining video memory value of the GPU device, the pre-selected GPU device can be obtained. The current video memory value of the pre-selected GPU device can be greater than the target video memory value of the inference task.
[0052] In one possible implementation, if the minimum number of devices is equal to 1, a GPU device whose current remaining video memory value is greater than a target video memory value is used as a pre-selected GPU device; if the minimum number of devices is greater than 1, a number of GPU device combinations are obtained based on the target video memory value of the inference task and the current remaining video memory value of each GPU device, and the GPU devices in the GPU device combinations are used as pre-selected GPU devices, wherein the sum of the current remaining video memory values of the pre-selected GPU devices in the GPU device combination is greater than the target video memory value of the inference task.
[0053] Specifically, when the minimum number of devices is equal to 1, the current remaining video memory value of the GPU device is greater than the target video memory value of the inference task, that is, a single GPU device can handle the inference task well, and the GPU device whose current remaining video memory value is greater than the target video memory value of the inference task can be used as a pre-selected GPU device; further, each pre-selected GPU device can be input into the first device list, and the first device list can include the second parameter of each pre-selected CPU device and the current remaining video memory value of each pre-selected CPU device. The first device list can be expressed as LIST1 = [device1, device2, device3, ...], where device1, device2, and device3 can be pre-selected GPU devices.
[0054] Specifically, when the minimum number of devices is greater than 1, it means that the current remaining video memory value of a single GPU device is less than the target video memory value of the inference task, and multiple GPU devices are needed to process the inference task. Then, the current remaining video memory value of each GPU in the GPU cluster can be obtained, and multiple GPU device combinations are formed according to the permutations and combinations of the current remaining video memory values of each GPU. Each GPU device combination can include at least two GPU devices. Each GPU device in the GPU device combination can be used as a pre-selected GPU device. The sum of the current remaining video memory values of the pre-selected GPU devices in the GPU device combination is greater than the target video memory value of the inference task, so that the multiple pre-selected GPU devices in the GPU device combination can jointly process the inference task. Further, the GPU device combination and each preset GPU device in each GPU device combination can be input into the second list, and the second device list is recorded according to each GPU combination. The second device list can be expressed as LIST2=[(device1, device2), (device1, device3), (devic2, device3), (device1, device2, device3)...], where device1, device2, and device3 can be pre-selected GPU devices.
[0055] S130: Inputting the first parameter of the inference task and the second parameter of the pre-selected GPU device into the first prediction model to obtain the expected response time of the inference task.
[0056] Specifically, the first parameter of the reasoning task and the second parameter of the pre-selected GPU device can be input into the first prediction model. The first prediction model can be used to predict the expected response time of the reasoning task. The expected response time is the processing time required to use the pre-selected GPU device. The second parameter includes the graphics card model, operating system model, remaining memory size, current video memory remaining value and other parameters of the GPU device.
[0057] In one possible implementation, if the minimum number of devices is equal to 1, the first parameter of the inference task and the second parameter of the preselected GPU device are input into the first prediction model according to the operating status of the preselected GPU device to obtain the expected response time of the inference task; if the minimum number of devices is greater than 1, the first parameter of the inference task and the second parameter of each preselected GPU device in the GPU device combination are input into the first prediction model to obtain the expected response time of the inference task.
[0058] Specifically, if the minimum number of devices is equal to 1, the running status of the preselected GPU device can be obtained, for example, the running status of each preselected GPU device in the first device list can be obtained, and the running status can be whether there are other tasks being processed in the preselected GPU device, that is, whether there are other tasks in the preselected GPU device that occupy the video memory in the preselected GPU device; when the running status of the preselected GPU device indicates that the preselected GPU device has no other tasks being processed, the second parameter of the preselected CPU device and the first parameter of the reasoning task can be input into the first prediction model to obtain the expected response time of the reasoning task; when the running status of the preselected GPU device indicates that the preselected GPU device has other tasks being processed, the second parameter of the preselected CPU device, the first parameter of the reasoning task and various parameters of other tasks, such as the task type of the task, can be input into the first prediction model to obtain the expected response time of the reasoning task. Further, the expected response time of the reasoning task calculated according to different running states of the preselected GPU device can be the same, that is, when the video memory occupied by other tasks processed in the preselected GPU device is too small, so as not to affect the speed of processing the reasoning task.
[0059] Specifically, if the minimum number of devices is greater than 1, the second parameter of each preselected GPU device in the GPU device combination can be obtained. For example, the second parameter of each preselected GPU device in each GPU device combination in the second device list can be obtained. The second parameter of each preselected GPU device in the GPU device combination, the communication bandwidth between each preselected GPU device, the average distance between each preselected GPU device, and the first parameter of the inference task can be input into the first prediction model to obtain the expected response time of the inference task.
[0060] S140: Obtain a target response time for the inference task, and obtain at least one target GPU device from pre-selected GPU devices according to the expected response time and the target response time.
[0061] Specifically, the target response time of the reasoning task can be obtained. The target response time can be the response time of the reasoning task processed in other GPU devices. Then, based on the expected response time and the target response time, at least one target GPU device is obtained from the pre-selected GPU devices. For example, when the minimum number of devices is equal to 1, a target GPU device can be obtained. If the minimum number of devices is greater than 1, a suitable GPU device combination can be obtained from the GPU device combination, and the pre-selected GPU device in the GPU device combination can be used as the target GPU device. The target GPU device can be a GPU device with higher efficiency in processing reasoning tasks. For example, the expected response time of the target GPU device is close to the target response time of the reasoning task. For example, the target response time of the reasoning task is 5 seconds, and the expected response time of the target GPU device is 5.5 seconds.
[0062] In one possible implementation, a preset GPU device or a combination of GPU devices is obtained when the expected response time is less than the target response time; the load capacity of each preselected GPU device or the load capacity of each GPU device combination is calculated, and the preselected GPU device or the preselected GPU device in the GPU device combination whose load capacity is greater than a preset threshold is used as the target GPU device.
[0063] Specifically, pre-selected GPU devices or preset GPU device combinations whose expected response time for inference tasks is less than the target response time can be obtained from the first device list or the second device list, and the pre-selected GPU devices or preset GPU device combinations whose expected response time is less than the target response time can be input into the third device list, and in the third device list, they can be arranged in order from large to small according to the difference between the target response time and the expected response time, and the pre-selected GPU devices or GPU device combinations that are ranked in the front have the shorter expected response time.
[0064] Specifically, the load capacity of the pre-selected GPU device or the load capacity of the GPU device combination may be the entire health score of the GPU cluster after the inference task is sent to the pre-selected GPU device or the pre-selected GPU device in the GPU device combination. The load capacity of each pre-selected GPU device or GPU device combination may be calculated by the following formula:
[0065]
[0066] Among them, Score is the load of the GPU device or GPU device combination, N is the number of tasks in the GPU cluster, and S i is the expected response time of the task, S i is the actual response time of the task, which may be the actual processing time of the task by the pre-selected GPU device or GPU device combination. i is the priority weight of the task, w i The value of can be 1, 2, 3, w i =1 means the task has low priority, w i =2 represents the priority of the task, w i =3 means that the task has a high priority. For example, the priority of the task can be obtained from the task requirement document.
[0067] Specifically, the first carrying capacity of each preselected GPU device or GPU device combination before sending the inference task can be calculated according to the order in the third device list, and the second carrying capacity of each preselected GPU device or GPU device combination adding the inference task to the preselected GPU device or the GPU device combination can be calculated. When it is calculated that the second carrying capacity of a preselected GPU device or GPU device combination is greater than the first carrying capacity, the preselected GPU or GPU device combination is used as the target GPU device until the preselected GPU devices or GPU device combinations in the third device list are calculated. If there is no preselected GPU device or GPU device combination in the third device list whose second carrying capacity is greater than the first carrying capacity, the inference task is temporarily suspended.
[0068] In a possible implementation, a target response time of the reasoning task is obtained from a database; and / or a first parameter of the reasoning task is input into a third prediction model to obtain the target response time.
[0069] Specifically, the target response time of the reasoning task can be obtained from the requirement document of the reasoning task, and various information of the reasoning task can be recorded in the requirement document.
[0070] Specifically, the first parameter of the reasoning task can also be input into the third prediction model. The third prediction model can be a KNN algorithm model. The third prediction model can be used to match the historical reasoning task that is closest to the parameters of the reasoning task in the historical reasoning task database, and the response time of the historical reasoning task can be used as the target response time of the reasoning task.
[0071] S150: Allocate the inference task to the target GPU device for processing.
[0072] Specifically, once a specific target GPU device is obtained, the inference task can be assigned to the target GPU device for processing, which can maximize the processing efficiency of the inference task and avoid wasting the computing power of the GPU.
[0073] In a possible implementation, after the reasoning task is assigned to the target GPU device, the system may record the second parameter of the GPU device used by the reasoning task and the actual response time of the reasoning task, and update them to the historical reasoning task database.
[0074] The present application discloses a method for allocating tasks to a GPU device, comprising obtaining a minimum number of required GPU devices according to a first parameter of an inference task, wherein the first parameter at least includes a target video memory value of the inference task; obtaining a preselected GPU device according to the minimum number of devices and a current video memory remaining value of the GPU device; inputting the first parameter of the inference task and a second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; obtaining a target response time of the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; and allocating the inference task to the target GPU device for processing, thereby being able to reasonably allocate computing power resources of the GPU device, improve the processing efficiency of computing power tasks, and reduce the waste of computing power resources.
[0075] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a task allocation device for a GPU device, an electronic device and corresponding embodiments.
[0076] Figure 2 It is a structural diagram of a task allocation device of a GPU device shown in an embodiment of the present application.
[0077] See also Figure 2 , a task allocation device 200 for a GPU device, comprising:
[0078] The first acquisition module 210 is used to acquire the minimum number of GPU devices required according to a first parameter of the inference task, where the first parameter at least includes a target video memory value of the inference task.
[0079] Specifically, the inference task may be an AI inference task, which requires the computing power of a GPU device (GPU video memory) to be processed. The first parameter of the inference task may include the task type of the inference task, such as a text recognition inference task, an image classification inference task, a speech recognition inference task, etc., a target video memory value, etc. The target video memory value may be the amount of video memory required for the inference task, that is, the GPU device requires a video memory of the target video memory value to process the inference task well. The GPU device may be a GPU device in a GPU cluster.
[0080] In a possible implementation, the first acquisition module 210 is also used to input the first parameter of the reasoning task into the second prediction model to obtain the target video memory value of the reasoning task; and obtain the minimum device quantity of the GPU device based on the target video memory value of the reasoning task and the maximum video memory value of the GPU device.
[0081] The second acquisition module 220 is used to obtain a pre-selected GPU device according to the minimum number of devices and the current remaining value of the video memory of the GPU device.
[0082] Specifically, after obtaining the minimum number of required GPU devices, the current remaining video memory value of each GPU device can be obtained. The current remaining video memory value can be the remaining video memory value of the GPU device. The remaining video memory value can be obtained by subtracting the used video memory value from the maximum video memory value. According to the current remaining video memory value of the GPU device, the pre-selected GPU device can be obtained. The current video memory value of the pre-selected GPU device can be greater than the target video memory value of the inference task.
[0083] In a possible implementation, the second acquisition module 220 is also used to, if the minimum number of devices is equal to 1, use a GPU device whose current remaining video memory value is greater than the target video memory value as a pre-selected GPU device; if the minimum number of devices is greater than 1, then based on the target video memory value of the inference task and the current remaining video memory value of each GPU device, obtain several GPU device combinations, and use the GPU devices in the GPU device combinations as pre-selected GPU devices, wherein the sum of the current remaining video memory values of the pre-selected GPU devices in the GPU device combination is greater than the target video memory value of the inference task.
[0084] The calculation module 230 is used to input the first parameter of the reasoning task and the second parameter of the pre-selected GPU device into the first prediction model to obtain the expected response time of the reasoning task.
[0085] Specifically, the first parameter of the reasoning task and the second parameter of the pre-selected GPU device can be input into the first prediction model. The first prediction model can be used to predict the expected response time of the reasoning task. The expected response time is the processing time required to use the pre-selected GPU device. The second parameter includes the graphics card model, operating system model, remaining memory size, current video memory remaining value and other parameters of the GPU device.
[0086] In one possible implementation, the computing module 230 is also used to, if the minimum number of devices is equal to 1, input the first parameter of the inference task and the second parameter of the preselected GPU device into the first prediction model according to the running status of the preselected GPU device to obtain the expected response time of the inference task; if the minimum number of devices is greater than 1, input the first parameter of the inference task and the second parameter of each preselected GPU device in the GPU device combination into the first prediction model to obtain the expected response time of the inference task.
[0087] The processing module 240 is used to obtain a target response time of the inference task, and obtain at least one target GPU device from pre-selected GPU devices according to the expected response time and the target response time.
[0088] Specifically, the target response time of the reasoning task can be obtained. The target response time can be the response time of the reasoning task processed in other GPU devices. Then, based on the expected response time and the target response time, at least one target GPU device is obtained from the pre-selected GPU devices. For example, when the minimum number of devices is equal to 1, a target GPU device can be obtained. If the minimum number of devices is greater than 1, a suitable GPU device combination can be obtained from the GPU device combination, and the pre-selected GPU device in the GPU device combination can be used as the target GPU device. The target GPU device can be a GPU device with higher efficiency in processing reasoning tasks. For example, the expected response time of the target GPU device is close to the target response time of the reasoning task. For example, the target response time of the reasoning task is 5 seconds, and the expected response time of the target GPU device is 5.5 seconds.
[0089] In one possible implementation, the processing module 240 is also used to obtain a preset GPU device or a GPU device combination when the expected response time is less than the target response time; calculate the load capacity of each preselected GPU device or the load capacity of each GPU device combination, and use the preselected GPU device or the preselected GPU device in the GPU device combination whose load capacity is greater than a preset threshold as the target GPU device.
[0090] In a possible implementation, the processing module 240 is further configured to obtain a target response time of the reasoning task from a database; and / or input a first parameter of the reasoning task into a third prediction model to obtain the target response time.
[0091] The allocation module 250 is used to allocate the inference task to the target GPU device for processing.
[0092] Specifically, when a specific target GPU device is obtained, the inference task can be assigned to the target GPU device for processing, which can maximize the processing efficiency of the inference task and avoid wasting the computing power of the GPU.
[0093] The present application provides a task allocation device for a GPU device, comprising obtaining a minimum number of required GPU devices according to a first parameter of an inference task, wherein the first parameter at least includes a target video memory value of the inference task; obtaining a preselected GPU device according to the minimum number of devices and a current video memory remaining value of the GPU device; inputting the first parameter of the inference task and a second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; obtaining a target response time of the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; and allocating the inference task to the target GPU device for processing, thereby being able to reasonably allocate computing power resources of the GPU device, improve the processing efficiency of computing power tasks, and reduce the waste of computing power resources.
[0094] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated again here.
[0095] Figure 3 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.
[0096] See also Figure 3 , the electronic device 300 includes a memory 310 and a processor 320.
[0097] The processor 320 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0098] The memory 310 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 320 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 310 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 310 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.
[0099] The memory 310 stores executable codes, and when the executable codes are processed by the processor 320 , the processor 320 can execute part or all of the methods described above.
[0100] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.
[0101] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.
[0102] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A task allocation method for a GPU device, characterized in that: include: According to the first parameter of the inference task, obtain the minimum number of GPU devices required; Obtaining a preselected GPU device according to the minimum number of devices and the current remaining video memory value of the GPU device; Inputting a first parameter of the inference task and a second parameter of the preselected GPU device into a first prediction model to obtain an expected response time of the inference task; Obtaining a target response time for the inference task, and obtaining at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; The inference task is assigned to the target GPU device for processing.
2. The method according to claim 1, characterized in that The step of obtaining the minimum number of GPU devices required according to the first parameter of the inference task includes: Inputting the first parameter of the reasoning task into the second prediction model to obtain the target video memory value of the reasoning task; The minimum device quantity of the GPU device is obtained according to the target video memory value of the inference task and the maximum video memory value of the GPU device.
3. The method according to claim 2, characterized in that The step of obtaining a preselected GPU device according to the minimum number of devices and the current remaining video memory value of the GPU device includes: If the minimum number of devices is equal to 1, the GPU device whose current video memory remaining value is greater than the target video memory value is used as the pre-selected GPU device; If the minimum number of devices is greater than 1, then several GPU device combinations are obtained according to the target video memory value of the reasoning task and the current remaining video memory value of each of the GPU devices, and the GPU devices in the GPU device combinations are used as the pre-selected GPU devices, wherein the sum of the current remaining video memory values of the pre-selected GPU devices in the GPU device combination is greater than the target video memory value of the reasoning task.
4. The method according to claim 3, characterized in that The step of inputting the first parameter of the inference task and the second parameter of the pre-selected GPU device into a first prediction model to obtain an expected response time of the inference task includes: If the minimum number of devices is equal to 1, inputting the first parameter of the inference task and the second parameter of the preselected GPU device into the first prediction model according to the running state of the preselected GPU device to obtain the expected response time of the inference task; If the minimum number of devices is greater than 1, the first parameter of the inference task and the second parameter of each of the preselected GPU devices in the GPU device combination are input into the first prediction model to obtain the expected response time of the inference task.
5. The method according to claim 4, characterized in that The step of obtaining the target response time of the inference task and obtaining at least one target GPU device from the pre-selected GPU devices according to the expected response time and the target response time includes: Acquire the preset GPU device or the GPU device combination when the expected response time is less than the target response time; The load capacity of each of the pre-selected GPU devices or the load capacity of each of the GPU device combinations is calculated, and the pre-selected GPU devices or the pre-selected GPU devices in the GPU device combination whose load capacity is greater than a preset threshold are used as the target GPU devices.
6. The method according to any one of claims 1 to 5, characterized in that: The obtaining the target response time of the reasoning task includes: Obtaining a target response time for the reasoning task from a database; And / or, inputting the first parameter of the reasoning task into a third prediction model to obtain the target response time.
7. A task allocation device for a GPU device, characterized in that: include: A first acquisition module, used for acquiring a minimum number of GPU devices required according to a first parameter of the inference task, wherein the first parameter at least includes a target video memory value of the inference task; A second acquisition module is used to obtain a pre-selected GPU device according to the minimum number of devices and the current remaining video memory value of the GPU device; A calculation module, used for inputting the first parameter of the reasoning task and the second parameter of the pre-selected GPU device into a first prediction model to obtain an expected response time of the reasoning task; A processing module, configured to obtain a target response time of the inference task, and obtain at least one target GPU device from the preselected GPU devices according to the expected response time and the target response time; The allocation module is used to allocate the inference task to the target GPU device for processing.
8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 6.