Resource allocation method, apparatus, device, and storage medium

Abstract

A resource allocation method, device, equipment and storage medium belong to the artificial intelligence technical field. The method comprises: acquiring state information of a first time unit, the state information being used to indicate the state of a GPU and at least one task. According to the state information, an action information is generated by a reinforcement learning model, and the action information is used to indicate the allocation mode of the GPU resource corresponding to the at least one task. After scheduling the GPU to execute the action information, the state information of a second time unit is determined. According to the loss function value calculated based on the state information and the action information, the parameters of the reinforcement learning model are adjusted to obtain a trained reinforcement learning model. The above method can acquire the state of the GPU and the task in real time, timely respond to the dynamic resource demand of the task, and improve the resource utilization efficiency. Moreover, the resource allocation mode is based on the state of the task, which enhances the adaptability of the resource allocation to the task demand, so that the resource allocation is more reasonable.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a resource allocation method, apparatus, device, and storage medium. Background Technology

[0002] GPUs (Graphics Processing Units) excel at handling tasks with high data parallelism, such as matrix operations and image processing. Proper GPU resource allocation can significantly improve the execution efficiency of parallel tasks, reduce resource waste, and ensure system stability and reliability under high load conditions.

[0003] Related technologies employ load balancing resource allocation methods to distribute GPU resources (such as GPU computing resources and GPU memory resources). This method evenly distributes GPU resources among running tasks (i.e., tasks awaiting GPU resource allocation) to achieve load balancing. Specifically, it first obtains the resource requirements of multiple tasks and calculates the average of these requirements. Then, it allocates GPU resources to each task based on this average. For example, if three tasks are running, the system will try to ensure that each task occupies approximately one-third of the GPU resources. When a new task starts or an existing task finishes, the system recalculates the load balancing allocation scheme, evenly distributing the released resources to other running tasks or to newly started tasks.

[0004] The above method does not consider the actual performance requirements of each task. For example, graphics rendering tasks have high requirements for GPU memory resources, while deep learning inference tasks have high requirements for GPU computing resources. This simple load balancing method can lead to unreasonable allocation of GPU resources. Summary of the Invention

[0005] This application provides a resource allocation method, apparatus, device, and storage medium. The technical solutions provided by this application are as follows:

[0006] According to one aspect of the embodiments of this application, a resource allocation method is provided, the method comprising:

[0007] Obtain the status information of the first time unit, the status information being used to indicate the status of the graphics processing unit (GPU) and the status of at least one task associated with the GPU;

[0008] The reinforcement learning model generates action information for the first time unit based on the state information of the first time unit. The action information is used to indicate the allocation method of GPU resources corresponding to the at least one task.

[0009] After scheduling the GPU to execute the operation indicated by the action information of the first time unit, the state information of the second time unit is determined, wherein the second time unit is the time unit following the first time unit;

[0010] Based on the loss function value calculated using the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, the parameters of the reinforcement learning model are adjusted to obtain the trained reinforcement learning model.

[0011] According to one aspect of the embodiments of this application, a resource allocation apparatus is provided, the apparatus comprising:

[0012] The first acquisition module is used to acquire the status information of the first time unit, the status information being used to indicate the status of the graphics processing unit (GPU) and the status of at least one task associated with the GPU;

[0013] The generation module is used to generate action information of the first time unit based on the state information of the first time unit through a reinforcement learning model. The action information is used to indicate the allocation method of GPU resources corresponding to the at least one task.

[0014] The determination module is used to determine the state information of the second time unit after scheduling the GPU to execute the operation indicated by the action information of the first time unit, wherein the second time unit is the time unit after the first time unit;

[0015] The adjustment module is used to adjust the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, so as to obtain the trained reinforcement learning model.

[0016] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described resource allocation method.

[0017] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described resource allocation method.

[0018] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described resource allocation method.

[0019] The technical solutions provided in this application have at least the following beneficial effects:

[0020] By using reinforcement learning models to determine GPU resource allocation, the system can acquire real-time GPU and task states, enabling timely responses to dynamic task demands for GPU resources and improving resource utilization efficiency. Furthermore, determining GPU resource allocation based on the state of at least one task enhances the adaptability of resource allocation to task requirements, resulting in more rational GPU resource allocation. Attached Figure Description

[0021] Figure 1 This is a flowchart of a possible implementation of the resource allocation method in this application;

[0022] Figure 2 This is a schematic diagram of system information interaction in one possible implementation of this application;

[0023] Figure 3 This is a flowchart of a GPU resource allocation method based on a reinforcement learning model in one possible implementation of this application;

[0024] Figure 4 This is a block diagram of a resource allocation device in one possible implementation of this application;

[0025] Figure 5 This is a structural block diagram of a computer device in one possible implementation of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] Before introducing and explaining the technical solution of this application, some concepts involved in this application will be defined and explained.

[0028] reinforcement learning

[0029] Reinforcement learning is a field within machine learning that primarily studies how an agent takes a series of actions within an environment to maximize cumulative reward. The agent observes the state of the environment, selects actions based on a policy, receives reward feedback from the environment, and adjusts its policy accordingly.

[0030] Deep Q-Network (DQN) is a commonly used reinforcement learning algorithm that uses neural networks to approximate the Q-value function. The Q-value represents the expected long-term reward of taking a specific action in a given state. By training the neural network, DQN can learn the optimal policy, that is, choose the action that brings the maximum Q-value.

[0031] GPU resource allocation

[0032] GPU resource allocation refers to the process of distributing available GPU resources to different tasks in a computing environment. The goal of GPU resource allocation is to optimize the utilization efficiency of GPU resources, ensuring that each task receives sufficient resources to complete efficiently while avoiding resource waste. GPU resource allocation typically occurs in a dynamically changing environment, where different tasks may have different resource requirements at different times. This application proposes a GPU resource allocation method based on a reinforcement learning model. Reinforcement learning models can learn through trial and error to find the optimal resource allocation strategy in a constantly changing environment, thereby achieving efficient GPU resource allocation in complex computing environments.

[0033] Please refer to Figure 1 The diagram illustrates a flowchart of a resource allocation method in one possible implementation of this application. The execution entity for each step of the method can be a computer device. The method may include at least one of steps 110 to 130.

[0034] Step 110: Obtain the state information of the first time unit. The state information is used to indicate the state of the GPU and the state of at least one task associated with the GPU.

[0035] The first time unit refers to a point in time or a period of time within a system's time cycle. At least one task refers to a task that requires GPU execution; in other words, at least one task is a task that requires GPU resources. Within the first time unit, at least one task can be a task that has already been allocated GPU resources or a task awaiting allocation of GPU resources. Tasks can be application tasks, and an application can include multiple tasks.

[0036] The GPU state refers to the current operating status of the GPU, indicating its resource usage to facilitate subsequent resource allocation by the reinforcement learning model. The state of at least one task indicates its own attributes (e.g., task type is deep learning) and its demand for GPU resources, helping the reinforcement learning model optimize task execution order and resource allocation. The state of at least one task directly affects subsequent GPU resource allocation, reflecting the actual use of GPU resources by at least one task. By combining the GPU state and task state, the reinforcement learning model can perceive GPU resource usage and changes in task demand for GPU resources in real time, improving GPU resource utilization. For example, when GPU memory utilization is low and a task requires memory, the model can adjust its resource allocation strategy promptly, allocating available memory to that task. Furthermore, resource allocation considering the state of at least one task allows the reinforcement learning model to more comprehensively understand task characteristics and demand changes, leading to more accurate resource allocation and task scheduling.

[0037] In some embodiments, the status information includes GPU information. The GPU information includes at least one of the following: GPU computing resource information and GPU memory resource information.

[0038] GPU information is used to describe the state of the aforementioned GPU. GPU computing resource information may include at least one of the following: the total number of computing resources, the number of allocated computing resources, and the number of unallocated computing resources. The number of allocated computing resources indicates the number of total computing resources that have been allocated to tasks, and the number of unallocated computing resources indicates the number of total computing resources that have not been allocated to tasks (i.e., the number of idle computing resources).

[0039] In some embodiments, GPU memory resource information may include at least one of the following: the total amount of memory resources, the amount of allocated memory resources, and the amount of unallocated memory resources. The amount of allocated memory resources indicates the number of total memory resources allocated to tasks, while the amount of unallocated memory resources indicates the number of total memory resources not allocated to tasks (i.e., the amount of free memory resources). In some embodiments, memory utilization (which can be determined based on the amount of allocated memory resources) and compute unit utilization (which can be determined based on the amount of allocated compute resources) can be directly obtained through interfaces provided by the GPU driver, which can monitor the GPU's hardware status information in real time. Memory utilization and compute unit utilization can be expressed numerically; for example, memory utilization can be expressed as a decimal between 0 and 1, and compute unit utilization can be expressed as a percentage value. The GPU driver described above is primarily responsible for directly interacting with the GPU hardware to obtain GPU hardware status information, such as memory utilization and compute unit utilization.

[0040] In some embodiments, GPU information may further include GPU temperature and GPU power consumption. GPU temperature indicates the current temperature state of the GPU, helping to monitor and manage heat dissipation and prevent overheating. GPU power consumption indicates the current power consumption of the GPU, used to optimize energy efficiency and power management. This application does not limit the content of the GPU information.

[0041] In some embodiments, the status information includes task information. The task information includes at least one of the following: resource request information for at least one task, type information for at least one task, priority information for at least one task, and historical resource usage information for at least one task. The resource request information indicates the type and quantity of GPU resources requested by the task; the type information indicates the functional classification of the task; the priority information indicates the importance of the task in the system; and the historical resource usage information indicates the historical usage of GPU resources by tasks.

[0042] Task information is used to describe the state of at least one of the aforementioned tasks. In some embodiments, resource request information is used to indicate the amount of GPU computing resources and / or GPU memory resources requested by the task. In some embodiments, resource request information can be obtained through a queue data structure maintained by the resource management module, which records all task requests waiting for resource allocation. The resource management module needs to communicate with the GPU driver to obtain GPU information and also interact with the application layer (i.e., the application to which at least one task belongs) to obtain task information (such as task type, resource request, etc.), and execute specific resource allocation operations based on the decisions of the reinforcement learning model.

[0043] Type information can include different categories such as deep learning, graphics rendering, and scientific computing. Different types of tasks have different requirements for GPU resources. For example, deep learning tasks typically have high requirements for GPU computing resources and large amounts of video memory, while graphics rendering tasks focus more on video memory resources. In some embodiments, the execution characteristics of a task can be determined based on its type information, thereby allocating computing resources reasonably and accelerating the task execution process. For example, for computationally intensive deep learning training tasks, the model can allocate computing resources reasonably based on the training progress and model structure characteristics to accelerate the training process. For example, when training a deep neural network, the model allocates more computing units to the computational task during the model parameter update phase to improve computational efficiency, thereby shortening the training time, accelerating the model's convergence speed, and improving the model's training efficiency and final performance. In some embodiments, resource requirement change information for the task is determined based on the task type information. This resource requirement change information indicates the task's demand for GPU resources at different stages (i.e., the type and quantity of GPU resources required by the task at different stages); based on the task's resource requirement change information, a certain amount of GPU resources are reserved for the task in the current resource allocation process. For example, a deep learning task may have low resource requirements in the early stages of training, but these requirements will increase dramatically as training progresses. The system can reserve an appropriate amount of GPU resources for this task during the current resource allocation process based on the task type information and the changing resource requirements. This dynamic resource reservation mechanism ensures that the resource requirements of the task at different stages are met in a timely manner, improving the overall system performance and resource utilization efficiency. In some embodiments, the task type can be declared to the resource management module by the application itself when the application to which the task belongs at startup, or it can be determined through analysis of the application's executable file; this application does not limit this.

[0044] In some embodiments, the GPU type is determined based on the task type. Specifically, the application scenario analysis module receives source data such as the type of the application (i.e., the task) and the characteristics of the computational task. For example, deep learning training tasks are characterized by a large number of matrix operations and data parallel processing; graphics rendering tasks emphasize graphics processing power and memory bandwidth. The application scenario analysis module transmits the analysis results to the GPU selection module. Based on the above information, the GPU selection module selects suitable GPU types from the GPU device library (containing performance parameters and characteristics of various GPU models). For example, in deep learning scenarios, GPUs with more computing cores and larger memory are selected; in graphics rendering scenarios, GPUs with strong graphics processing power and high memory bandwidth are selected. The finally determined GPU device type is provided to the system administrator or hardware installer in the form of a configuration file or hardware installation list for subsequent hardware procurement and installation.

[0045] In some implementations, tasks with higher importance to the system are given priority in GPU resource allocation; conversely, tasks with lower importance are not allocated resources when GPU resources are scarce. In some embodiments, tasks with extremely high real-time requirements can be assigned higher priority. For example, for tasks with extremely high real-time requirements (such as game rendering and real-time video processing), the model can ensure sufficient GPU resources are allocated at critical moments (such as intense battle scenes in a game or high-definition scene transitions in video playback) to guarantee stable frame rates and low latency. For instance, in game development, when a player enters a battle scene, the model quickly allocates more computing units and video memory bandwidth to the game rendering task to ensure smooth rendering of special effects and character models, avoiding stuttering and greatly improving the player's gaming experience. Task priorities can be preset by users or system administrators based on task importance, or dynamically adjusted according to the task's real-time performance requirements. This application does not limit this.

[0046] In some embodiments, the GPU type is determined based on compatibility with the host system. Specifically, the hardware detection module acquires hardware information from the host system, such as motherboard model, PCIe slot specifications, and power supply wattage. This information is then transmitted to the compatibility check module, which checks the compatibility between the selected GPU and the host system against the GPU's hardware requirements (e.g., need for a PCIe 4.0 slot, specific power supply support, etc.). Simultaneously, the scalability assessment module evaluates the feasibility of future GPU expansion based on factors such as host chassis space and cooling capacity. The results of the compatibility check and scalability assessment are reported to the system administrator. If incompatibility or poor scalability is found, the administrator adjusts the GPU selection or upgrades the host system accordingly.

[0047] In some embodiments, time-series data is used to represent changes in resource usage of a task over a past period. Specifically, this may include the task's usage of GPU computing resources and GPU memory resources over a past period. Historical resource usage information can be used to provide a reference for current and future resource allocation. Specifically, time-series analysis can be used to analyze the usage trends of GPU computing and memory resources by tasks in different time periods, predicting future resource demands. For example, moving averages, exponential smoothing, and other methods can be used to smooth the data and identify trends. Periodic analysis can also be used to identify periodic patterns in task GPU resource usage (such as daily or weekly peak usage) and predict future resource demands based on these patterns. The above methods, based on historical resource usage information, predict future peak resource demands and reserve a certain amount of resources in the current resource allocation to cope with potential future resource shortages.

[0048] The methods described above, by leveraging GPU information, help reinforcement learning models understand GPU resource usage in a timely manner, enabling them to make reasonable resource allocation strategies. Task information allows for a better understanding of task requirements, ensuring that subsequent resource allocation adapts to these requirements, improving overall system efficiency and task completion quality. Furthermore, reinforcement learning models can continuously learn and adapt to the dynamic changes in GPU resource requirements at different stages of a task. Whether it's the fluctuations in resource requirements during different training stages (data loading, computationally intensive, parameter update, etc.) in deep learning training tasks, or the adjustments to resource requirements in graphics rendering tasks as scene complexity changes, the model can respond promptly. For example, during deep learning training, when the model transitions from the data loading stage to the computationally intensive stage, the computational resource requirements increase sharply. The model can quickly detect this change and reallocate more computing units to the task, ensuring smooth task execution while avoiding excessive and unnecessary computational resource consumption during the data loading stage, thus improving overall resource utilization. As the system runs, the model, through continuous experience and optimization strategies, can better predict future task resource requirements and prepare and adjust resources in advance. For example, for a task that operates periodically and has similar resource demand patterns, the model can accurately predict the changes in resource demand in the next cycle after several cycles, allocate resources in advance, reduce resource waste, and further improve resource utilization.

[0049] Step 120: Based on the state information of the first time unit, the reinforcement learning model generates action information for the first time unit. The action information is used to indicate the allocation method of GPU resources for at least one task.

[0050] The GPU resource allocation method refers to the specific allocation strategy or resource scheduling scheme for GPU resources. The GPU resource allocation method for at least one task refers to the specific way GPU resources are allocated in a specific proportion or quantity for each task. For example, for tasks with high real-time requirements, more GPU resources can be allocated in the first time unit; for tasks with low real-time requirements, fewer or no GPU resources can be allocated in the first time unit.

[0051] In some embodiments, when the task is one that has already been allocated GPU resources, the GPU resource allocation method for that task refers to adjusting the allocation ratio or quantity of those GPU resources. When the task is one that needs to be allocated GPU resources, the GPU resource allocation method for that task refers to determining the allocation ratio or quantity of those GPU resources.

[0052] In some embodiments, the action information includes at least one of the following: task allocation information, task allocation adjustment information, and pause / resume information; wherein, the task allocation information is used to indicate the type and quantity of GPU resources allocated to at least one task, the task allocation adjustment information is used to indicate the type and quantity of GPU resources allocated to at least one task after adjustment, and the pause / resume information is used to indicate pausing the resource allocation of at least one task and resuming the resource allocation of the paused task.

[0053] GPU resources include the aforementioned GPU computing resources and video memory resources. In some embodiments, the allocation of GPU computing resources can be achieved by modifying the configuration of the GPU task scheduler. The resource management module sends instructions to the GPU task scheduler based on the decisions of the reinforcement learning model, specifying the allocation of a specific number of computing units to the target task. For example, functions provided by the MUSA (Meta Silicon Unified Architecture) programming interface can be used to adjust the allocation of computing resources for tasks. The allocation of video memory resources can be accomplished by modifying the parameters of the video memory management unit. The resource management module interacts with the GPU's video memory management unit, allocating video memory quotas for different tasks according to the reinforcement learning model. This involves the partitioning and allocation of the video memory address space to ensure that each task receives appropriate video memory resources.

[0054] In some embodiments, the type and amount of GPU resource allocation for a task can be adjusted. This adjustment may be due to changes in task requirements or to optimize overall system resource utilization. For example, in a graphics rendering task, if it is found that the scene rendering for that task requires more video memory resources, the system can dynamically increase the video memory allocation for that task to avoid insufficient video memory during rendering. For example, when high-priority tasks are added, more GPU resources can be allocated to these tasks first, while reducing resource allocation for low-priority tasks to ensure that high-priority tasks can complete quickly.

[0055] In some embodiments, when GPU resource utilization exceeds a certain threshold, resource allocation for some non-critical tasks (i.e., low-priority tasks) can be paused; when GPU resource utilization falls below a certain threshold, resource allocation for paused tasks (i.e., low-priority tasks) can be resumed. In other words, when GPU resources are scarce, resource allocation for some non-critical tasks can be paused to prioritize the execution of critical tasks; when resources are sufficient, resource allocation for paused tasks can be resumed to improve the overall resource utilization of the system. In some embodiments, pausing or resuming GPU resource allocation is achieved by controlling the on / off switch of GPU resource allocation. The resource management module can set flags at the system level or call interface functions provided by the GPU driver to pause or resume application access to GPU resources. When an application is paused, its tasks on the GPU are suspended, and the released GPU resources can be used by tasks of other applications; when resource allocation for a paused task is resumed, the task can continue execution from the paused state.

[0056] The aforementioned method allows the reinforcement learning model to dynamically manage and optimize GPU resource allocation based on action information, ensuring efficient task execution and rational utilization of GPU resources. Furthermore, the action space of this reinforcement learning model includes various flexible resource allocation operations, such as task allocation information that precisely adjusts the number of computing units allocated and the proportion of GPU memory allocation. This enables fine-grained adjustments to resource allocation based on the actual needs of the task, rather than simple coarse-grained allocation. For example, for a graphics rendering task with extremely high GPU memory bandwidth requirements, the reinforcement learning model can dynamically adjust the GPU memory allocation ratio based on the real-time progress of the task, ensuring that GPU memory bandwidth always meets the task's needs.

[0057] In some embodiments, the DQN algorithm can be used to train reinforcement learning models. The DQN algorithm approximates the Q-value function by constructing a neural network, effectively handling problems involving high-dimensional state spaces and continuous action spaces, making it well-suited for GPU resource allocation scenarios. In GPU resource allocation, the state space contains GPU and task information, while the action space contains various resource allocation operations. The DQN algorithm can learn the complex relationship between states and actions through the neural network, thereby making optimal resource allocation decisions. The input layer of the neural network receives a state space vector, and the number of neurons is the same as the dimension of the state space. The state space can contain 6 elements (such as memory usage, computation unit utilization, resource request information, task type information, task priority information, and historical resource usage information), in which case the input layer has 6 neurons. Multiple hidden layers can be set, with the number of neurons in each layer determined according to the actual situation. Generally, the number of neurons in the hidden layers can gradually decrease; for example, the first hidden layer has 32 neurons, the second layer has 16 neurons, and so on. The activation function for the hidden layer can be the ReLU (Rectified Linear Unit) function, which accelerates the convergence speed of the neural network while avoiding the vanishing gradient problem. The number of neurons in the output layer is the same as the dimension of the action space, and each neuron corresponds to a Q-value estimate (i.e., a predicted Q-value) for an action. The activation function for the output layer can be a linear function to ensure the true range of the output Q-value. Initially, without training, the reinforcement learning model randomly generates resource allocation action information (such as randomly allocating computing units and GPU memory), which serves as the initial source data for resource allocation decisions. The reinforcement learning model transmits this resource allocation action information to the resource management module to schedule the GPU to perform the actual resource allocation operation. Simultaneously, it collects the current state, action, reward (the initial reward can be set according to simple rules, such as a small reward for no obvious conflict after resource allocation and a penalty for conflict), and next state information to form an experience tuple, which is stored in the experience replay buffer. After a certain number (e.g., 1000) of random allocation iterations, sufficient initial experience data is collected, providing a foundation for subsequent model training.

[0058] In some embodiments, based on the predicted Q-values ​​of multiple action information output by the output layer, the action information corresponding to the largest predicted Q-value is determined as the action information of the first time unit. At each time step (i.e., each time unit), the predicted Q-value of each action information is calculated based on the state information of the current time unit (i.e., the first time unit). Specifically, the state vector corresponding to the state information of the first time unit is input into the neural network, and after forward propagation calculation, the Q-value corresponding to each neuron in the output layer is obtained. For example, if there are three action information in the action space (such as allocating computing units, adjusting memory ratio, pausing / resuming task allocation), the output layer will obtain three predicted Q-values. The resource management module executes corresponding resource allocation operations based on the decisions made by the reinforcement learning model. For example, the action information corresponding to the largest Q-value is selected as the action information of the first time unit.

[0059] Step 130: After scheduling the GPU to execute the operation indicated by the action information of the first time unit, determine the state information of the second time unit, which is the time unit after the first time unit.

[0060] In some embodiments, the second time unit is the next time unit after the first time unit. The resource management module can schedule the GPU to execute this action information. For example, if the action information of the first time unit is to allocate a specific number of computing units to a task, the resource management module will call the interface function of the GPU task scheduler to allocate the specified number of computing units. For instance, if it is decided to allocate 10 computing units to a deep learning task, the resource management module will send an instruction to the GPU task scheduler to allocate 10 computing units to the task. If the decision is to adjust the memory allocation ratio, the resource management module will interact with the memory management unit to modify the memory allocation parameters. For example, adjusting the memory allocation ratio of a task from 30% to 40%. If the decision is to pause or resume the resource allocation of a task, the resource management module will implement the corresponding operation by controlling the resource allocation switch, such as pausing the GPU resource allocation of a graphics rendering task and releasing the resources it occupies.

[0061] After executing the action information (i.e., GPU resource allocation) of the first time unit, the resource management module needs to interact with the GPU driver to collect GPU information for the second time unit; on the other hand, it needs to interact with at least one task (i.e., the application layer) to collect task information for the second time unit.

[0062] Please refer to Figure 2This diagram illustrates a possible implementation of the system information interaction in this application. The system includes a GPU, an application layer (i.e., tasks), a resource management module, and a reinforcement learning model. The application layer can send task information, including resource request information, to the resource management module. The resource management module interacts with the GPU, schedules the GPU to perform resource allocation operations, and feeds back state information (including task information and GPU information) to the reinforcement learning model module. The reinforcement learning model module makes decisions based on the received state information, generates action information (i.e., resource allocation strategy), and passes it to the resource management module. Specifically, when the application starts, it generates resource request data (i.e., task information) based on its own characteristics and needs (such as the structure of the deep learning model, the complexity of the graphics rendering scene, etc.), which may include application type (i.e., task type), priority, resource requests, etc. The application transmits this request data to the resource management module, which forwards the information to the reinforcement learning model. The model needs to consider the actual needs of the application (i.e., tasks) when making resource allocation decisions. Simultaneously, the reinforcement learning model feeds back the resource allocation results to the application. The application adjusts the execution method of computational tasks based on the allocation results. For example, it adjusts the data parallelism or batch size in deep learning, and the rendering parameters in graphics rendering, to adapt to the allocated resources, and then starts the application's computational tasks. Furthermore, during operation, the application periodically collects task performance metrics (such as task completion progress, frame rate, and computational accuracy) and transmits this performance data to the reinforcement learning model, providing feedback information.

[0063] In the aforementioned method, the resource management module collaborates closely with the reinforcement learning model. The resource management module collects GPU and task information and provides it to the reinforcement learning model as a decision-making basis. Simultaneously, the resource allocation decisions (i.e., action information) output by the reinforcement learning model are executed by the resource management module, which schedules the GPUs to perform these decisions. This interaction mechanism ensures timely information transmission and effective resource allocation. Specifically, the resource management module can obtain the latest status information of GPU and task information in real time and transmit it to the reinforcement learning model. After the model makes an accurate decision based on this information, the resource management module quickly executes the allocation operation, enabling the entire system to respond rapidly to changes in resource demand. Effective interaction between the resource management module and the reinforcement learning model is a crucial element in achieving the invention's objectives. It ensures the smoothness and accuracy of the resource allocation process, helps improve resource utilization, enhances overall system performance, and strengthens the adaptability of resource allocation. Furthermore, this application emphasizes the integration of the application (task) with the resource allocation system. The application sends detailed resource requests to the resource management module upon startup and provides performance metrics during operation. This integration allows the resource allocation system to better understand the application's (task's) needs and operational status, thereby making more reasonable resource allocation decisions.

[0064] Step 140: Adjust the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, to obtain the trained reinforcement learning model.

[0065] In reinforcement learning, the loss function is a key metric for evaluating the difference between model predictions and actual results. Minimizing the loss function can optimize the parameters of a reinforcement learning model, enabling it to better learn the dynamics of the environment and optimal resource allocation strategies. For specific model training methods, please refer to the relevant content below.

[0066] In summary, the technical solution provided in this application, by using a reinforcement learning model to determine the allocation method of GPU resources, can obtain the GPU state and task state in real time, thereby responding promptly to the dynamic demands of tasks for GPU resources and improving resource utilization efficiency. Furthermore, determining the GPU resource allocation method based on the state of at least one task enhances the adaptability of resource allocation to task demands, making GPU resource allocation more rational.

[0067] The following describes the training method for the reinforcement learning model, that is, step 140 above can be implemented as the following steps 141-142. Figure 1 (Not shown in the image).

[0068] Step 141: Determine reward information based on the status information of the second time unit and the action information of the first time unit. The reward information is used to indicate the execution effect of the action information.

[0069] The execution effect is used to indicate the impact of executing action information on system performance and task completion, including but not limited to whether the task is completed on time, whether resource utilization is improved, whether task latency is reduced, and whether the overall system efficiency is improved. This application does not limit this. In this application, the reward information is the reward information of the first time unit. In reinforcement learning models, reward information is a way to evaluate the effectiveness of actions taken by the GPU (i.e., action information). Specifically, reward information is a quantitative indicator used to indicate the execution effect of action information and to evaluate the merits of the resource allocation strategy adopted by the reinforcement learning model. For example, after executing the action information of the first time unit, if an application meets its performance requirements (such as task completion time, frame rate, etc.) and the overall utilization of GPU resources is high, a higher reward is given; conversely, if resource waste occurs or the application performance is substandard, a lower reward or even a penalty is given. Through continuous trial and error and accumulation of reward feedback, the reinforcement learning model can gradually learn to take the optimal action in different states to maximize the accumulated reward. In other words, the overall goal of the reinforcement learning algorithm is to maximize the accumulated reward information by continuously learning and optimizing the resource allocation strategy. Specifically, while meeting the performance requirements of each task (such as task completion time, frame rate, etc.), it is necessary to improve the overall utilization of GPU resources and reduce resource waste.

[0070] In some embodiments, after determining the reward information, the experience tuple (s, a, r, s') consisting of the state information s of the first time unit, the action information a of the first time unit, the reward information r, and the state information s' of the second time unit is stored in the experience replay buffer. The experience replay buffer is a queue structure used to store experience tuples (state, action, reward, next state) generated during system operation. Specifically, during system operation, the state, action, reward, and next state information of each resource allocation are collected to form experience tuples. These experience tuples are then transmitted to the experience replay buffer. When the experience replay buffer is full, it is replaced according to a set strategy (e.g., first-in, first-out). When training is needed later, a batch of experience tuple data is randomly sampled from the experience replay buffer and transmitted to the reinforcement learning model for training. The size of the buffer can be set, for example, to 10,000 experience data points. Initially, the buffer is empty, waiting for subsequent experience tuple data to fill it. If the experience replay buffer is full, the oldest stored experience tuple is deleted according to the first-in, first-out principle to make room for new experience tuple data. A batch of experience tuples can be periodically sampled from the experience replay buffer; for example, 32 experience tuples can be sampled every 100 time steps. These sampled experience tuples are used to train a reinforcement learning model, allowing the model to learn from past experiences and continuously optimize its decision-making strategy. For details, please refer to the corresponding description below.

[0071] Step 142: Based on the loss function value calculated using reward information, state information of the first time unit, state information of the second time unit, and action information of the first time unit, adjust the parameters of the reinforcement learning model to obtain the trained reinforcement learning model.

[0072] The above method, by explicitly defining reward information, makes the core training objective of the reinforcement learning model to continuously optimize and learn resource allocation strategies in order to maximize the accumulation of reward information, thereby ensuring the rationality of reinforcement learning resource allocation after training.

[0073] In some embodiments, adjusting the parameters of a reinforcement learning model based on a loss function value calculated using reward information, state information of a first time unit, state information of a second time unit, and action information of the first time unit to obtain a trained reinforcement learning model includes: determining a predicted Q-value corresponding to the action information of the first time unit using the reinforcement learning model based on the state information of the first time unit, wherein the predicted Q-value indicates an estimated value of the cumulative reward information expected to be obtained after executing the action information; determining a predicted Q-value corresponding to the action information of the second time unit using the reinforcement learning model based on the state information of the second time unit; determining a target Q-value based on the predicted Q-value corresponding to the action information of the second time unit and the reward information; and adjusting the parameters of the reinforcement learning model to minimize the difference between the predicted Q-value corresponding to the action information of the first time unit and the target Q-value to obtain a trained reinforcement learning model.

[0074] The predicted Q-value measures the expected long-term reward after performing a certain action. The loss function value reflects the difference between the model's predicted Q-value and the actual target Q-value, i.e., the model's prediction accuracy. A smaller loss function value means that the model's prediction is closer to the actual cumulative reward, indicating better model performance. In some embodiments, the reinforcement learning model determines the predicted Q-values ​​corresponding to multiple action information in the second time unit based on the state information of the second time unit; the action information corresponding to the largest predicted Q-value is determined as the action information of the second time unit.

[0075] In some embodiments, the sum of the predicted Q-value corresponding to the action information of the second time unit and the reward information is determined as the target Q-value. In some embodiments, a discount factor is determined as the product of the predicted Q-value corresponding to the action information of the second time unit, wherein the discount factor is used to measure the importance of future rewards to the current decision; the sum of this product and the reward information is determined as the target Q-value. In some embodiments, the predicted Q-value is calculated based on the parameters of the current reinforcement learning model, and then the target Q-value is calculated according to the Bellman equation. The calculation of the target Q-value is based on the maximum Q-value estimate of the next state (i.e., the predicted Q-value corresponding to the action information of the second time unit) and the currently obtained reward value, as shown in Formula 1 below:

[0076] Q target =r + γ × max a′ Q(s′,a′;θ target )(Formula 1)

[0077] Where s′ represents the state information of the second time unit, a′ represents the action information of the second time unit, and max a′ Q(s′,a′;θ targetLet γ be the predicted Q-value corresponding to the action information in the second time unit, r be the reward information, and γ be a discount factor used to measure the importance of future rewards to the current decision, with a value between 0 and 1. When it is close to 0, the agent pays more attention to the current immediate reward and less attention to future rewards; as it gradually approaches 1, the agent gives higher weight to future rewards and considers the impact of the decision in the longer term. target Let θ be the target Q value. target These are the parameters of the target network, which is used to calculate a stable target Q-value. The network parameters of the trained reinforcement learning model can be periodically copied into the target network.

[0078] The above method adjusts the model parameters by minimizing the difference between the predicted Q value and the target Q value, enabling the model to gradually optimize the decision-making strategy.

[0079] In some embodiments, before adjusting the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit to obtain the trained reinforcement learning model, the method further includes: sampling M empirical tuples from N empirical tuples, each empirical tuple including the state information of the first time unit, the action information of the first time unit, reward information, and the state information of the second time unit, where the reward information is used to indicate the execution effect of the action information, N is a positive integer, and M is a positive integer less than or equal to N; adjusting the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit to obtain the trained reinforcement learning model. The learning model includes: determining the predicted Q-value corresponding to the action information of the first time unit in each experience tuple using a reinforcement learning model based on the state information of the first time unit in each experience tuple; the predicted Q-value is used to indicate the estimated value of the cumulative reward information that can be obtained after executing the action information; determining the predicted Q-value corresponding to the action information of the second time unit in each experience tuple using a reinforcement learning model based on the state information of the second time unit in each experience tuple; determining the target Q-value based on the predicted Q-value corresponding to the action information of the second time unit in each experience tuple and the reward information; and adjusting the parameters of the reinforcement learning model to minimize the difference between the predicted Q-value corresponding to the action information of the first time unit in each experience tuple, thereby obtaining the trained reinforcement learning model.

[0080] During system operation, historical data on GPU resource allocation is continuously collected, including the state information of the current time unit (i.e., the state information of the first time unit), the action information of the current time unit (i.e., the action information of the first time unit), the reward information of the current time unit (i.e., the reward information of the first time unit), and the state information of the next time unit (i.e., the state information of the second time unit). This data is continuously accumulated in the experience replay buffer, forming a rich training dataset. By collecting a large amount of resource allocation experience under different states, a diverse data source is provided for the training of the reinforcement learning model, which helps the model learn more comprehensive and effective resource allocation strategies.

[0081] In some embodiments, M empirical tuples are sampled from the N empirical tuples included in the empirical replay buffer and used as training samples. For each training sample, a predicted Q-value is calculated based on the parameters of the current reinforcement learning model, and then a target Q-value is calculated according to the Bellman equation. Please refer to the corresponding description above for specific steps.

[0082] The loss function L(θ) between the predicted Q-value and the target Q-value can be calculated using the mean squared error loss function, as shown in Formula 2 below:

[0083]

[0084] Where M is the number of samples, θ is the network parameters of the current reinforcement learning model, i is the i-th empirical tuple sampled, and s i For the state information of the first time unit, a i For the action information of the first time unit, Q(s) i ,a i ;θ) is the predicted Q value corresponding to the first time unit, Q target (s i ,a i Let Q be the target Q-value. The gradient of the network parameters is calculated based on the loss function, and the gradient descent algorithm is used to update the network parameters, making the predicted Q-value output by the network more accurately approximate the Q-value under the optimal policy. During the update process, the learning rate controls the step size of the parameter updates and needs to be adjusted according to the actual situation. It starts with a large value and gradually decreases as training progresses to ensure stable convergence of the model. In some embodiments, training stops when the difference between the predicted Q-value corresponding to the action information in the first time unit and the target Q-value is less than a certain threshold, resulting in the trained reinforcement learning model. In some embodiments, training stops when the number of training epochs exceeds a certain threshold, resulting in the trained reinforcement learning model.

[0085] The above method involves the reinforcement learning model collecting new state information and task performance feedback after each resource allocation decision and execution, calculating reward values, and storing the corresponding experience tuple data in an experience replay buffer. Subsequently, data is periodically sampled from the experience replay buffer for training and updating model parameters, thereby continuously optimizing resource allocation strategies based on the past performance of the application / task. For example, for tasks with periodic changes in resource requirements, the model can proactively plan resource allocation based on historical resource usage patterns.

[0086] In some embodiments, the use of the reinforcement learning model may involve the following steps: obtaining first state information, which indicates the state of the GPU and the state of at least one task associated with the GPU; generating predicted Q values ​​for K action information based on the first state information using the trained reinforcement learning model, where K is an integer greater than 1, and the predicted Q values ​​indicate the estimated value of the cumulative reward information that can be obtained after executing the action information; and determining the target action information from the K action information based on the predicted Q values ​​of the K action information.

[0087] In some embodiments, the action information corresponding to the maximum predicted Q-value is determined as the target action information. During the model usage phase, the predicted Q-value of each action information is first calculated based on the input first state information. Then, the resource management module selects the action with the largest Q-value based on these predicted Q-values ​​and executes the corresponding resource allocation operation. When faced with the GPU resource requirements of different tasks, the model calculates the predicted Q-values ​​of action information such as allocating different numbers of computing units, adjusting the video memory ratio, pausing or resuming GPU resource allocation for a task, etc. The resource management module then selects the optimal action information from these, thereby achieving accurate resource allocation decisions to meet the needs of the task and improve the utilization of GPU resources.

[0088] The above method allows the model to learn the relationship between state information and actions, as well as the corresponding expected rewards, during the training phase. During the usage phase, the model uses the knowledge learned in training to evaluate the current state information and select the action with the highest expected reward. This ensures that the system can make optimal decisions in actual operation, thereby improving resource utilization efficiency and task execution performance.

[0089] The following describes the specific method for determining reward information.

[0090] In some embodiments, determining reward information based on the state information of the second time unit and the action information of the first time unit includes: determining GPU resource usage information based on the state information of the second time unit, wherein the resource usage information is used to indicate the usage of GPU resources; determining performance achievement information for at least one task after scheduling the GPU to execute the operation indicated by the action information of the first time unit, wherein the performance achievement information is used to indicate the degree of deviation between the performance of at least one task and the expected performance; and determining reward information based on the resource usage information and the performance achievement information.

[0091] Resource usage information can be represented by quantifiable values. A higher value indicates a higher utilization rate of GPU resources (such as computing and memory resources), while a lower value indicates a lower utilization rate. Higher utilization (i.e., a higher resource usage value) means that GPU resources are being fully utilized, thus a larger reward value is tended. Conversely, lower utilization (i.e., a smaller resource usage value) means that the task is not being fully utilized, thus a smaller reward value is tended.

[0092] Performance attainment information can be represented by quantifiable values, indicating the degree to which at least one task's performance meets or fails to meet expectations. When at least one task's performance falls short of expectations—that is, when performance attainment information indicates the degree of failure—a larger value indicates a greater deviation between the task's performance and the expected performance, while a smaller value indicates a smaller deviation. A larger deviation (i.e., a larger performance attainment value) indicates worse task performance or a greater degree of failure, thus a smaller reward value is preferred. This indicates that the action information of the current first time unit has caused a performance decrease, requiring parameter adjustments to optimize the model and select better action information. Conversely, a smaller deviation (i.e., a smaller performance attainment value) indicates that the task's performance is closer to or less than expected, thus a larger reward value is preferred compared to a larger performance attainment value. In some embodiments, where performance compliance information is used to indicate the degree to which the performance of at least one task is substandard, the negative of the difference between the value of resource usage information and the value of performance compliance information is determined as reward information.

[0093] Understandably, when the performance of at least one task reaches or exceeds the expected performance, the reward information should be maximized compared to when the performance of at least one task fails to meet the expected performance. This means that the current action information helps improve task performance so as to make better decisions in similar situations in the future.

[0094] The above methods use resource usage information to reflect the actual use of GPU resources, such as the utilization rate of computing and video memory resources. This helps the model understand whether current resources are being fully utilized and whether there are any idle periods. Performance compliance information reflects the deviation between the actual and expected performance of a task, such as whether the task's execution time, throughput, or latency meet expectations. Combining these two types of information allows for a more comprehensive assessment of the overall system status, thus avoiding the limitations of using a single metric to determine reward information. For example, even if GPU resources are fully utilized, if task performance does not meet expectations, simply considering resource utilization may lead to incorrect resource allocation decisions.

[0095] In some embodiments, resource usage information includes resource utilization information and resource waste information. Resource utilization information indicates the allocated resources in the GPU resources, and resource waste information indicates the unallocated resources in the GPU resources. Determining GPU resource usage information based on the state information of the second time unit includes: determining the resource occupancy of a first task based on the state information of the second time unit, where the first task refers to at least one task for which GPU resources have been allocated, and the resource occupancy quantity refers to the amount of GPU resources occupied by the first task; determining resource utilization information based on the sum of the resource occupancy quantity and the total GPU resources; and determining resource waste information based on the sum of the number of idle resources and the total GPU resources, where the number of idle resources refers to the number of unallocated resources in the GPU resources.

[0096] In some embodiments, reward information is determined based on resource utilization information, resource waste information, and performance achievement information. In some embodiments, the difference between resource utilization information and resource waste information is determined; the reward information is determined based on the difference between this difference and the performance achievement information, wherein the performance achievement information is used to indicate the degree to which the performance of at least one task is substandard. As shown in Formula 3 below:

[0097]

[0098] Where R represents reward information, U represents GPU resource usage information, W represents resource waste information, and D represents reward information. i The performance compliance information for the i-th task is defined by α, β, and γ, which are weighting parameters that can be set based on the experience of relevant technical personnel. This application does not impose any restrictions on them.

[0099] In some embodiments, the ratio of resource usage to the total GPU resources is determined as resource utilization information. In some embodiments, resource utilization information includes computing resource utilization information and video memory resource utilization information. Computing resource utilization information is used to indicate the allocated computing resources in the GPU resources, and video memory resource utilization information is used to indicate the allocated video memory resources in the GPU resources. In some embodiments, computing resource utilization information is determined based on the computing resource usage of the first task and the total GPU computing resources. In some embodiments, the ratio of computing resource usage of the first task to the total GPU computing resources is determined as computing resource utilization information. In some embodiments, video memory resource utilization information is determined based on the video memory usage of the first task and the total GPU video memory resources. In some embodiments, the ratio of video memory usage of the first task to the total GPU video memory resources is determined as video memory resource utilization information. In some embodiments, the sum of computing resource utilization information and video memory resource utilization information is determined as resource utilization information. As shown in Formula 4 below:

[0100]

[0101] Where U represents GPU resource usage information, C i C represents the computational resource consumption of the first task i. total M represents the total computing resources of the GPU. i M represents the video memory usage of the first task i. tltal This represents the total GPU memory resources.

[0102] In some embodiments, the ratio of the number of free resources to the total number of GPU resources is determined as resource waste information. In some embodiments, resource waste information includes computational resource waste information and video memory resource waste information. Computational resource waste information indicates the amount of unallocated computational resources in the GPU resources, and video memory resource waste information indicates the amount of unallocated video memory resources in the GPU resources. In some embodiments, computational resource waste information is determined based on the number of free computational resources and the total number of GPU computational resources. In some embodiments, the ratio of the number of free computational resources to the total number of GPU computational resources is determined as computational resource waste information. In some embodiments, video memory resource waste information is determined based on the amount of free video memory resources and the total number of GPU video memory resources. In some embodiments, the ratio of the number of free video memory resources to the total number of GPU video memory resources is determined as video memory resource waste information. In some embodiments, resource waste information is determined based on computational resource waste information and video memory resource waste information. In some embodiments, the sum of computational resource waste information and video memory resource waste information is determined as resource waste information, as follows:

[0103] As shown in Equation 5:

[0104]

[0105] Where W represents GPU resource waste information, C idle M represents the amount of unallocated computing resources. idle C represents the amount of unallocated video memory resources. total M represents the total computing resources of the GPU. total This represents the total GPU memory resources.

[0106] The aforementioned method, in determining reward information, comprehensively considers GPU resource utilization information, GPU resource waste information, and task performance achievement information, and balances these factors through adjustable weight coefficients. Compared to existing technologies that lack comprehensive evaluation methods for resource allocation effectiveness, the reward information in this application can more effectively guide reinforcement learning models to learn optimal resource allocation strategies. For example, when resource utilization is high and task performance is good, the model receives a positive reward, prompting it to maintain or optimize the current resource allocation method; conversely, when there is significant resource waste or task performance fails to meet targets, the model is penalized, thereby adjusting its resource allocation strategy. The optimized reward function is a key driving force for reinforcement learning models to continuously improve resource allocation decisions, playing a crucial role in achieving the invention's objectives of improving resource utilization, enhancing task execution efficiency, and optimizing overall system performance. It can guide the model to find the optimal balance between resource utilization and task performance in complex and ever-changing application scenarios.

[0107] In some embodiments, after scheduling the GPU to execute the operation indicated by the action information of the first time unit, determining performance compliance information for at least one task includes: determining the actual completion time of a second task, wherein the second task refers to a task in the at least one task that is in a completed state after scheduling the GPU to execute the operation indicated by the action information of the first time unit; and determining performance compliance information based on the difference between the actual completion time and the expected completion time.

[0108] The second task indicates a task that has completed execution and no longer requires further processing after the action information of the first time unit was executed by the scheduling GPU. The completion status of this second task is usually confirmed by the system monitoring mechanism to ensure that its execution has ended and the corresponding GPU resources can be released or reallocated to other tasks that have not yet completed.

[0109] In some embodiments, performance compliance information is used to indicate the degree to which task performance fails to meet standards. When the actual completion time is less than the expected completion time, the performance compliance information is a first value; when the actual completion time is greater than the expected completion time, the performance compliance information is a second value. The first value is less than the second value, and the first and second values ​​are set based on the experience of relevant technical personnel.

[0110] In some embodiments, performance compliance information is used to indicate the degree to which task performance fails to meet standards. When the actual completion time is longer than the expected completion time, performance compliance information is determined based on the difference between the actual and expected completion times. In some embodiments, when the actual completion time is longer than the expected completion time, the difference between the actual and expected completion times is determined; the ratio of this difference to the expected completion time is used as performance compliance information, as shown in Formula 6 below:

[0111]

[0112] Among them, D i T represents the degree to which task i's performance is substandard. i {actual} T is the actual completion time of task i. i {expected} α is the expected completion time, and β and γ are the corresponding weighting coefficients, which can be adjusted according to the actual situation. This application does not limit them.

[0113] In some embodiments, if the actual completion time is greater than the expected completion time, the difference between the actual completion time and the expected completion time is determined; if the difference is less than or equal to a certain threshold, the task is considered to have met the target. If the difference is greater than a certain threshold, the task is considered to have failed the target, and the ratio of the difference to the expected completion time is used as performance compliance information. As shown in Formula 7 below:

[0114]

[0115] Here, ∈1 is an acceptable time deviation threshold. When the deviation between the actual completion time and the expected completion time is within the threshold range, the task performance is considered to be up to standard.

[0116] In some embodiments, after scheduling the GPU to execute the operation indicated by the action information of the first time unit, determining the performance compliance information of at least one task includes: determining the actual frame rate of a third task, wherein the third task refers to a task in the at least one task that has been allocated GPU resources and involves screen rendering; and determining the performance compliance information based on the difference between the actual frame rate and the expected frame rate.

[0117] In some embodiments, frame rate stability is crucial for graphics rendering or video processing tasks. Performance compliance information is used to indicate the degree to which task performance fails to meet requirements. When the actual frame rate is greater than or equal to the expected frame rate, the performance compliance information is a third value; when the actual frame rate is less than the expected frame rate, the performance compliance information is a fourth value, where the third value is less than the fourth value. The third and fourth values ​​are set based on the experience of those skilled in the art. In some embodiments, when the actual frame rate is less than the expected frame rate, the performance compliance information is determined based on the difference between the expected and actual frame rates. In some embodiments, when the actual frame rate is less than the expected frame rate, the difference between the actual and expected frame rates is determined; the ratio of this difference to the expected frame rate is used as the performance compliance information.

[0118] In some embodiments, when the actual frame rate is lower than the expected frame rate, the difference between the actual and expected frame rates is determined; if this difference is less than or equal to a certain threshold, the task is considered to have met the target. If the difference is greater than a certain threshold, the task is considered to have failed the target, and the ratio of this difference to the expected frame rate is used as performance compliance information. As shown in Formula 8 below:

[0119]

[0120] Among them, the expected frame rate is Actual frame rate ∈2 is an acceptable frame rate deviation threshold. When the deviation between the actual frame rate and the expected frame rate is within the threshold range, the task performance is considered to be up to standard.

[0121] like Figure 3 The diagram illustrates a flowchart of a GPU resource allocation method based on a reinforcement learning model, as shown in one possible implementation of this application. It may include the following steps S1 to S3.

[0122] Step S1: Input the state information into the reinforcement learning model. The state information may include GPU information and task information. The task information may include resource request information sent by the upper-layer application (i.e., the task), such as the type and quantity of GPU resources requested.

[0123] Step S2: Determine reward information and update resource allocation scheme. Based on resource usage information and performance target achievement information sent by the upper-layer application (i.e., the task), determine reward information. The reinforcement learning model updates the GPU resource allocation method based on the reward information, that is, determines the predicted Q value for each action.

[0124] Step S3: Select the resource allocation method with the highest expected return for the task, that is, select the action information corresponding to the maximum predicted Q value as the resource allocation method.

[0125] The methods described above, in order to evaluate whether the performance of a task meets the standards, especially for tasks involving time sensitivity and frame rate requirements, propose various approaches to determine performance compliance information. These methods quantify the degree of deviation in task performance by comparing the actual completion time with the expected completion time, or the actual frame rate with the expected frame rate.

[0126] The following are embodiments of the apparatus of this application. For details not described in the apparatus embodiments of this application, please refer to the method embodiments above.

[0127] Please refer to Figure 4 This diagram illustrates a block diagram of a resource allocation apparatus provided in one possible implementation of this application. The apparatus has the function of implementing the above-described resource allocation method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 400 may include: a first acquisition module 410, a generation module 420, a determination module 430, and an adjustment module 440.

[0128] The first acquisition module 410 is used to acquire the status information of the first time unit, the status information being used to indicate the status of the GPU and the status of at least one task related to the GPU.

[0129] The generation module 420 is used to generate action information of the first time unit based on the state information of the first time unit through a reinforcement learning model. The action information is used to indicate the allocation method of GPU resources corresponding to the at least one task.

[0130] The determination module 430 is used to determine the state information of the second time unit after scheduling the GPU to execute the operation indicated by the action information of the first time unit, wherein the second time unit is the time unit after the first time unit.

[0131] The adjustment module 440 is used to adjust the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, so as to obtain the trained reinforcement learning model.

[0132] In some embodiments, the adjustment module 440 is used to determine reward information based on the state information of the second time unit and the action information of the first time unit, wherein the reward information is used to indicate the execution effect of the action information; and to adjust the parameters of the reinforcement learning model based on the loss function value calculated based on the reward information, the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, to obtain the trained reinforcement learning model.

[0133] In some embodiments, the adjustment module 440 is configured to determine the resource usage information of the GPU based on the state information of the second time unit, the resource usage information being used to indicate the usage of GPU resources; after scheduling the GPU to execute the operation indicated by the action information of the first time unit, determine the performance target information of the at least one task, the performance target information being used to indicate the degree of deviation between the performance of the at least one task and the expected performance; and determine the reward information based on the resource usage information and the performance target information.

[0134] In some embodiments, the resource usage information includes resource utilization information and resource waste information. The resource utilization information indicates the allocated resources in the GPU resources, and the resource waste information indicates the unallocated resources in the GPU resources. The adjustment module 440 is configured to determine the resource occupancy of a first task based on the status information of the second time unit. The first task refers to the task among the at least one task that has been allocated GPU resources, and the resource occupancy quantity refers to the quantity of GPU resources occupied by the first task. The resource utilization information is determined based on the resource occupancy quantity and the sum of the GPU resources. The resource waste information is determined based on the number of idle resources and the sum of the GPU resources, where the number of idle resources refers to the number of unallocated resources in the GPU resources.

[0135] In some embodiments, the adjustment module 440 is used to determine the actual completion time of a second task, wherein the second task refers to a task in the at least one task that is in a completed state after scheduling the GPU to execute the operation indicated by the action information of the first time unit; and to determine the performance compliance information based on the difference between the actual completion time and the expected completion time; or, to determine the actual frame rate of a third task, wherein the third task refers to a task in the at least one task that has been allocated GPU resources and involves screen rendering; and to determine the performance compliance information based on the difference between the actual frame rate and the expected frame rate.

[0136] In some embodiments, the adjustment module 440 is configured to: determine, using the reinforcement learning model, a predicted Q-value corresponding to the action information of the first time unit based on the state information of the first time unit, wherein the predicted Q-value indicates an estimated value of the cumulative reward information expected to be obtained after executing the action information; determine, using the reinforcement learning model, a predicted Q-value corresponding to the action information of the second time unit based on the state information of the second time unit; determine a target Q-value based on the predicted Q-value corresponding to the action information of the second time unit and the reward information; and adjust the parameters of the reinforcement learning model to minimize the difference between the predicted Q-value corresponding to the action information of the first time unit and the target Q-value, thereby obtaining the trained reinforcement learning model.

[0137] In some embodiments, the status information includes GPU information and task information; the GPU information includes at least one of the following: GPU computing resource information and GPU memory resource information; the task information includes at least one of the following: resource request information for each of the at least one task, type information for each of the at least one task, priority information for each of the at least one task, and historical resource usage information for each of the at least one task; wherein, the resource request information is used to indicate the type and quantity of GPU resources requested by the task, the type information is used to indicate the functional classification of the task, the priority information is used to indicate the importance of the task in the system, and the historical resource usage information is used to indicate the historical usage of GPU resources by the task.

[0138] In some embodiments, the action information includes at least one of the following: task allocation information, task allocation adjustment information, and pause / resume information; wherein the task allocation information is used to indicate the type and quantity of GPU resources allocated to the at least one task respectively, the task allocation adjustment information is used to indicate the type and quantity of GPU resources allocated to the at least one task respectively after adjustment, and the pause / resume information is used to indicate pausing the resource allocation of the at least one task and resuming the resource allocation of the paused task.

[0139] In some embodiments, the device 400 further includes: a sampling module ( Figure 4 (Not shown in the image).

[0140] The sampling module is used to sample M experience tuples from N experience tuples. Each experience tuple includes the state information of the first time unit, the action information of the first time unit, the reward information, and the state information of the second time unit. The reward information is used to indicate the execution effect of the action information. N is a positive integer, and M is a positive integer less than or equal to N.

[0141] The adjustment module 400 is used to determine, through the reinforcement learning model, the predicted Q-value corresponding to the action information of the first time unit in each experience tuple, based on the state information of the first time unit in each experience tuple. The predicted Q-value is used to indicate the estimated value of the cumulative reward information that can be obtained after executing the action information. It also determines, through the reinforcement learning model, the predicted Q-value corresponding to the action information of the second time unit in each experience tuple, based on the state information of the second time unit in each experience tuple. Furthermore, it determines a target Q-value based on the predicted Q-value corresponding to the action information of the second time unit in each experience tuple and the reward information. This minimizes the difference between the predicted Q-value corresponding to the action information of the first time unit in each experience tuple and the target Q-value, thereby adjusting the parameters of the reinforcement learning model to obtain the trained reinforcement learning model.

[0142] In some embodiments, the device 400 further includes: a second acquisition module ( Figure 4 (Not shown in the image).

[0143] The second acquisition module is configured to acquire first state information, which indicates the state of the GPU and the state of at least one task related to the GPU; generate predicted Q values ​​for K action information based on the first state information using the trained reinforcement learning model, where K is an integer greater than 1, and the predicted Q values ​​indicate the estimated value of the cumulative reward information that can be obtained after executing the action information; and determine target action information from the K action information based on the predicted Q values ​​of the K action information.

[0144] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0145] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0146] Please refer to Figure 5 The diagram shows a structural block diagram of a computer device 500 provided in one embodiment of this application.

[0147] Typically, computer device 500 includes a processor 510 and a memory 520.

[0148] Processor 510 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 510 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 510 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 510 may integrate a GPU, which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 510 may also include an AI processor for handling computational operations related to machine learning.

[0149] The memory 520 may include one or more computer-readable storage media, which may be non-transitory. The memory 520 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 520 are used to store a computer program configured to be executed by one or more processors to implement the resource allocation method described above.

[0150] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the computer device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0151] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the resource allocation method described above.

[0152] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0153] In some embodiments, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium, the processor reading from the computer-readable storage medium and executing the computer program to implement the above-described resource allocation method.

[0154] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0155] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A resource allocation method, characterized in that, The method includes: The status information of a first time unit is obtained, the status information indicating the status of the graphics processing unit (GPU) and the status of at least one task related to the GPU; wherein, the status information includes GPU information and task information; the GPU information includes the GPU's computing resource information and video memory resource information obtained in real time through an interface provided by the GPU driver; the task information includes at least one of the following: resource request information, type information, priority information, and historical resource usage information for each of the at least one task; wherein, the resource request information indicates the type and quantity of GPU resources requested by the task, the type information indicates the functional classification of the task, the priority information indicates the importance of the task in the system, and the historical resource usage information indicates the historical usage of GPU resources by the task; The reinforcement learning model generates action information for the first time unit based on the state information of the first time unit. The action information is used to indicate the allocation method of GPU resources corresponding to the at least one task. The action information includes task allocation adjustment information and pause / resume information. The task allocation adjustment information is used to indicate the type and quantity of GPU resources allocated to the at least one task after adjustment. The pause / resume information is used to indicate pausing the resource allocation of the at least one task and resuming the resource allocation of the paused task. After scheduling the GPU to execute the operation indicated by the action information of the first time unit, the state information of the second time unit is determined, wherein the second time unit is the time unit following the first time unit; Based on the loss function value calculated using the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, the parameters of the reinforcement learning model are adjusted to obtain the trained reinforcement learning model. The loss function value is calculated based on reward information, which is determined according to resource usage information and performance achievement information. The resource usage information includes resource utilization information and resource waste information. The resource utilization information indicates the allocated resources in the GPU, and the resource waste information indicates the unallocated resources in the GPU. The performance achievement information is determined based on the difference between the actual completion time and the expected completion time of the task, or based on the difference between the actual frame rate and the expected frame rate of the task.

2. The method according to claim 1, characterized in that, The step of adjusting the parameters of the reinforcement learning model based on the loss function value calculated using the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit to obtain the trained reinforcement learning model includes: Based on the status information of the second time unit and the action information of the first time unit, reward information is determined, and the reward information is used to indicate the execution effect of the action information; Based on the loss function value calculated using the reward information, the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, the parameters of the reinforcement learning model are adjusted to obtain the trained reinforcement learning model.

3. The method according to claim 2, characterized in that, The step of determining reward information based on the state information of the second time unit and the action information of the first time unit includes: Based on the status information of the second time unit, the resource usage information of the GPU is determined, and the resource usage information is used to indicate the usage of the GPU resources; After scheduling the GPU to execute the operation indicated by the action information of the first time unit, performance compliance information of the at least one task is determined, the performance compliance information being used to indicate the degree of deviation between the performance of the at least one task and the expected performance; The reward information is determined based on the resource usage information and the performance achievement information.

4. The method according to claim 3, characterized in that, The step of determining the GPU's resource usage information based on the state information of the second time unit includes: Based on the status information of the second time unit, the resource usage of the first task is determined. The first task refers to the task among the at least one task that has been allocated the GPU resources. The resource usage refers to the amount of GPU resources used by the first task. The resource utilization information is determined based on the amount of resources occupied and the total amount of GPU resources. The resource waste information is determined based on the number of idle resources and the total number of GPU resources, where the number of idle resources refers to the number of unallocated resources in the GPU resources.

5. The method according to claim 3, characterized in that, After scheduling the GPU to execute the operation indicated by the action information of the first time unit, determining the performance compliance information of the at least one task includes: Determine the actual completion time of the second task, where the second task refers to the task in the at least one task that is in a completed state after the GPU is scheduled to execute the operation indicated by the action information of the first time unit; determine the performance target achievement information based on the difference between the actual completion time and the expected completion time; or, Determine the actual frame rate of the third task, where the third task refers to the task among the at least one task that has been allocated GPU resources and involves screen rendering; determine the performance compliance information based on the difference between the actual frame rate and the expected frame rate.

6. The method according to claim 2, characterized in that, The step of adjusting the parameters of the reinforcement learning model based on the loss function value calculated using the reward information, the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit to obtain the trained reinforcement learning model includes: The reinforcement learning model determines the predicted Q value corresponding to the action information of the first time unit based on the state information of the first time unit. The predicted Q value is used to indicate the estimated value of the cumulative reward information that can be obtained after executing the action information. The reinforcement learning model determines the predicted Q value corresponding to the action information of the second time unit based on the state information of the second time unit. The target Q value is determined based on the predicted Q value corresponding to the action information of the second time unit and the reward information; The parameters of the reinforcement learning model are adjusted to minimize the difference between the predicted Q-value corresponding to the action information of the first time unit and the target Q-value, thereby obtaining the trained reinforcement learning model.

7. The method according to claim 1, characterized in that, The action information also includes: task allocation information; The task allocation information is used to indicate the type and quantity of GPU resources allocated to each of the at least one task.

8. The method according to claim 1, characterized in that, Before adjusting the parameters of the reinforcement learning model based on the loss function value calculated using the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, to obtain the trained reinforcement learning model, the method further includes: M experience tuples are sampled from N experience tuples. Each experience tuple includes the state information of the first time unit, the action information of the first time unit, the reward information, and the state information of the second time unit. The reward information is used to indicate the execution effect of the action information. N is a positive integer, and M is a positive integer less than or equal to N. The step of adjusting the parameters of the reinforcement learning model based on the loss function value calculated using the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit to obtain the trained reinforcement learning model includes: The reinforcement learning model determines the predicted Q value corresponding to the action information of the first time unit in each experience tuple based on the state information of the first time unit in each experience tuple. The predicted Q value is used to indicate the estimated value of the cumulative reward information that can be obtained after executing the action information. The reinforcement learning model determines the predicted Q value corresponding to the action information of the second time unit in each of the experience tuples based on the state information of the second time unit in each of the experience tuples. The target Q value is determined based on the predicted Q value corresponding to the action information of the second time unit in each of the experience tuples and the reward information; The parameters of the reinforcement learning model are adjusted to minimize the difference between the predicted Q-value and the target Q-value corresponding to the action information of the first time unit in each empirical tuple, thereby obtaining the trained reinforcement learning model.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Obtain first state information, which is used to indicate the state of the GPU and the state of at least one task associated with the GPU; The trained reinforcement learning model generates predicted Q values ​​for each of the K action information based on the first state information, where K is an integer greater than 1. The predicted Q values ​​are used to indicate the estimated value of the cumulative reward information that can be obtained after performing the action information. Based on the predicted Q value of each of the K action information, the target action information is determined from the K action information.

10. A resource allocation device, characterized in that, The device includes: A first acquisition module is configured to acquire status information of a first time unit. The status information indicates the status of a graphics processing unit (GPU) and the status of at least one task associated with the GPU. The status information includes GPU information and task information. The GPU information includes computing resource information and video memory resource information of the GPU, acquired in real-time through an interface provided by the GPU driver. The task information includes at least one of the following: resource request information, type information, priority information, and historical resource usage information for each of the at least one task. The resource request information indicates the type and quantity of GPU resources requested by the task; the type information indicates the functional classification of the task; the priority information indicates the importance of the task in the system; and the historical resource usage information indicates the historical usage of GPU resources by the task. A generation module is used to generate action information for the first time unit based on the state information of the first time unit using a reinforcement learning model. The action information is used to indicate the allocation method of GPU resources corresponding to the at least one task. The action information includes task allocation adjustment information and pause / resume information. The task allocation adjustment information is used to indicate the type and quantity of GPU resources allocated to the at least one task after adjustment. The pause / resume information is used to indicate pausing the resource allocation of the at least one task and resuming the resource allocation of the paused task. The determination module is used to determine the state information of the second time unit after scheduling the GPU to execute the operation indicated by the action information of the first time unit, wherein the second time unit is the time unit after the first time unit; An adjustment module is used to adjust the parameters of the reinforcement learning model based on the loss function value calculated based on the state information of the first time unit, the state information of the second time unit, and the action information of the first time unit, to obtain the trained reinforcement learning model. The loss function value is calculated based on reward information, which is determined based on resource usage information and performance achievement information. The resource usage information includes resource utilization information and resource waste information. The resource utilization information indicates the allocated resources in the GPU resources, and the resource waste information indicates the unallocated resources in the GPU resources. The performance achievement information is determined based on the difference between the actual completion time and the expected completion time of the task, or based on the difference between the actual frame rate and the expected frame rate of the task.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as claimed in any one of claims 1 to 9.

Images