A method and apparatus for dynamically expanding and shrinking a GPU memory privatization pool

Abstract

This application provides a method, apparatus, device, and storage medium for dynamic scaling control of a private GPU memory pool based on deep reinforcement learning. It relates to the technical field of computer system resource management and deep reinforcement learning applications. The method constructs a private memory pool and uses a pre-trained deep reinforcement learning model to dynamically control its scaling. The method uses GPU utilization, memory usage, memory fragmentation rate, memory pool size, and allocation rate as the state space, outputting continuous action decisions of maintaining, expanding, or shrinking the pool, and adjusting the memory pool capacity through CUDA or PyTorch interfaces. During training, a multi-dimensional reward function is used to optimize the strategy model, including a comprehensive evaluation of GPU utilization deviation, memory fragmentation rate, over-limit penalty, action fluctuation, and execution success. This scheme improves the utilization efficiency of GPU memory resources, reduces memory waste and system fluctuations, and enhances the automation and adaptability of memory management.

Description

Technical Field

[0001] This invention relates to the technical field of computer system resource management and deep reinforcement learning applications, specifically to a method, storage medium, device, and equipment for dynamic expansion and contraction control of GPU memory privatization pools based on deep reinforcement learning. Background Technology

[0002] With the increasing complexity of GPU computing tasks, memory resource management has become a crucial factor affecting system performance and stability. Traditional memory management often employs static allocation and scaling strategies, which struggle to handle challenges such as multi-process concurrency, large load fluctuations, and heterogeneous resource requirements. Especially in high-concurrency scenarios, frequent memory allocation and deallocation can easily lead to memory fragmentation, reducing overall utilization and even causing OutOfMemoryError (OOM) errors, thus impacting task execution efficiency and system stability.

[0003] To alleviate these issues, the industry has attempted to introduce memory pooling mechanisms, using private memory pools to pre-allocate resources, reduce memory fragmentation, and improve allocation efficiency. However, the static configuration of private pools is ill-suited to dynamically changing workloads, easily leading to resource waste or strain. Existing solutions typically rely on manually setting expansion and contraction thresholds, lacking a comprehensive consideration of GPU utilization, fragmentation rate, and OOM risk, making it difficult to achieve a dynamic balance between resource utilization and system stability. Furthermore, traditional methods lack adaptability to runtime load changes, resulting in a sharp performance drop under sudden loads. Therefore, there is an urgent need for a dynamic memory pool management mechanism that is adaptive, multi-objective optimized, and possesses online learning capabilities to improve GPU resource utilization efficiency and system robustness. Summary of the Invention

[0004] Based on this, in order to solve the technical problems of low efficiency in the allocation of existing GPU memory resources and the inability to dynamically adapt to changes in the workload of deep learning tasks, resulting in memory waste or shortage, a method, storage medium, device and equipment for dynamic expansion and contraction control of GPU memory private pool based on deep reinforcement learning is proposed.

[0005] This invention protects a method for dynamically scaling up and down a private GPU memory pool based on deep reinforcement learning. The method constructs a private memory pool through a GPU memory management interface and automatically scales up and down the pool using a deep reinforcement learning model. The control method includes: acquiring the current running status information of the GPU, including GPU utilization, memory usage, memory fragmentation rate, memory pool size, and memory allocation rate; inputting the running status information into a pre-trained deep reinforcement learning policy model; outputting scaling up / down action decisions from the policy model in a continuous action space, including maintaining, expanding, or shrinking the memory pool size and the corresponding adjustment ratio; and dynamically scaling up and down the private GPU memory pool based on the scaling up / down action decisions.

[0006] Furthermore, the pre-training process of the deep reinforcement learning model includes constructing a reinforcement learning environment, which includes the action space and state space of the environment. The action space of the environment includes scaling types and adjustment ratios, where the scaling types include maintaining, expanding, or shrinking, and the adjustment ratio ranges from 0.5 to 2.0. The state space of the environment is a continuous state vector containing five dimensions, representing GPU utilization, memory usage, memory fragmentation rate, current memory pool size, and memory allocation rate, respectively. In each training step, the memory pool size is updated after the action is executed, and the actual capacity of the memory pool is set through the CUDA or PyTorch interface. The updated state information is obtained, and the reward value of the current action is calculated based on a multi-dimensional reward function, which includes a penalty for deviation in GPU utilization, a reverse reward for memory fragmentation rate, a penalty for exceeding memory limits, a penalty for action fluctuation, and a reward for successful execution.

[0007] Furthermore, when the GPU memory usage exceeds a preset threshold or the training steps reach the maximum number of steps, the current training round is marked as over.

[0008] Furthermore, the scaling action decision output by the policy model in the continuous action space includes: during the dynamic scaling control process, an online inference service continuously acquires the current running state information of the GPU and outputs scaling action decisions based on a pre-trained deep reinforcement learning policy model; the online inference service periodically performs state acquisition, action prediction, and memory pool adjustment operations by running class instances containing prediction logic; the class instance contains a circular buffer for recording historical states, and the maximum storage length of the circular buffer is 60 time units.

[0009] Furthermore, in each decision cycle, the actual capacity of the video memory pool is set through the CUDA or PyTorch interface, and the current state, action decision, and adjusted video memory pool size are recorded as log information. The execution of the action decision is achieved by parsing the action type and adjustment ratio, where the action type includes expansion, contraction, or maintenance. The expansion operation adjusts the video memory pool size through multiplication, and the contraction operation adjusts the video memory pool size through division.

[0010] Furthermore, obtaining the current GPU operating status information includes: obtaining the GPU device handle through the built-in library, and obtaining the GPU utilization, total video memory, used video memory, and free video memory based on the GPU device handle; obtaining the currently allocated video memory through the CUDA or PyTorch interface, and calculating the video memory fragmentation rate based on the difference between free video memory and allocated video memory; the calculated video memory fragmentation rate is compared with zero using a maximum function to ensure that the video memory fragmentation rate is non-negative; the current operating status information also includes the ratio of the video memory pool size to the total video memory, and the video memory allocation rate calculated based on the video memory allocation situation in the last 60 seconds.

[0011] Furthermore, the dynamic scaling of the GPU memory private pool based on scaling action decisions includes: setting a running mode selection mechanism during the dynamic scaling control process, with running modes including automatic mode and manual mode; in automatic mode, the action decision is executed by a pre-trained deep reinforcement learning policy model; in manual mode, external instructions are received through API or CLI interface to execute memory pool adjustment operations; after each scaling action is executed, feedback signals after memory pool adjustment are collected and returned to the decision engine for use in subsequent action decisions; the feedback signals include the adjusted memory pool size, GPU utilization, memory usage rate, memory fragmentation rate, and memory allocation rate.

[0012] This invention protects a dynamic scaling control device for a private GPU memory pool based on deep reinforcement learning, comprising: a memory status acquisition module for acquiring current GPU operating status information, including GPU utilization, memory usage, memory fragmentation rate, memory pool size, and memory allocation rate; a policy model inference module for inputting the operating status information into a pre-trained deep reinforcement learning policy model and outputting scaling action decisions in a continuous action space, including maintaining, expanding, or shrinking the memory pool size and the corresponding adjustment ratio; and a memory pool adjustment module for dynamically scaling the private GPU memory pool based on the scaling action decisions; wherein the policy model constructs a private memory pool through a GPU memory management interface and performs automated scaling control on the private memory pool.

[0013] This invention protects an electronic device, comprising: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. The machine-readable instructions are executed by the processor to perform the steps of a dynamic scaling control method for a GPU memory privatization pool based on deep reinforcement learning.

[0014] This invention protects a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of a method for dynamically expanding and shrinking a GPU memory privatization pool based on deep reinforcement learning.

[0015] This invention protects a method for dynamically expanding and shrinking a private GPU memory pool based on deep reinforcement learning. By constructing a reinforcement learning environment with specific action and state spaces, the deep reinforcement learning model can learn the optimal memory pool adjustment strategy to adapt to different GPU load conditions during the pre-training phase. The action space includes three types: maintain, expand, or shrink, with continuous adjustment ratios between 0.5 and 2.0. The state space consists of five continuous dimensions: GPU utilization, memory usage, memory fragmentation rate, current memory pool size, and memory allocation rate. By executing actions and updating the memory pool size at each training step, and obtaining real-world feedback through CUDA or PyTorch interfaces, the method optimizes the strategy based on a multi-dimensional reward function that includes GPU utilization deviation penalties, memory fragmentation rate inverse rewards, memory over-limit penalties, action fluctuation penalties, and execution success rewards. This allows for efficient dynamic scaling of the private memory pool within a process, enabling each process to independently create and manage its own memory pool in a soft-isolated environment. This method improves GPU resource utilization and reduces memory waste and system fluctuations through dynamic management. Furthermore, in each decision cycle, the method actually sets the memory pool capacity via CUDA or PyTorch interfaces and records the status, actions, and adjusted results as log information. Simultaneously, it executes expansion, contraction, or hold operations by parsing the action type and adjustment ratio. Expansion uses multiplicative adjustment, and contraction uses division adjustment, thus ensuring the continuity and stability of the adjustment process and enhancing the interpretability and controllability of strategy execution. In addition, the method introduces a running mode selection mechanism, supporting switching between automatic and manual modes. In automatic mode, the pre-trained model makes autonomous decisions, while in manual mode, it receives external instructions via API or CLI interfaces for flexible control. After each expansion / contraction action, it collects feedback signals including memory pool size, GPU utilization, memory usage, memory fragmentation rate, and memory allocation rate, and sends them back to the decision engine for subsequent strategy optimization, thereby achieving closed-loop control and improving the system's adaptability and robustness. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0017] Figure 1 This application provides a flowchart of a method for dynamically expanding and shrinking a GPU memory privatization pool, as illustrated in an embodiment.

[0018] Figure 2 This application provides a flowchart of a GPU runtime status monitoring method in a dynamic expansion / shrinkage control method for a GPU memory privatization pool.

[0019] Figure 3 This application provides a flowchart of the training process for a deep reinforcement learning strategy model in a method for dynamically expanding and shrinking a GPU memory privatization pool.

[0020] Figure 4 This application provides a flowchart of a dynamic scaling control method for a GPU memory privatization pool, as illustrated in the embodiments of this application.

[0021] Figure 5 This application provides an embodiment of a method for dynamically expanding and shrinking a GPU memory privatization pool, which includes another flowchart of dynamic memory pool adjustment.

[0022] Figure 6 This application provides a flowchart of a method for manually and automatically switching between dynamic scaling up and down in a GPU memory privatization pool dynamic scaling up and down control method.

[0023] Figure 7 The following is a flowchart of the manual / automatic switching device for dynamic scaling up and down in a GPU memory privatization pool dynamic scaling up and down control method provided in this application embodiment;

[0024] Figure 8 This application provides a diagram of a dynamic expansion / shrinkage control device for a GPU memory privatization pool.

[0025] Figure 9 This application provides an embodiment of an electronic device for dynamically expanding and shrinking a GPU memory privatization pool. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0027] Research has revealed that existing GPU memory management solutions mostly employ static allocation or simple rule-based dynamic adjustment strategies, which are ill-suited to the complex and ever-changing GPU workload scenarios. During deep learning training or large-scale parallel computing, memory requirements are highly dynamic and uncertain, leading to low memory resource utilization, severe fragmentation, and even task failures or performance degradation caused by insufficient memory. Furthermore, traditional methods lack a comprehensive understanding of the overall GPU operating status, failing to achieve intelligent, fine-grained control of the memory pool and limiting the optimization of overall system performance.

[0028] When multiple processes share a GPU, it's necessary to isolate process memory access to avoid interference between them. Currently, the main isolation methods are hard isolation and soft isolation. In soft isolation, each process can independently create and manage its own memory pool, achieving in-process memory pool privatization and constructing a private memory pool, thus avoiding cross-process sharing. However, current technologies lack a dynamic adjustment scheme for the private memory pool during soft isolation, leading to memory pool privatization consuming GPU resources, which can cause memory fragmentation and decreased utilization, especially in multi-process concurrent scenarios.

[0029] The system employs a soft isolation mechanism to achieve multi-process memory isolation, creating an independent instance for each computing task through a Python multi-process module. During initialization, it supports pre-allocating contiguous memory blocks to reduce runtime fragmentation and uses a dynamic verification mechanism to ensure that adjustment requests do not exceed physical memory limits. Core parameters include initial pool size, minimum reserved space, and fragmentation rate threshold. Real-world testing on an RTX 3090 shows that dynamic latency adjustment ranges from 50 to 200 milliseconds, and the long-term fragmentation rate can be stably kept below 10%.

[0030] Based on this, please refer to Figure 1 This application provides a method for dynamically expanding and shrinking a GPU memory private pool based on deep reinforcement learning. A private memory pool is constructed through the GPU memory management interface. Through a standardized initialization interface, users can configure private pools ranging from 4GB to 90% of the total memory for different processes. The private memory pool is then automatically expanded and shrunk using a deep reinforcement learning model, supporting dynamic expansion or contraction during runtime. The control method includes:

[0031] S101 Obtain the current operating status information of the GPU, which includes GPU utilization, video memory usage, video memory fragmentation rate, video memory pool size, and video memory allocation rate;

[0032] S102 The running state information is input into the pre-trained deep reinforcement learning policy model, which is constructed based on the PPO algorithm. The reinforcement learning environment defines the state space and action space.

[0033] S103 The strategy model outputs expansion and contraction action decisions in the continuous action space, the expansion and contraction action decisions including maintaining, expanding or shrinking the memory pool size and the corresponding adjustment ratio;

[0034] S104 Based on the expansion and contraction action decision, dynamically expand and contract the GPU memory private pool.

[0035] This application provides a method for dynamic scaling control of a GPU memory privatization pool based on deep reinforcement learning. By introducing a deep reinforcement learning model, the method can perceive the GPU's operating status in real time and make intelligent scaling decisions, thereby effectively improving the utilization of memory resources and task execution efficiency. This method performs decision output in a continuous action space, supports more refined memory adjustment strategies, reduces memory fragmentation, improves system stability and throughput, and is suitable for complex and variable computing scenarios, exhibiting good adaptability and scalability.

[0036] In step S101, the current operating status information of the GPU is obtained. This operating status information includes GPU utilization, memory usage, memory fragmentation rate, memory pool size, and memory allocation rate. GPU utilization measures the busyness of the calculation cores, while memory usage represents the proportion of memory capacity that is occupied.

[0037] Specifically, the system obtains real-time operating status information of the GPU device through the GPU management interface. The operating status information includes key indicators such as GPU utilization, video memory usage, video memory fragmentation rate, video memory pool size, and video memory allocation rate.

[0038] In step S1011, the handle of the GPU device is first obtained through a built-in GPU management library (such as pynvml), and basic operating status data such as the current GPU utilization, total video memory, used video memory, and free video memory are obtained based on this handle. Specifically, the system calls the DeviceGetUtilizationRates interface in pynvml to obtain the GPU utilization, and calls the DeviceGetMemoryInfo interface in pynvml to obtain the values ​​of total video memory, used video memory, and free video memory, thereby providing basic data support for subsequent calculations of video memory utilization and fragmentation rate.

[0039] After obtaining basic data from the GPU device handle, step S1012 further obtains the currently allocated video memory size through the video memory management interface provided by CUDA or PyTorch. For example, in the PyTorch framework, the total amount of currently allocated video memory can be obtained through the torch.cuda.memory_allocated() interface. Based on this allocated video memory and the free video memory obtained in S1011, the system calculates the video memory fragmentation rate. Specifically, the video memory fragmentation rate is equal to the ratio of free video memory minus allocated video memory to the total video memory, i.e., fragment_ratio = (free_mem - cuda_alloc) / total_mem, where free_mem is free video memory, cuda_alloc is allocated video memory, and total_mem is the total video memory.

[0040] In step S1013, the system performs non-negative correction on the memory fragmentation rate calculated in S1012. Since in certain operating states, the amount of free memory may be less than the allocated memory, resulting in a negative fragmentation rate, the system compares the fragmentation rate result with zero using the max function to ensure that the final output memory fragmentation rate is non-negative, i.e., frag_ratio = max(0, frag_ratio), thereby guaranteeing the rationality of the input data and the stability of model training.

[0041] In step S1014, the system calculates the ratio of the current memory pool size to the total memory size to obtain the proportion of the memory pool size in the overall memory, which is used as part of the operating status information. Simultaneously, based on the memory allocation data over the past 60 seconds, the system calculates the memory allocation rate to reflect the dynamic trend of current GPU memory resources. Specifically, the memory allocation rate can be calculated by statistically analyzing the total amount or average allocation size of memory per unit time to characterize the memory demand intensity of the current GPU workload.

[0042] Specifically, the calculation process for each part is as follows:

[0043] # GPU utilization (computational load)

[0044] util = pynvml.nvmlDeviceGetUtilizationRates(handle)

[0045] gpu_util = util.gpu / 100.0

[0046] # Video memory usage

[0047] mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)

[0048] mem_usage = mem_info.used / mem_info.total

[0049] # Memory fragmentation rate

[0050] cuda_alloc = torch.cuda.memory_allocated()

[0051] frag_ratio = (mem_info.free - cuda_alloc) / mem_info.total

[0052] Through the implementation steps S1011 to S1014 described above, the system has completely acquired the current operating status information of the GPU, including five dimensions of status parameters such as GPU utilization, memory usage, memory fragmentation rate, the ratio of memory pool size to total memory, and memory allocation rate. This provides a comprehensive and accurate input basis for the subsequent strategy decision-making of the deep reinforcement learning model.

[0053] In step S102, the system inputs the GPU's current operating state information obtained in S101 into the pre-trained deep reinforcement learning policy model to drive the model to make scaling decisions. This policy model is built based on the PPO algorithm and can output precise scaling control instructions in the continuous action space, thereby realizing dynamic management of the GPU memory private pool.

[0054] In step S1021, a reinforcement learning environment is constructed and trained as the basis for running the deep reinforcement learning policy model. This reinforcement learning environment defines a state space and an action space: the state space is a continuous state vector containing five dimensions, representing GPU utilization, memory usage, memory fragmentation rate, the ratio of the current memory pool size to the total memory, and the memory allocation rate, respectively; the action space consists of scaling types and adjustment ratios, where scaling types include three operations: maintain, expand, or shrink, and the adjustment ratio ranges from 0.5 to 2.0, representing the magnitude of the memory pool size adjustment.

[0055] This step completes the construction and initialization of the reinforcement learning environment, mainly including the definition of the state space and action space. The state space consists of a continuous vector composed of five key GPU performance metrics, while the action space defines three adjustment operations of the memory pool and their magnitude ranges. This environment provides a standardized interaction interface and evaluation framework for the subsequent training and operation of deep reinforcement learning models.

[0056] In step S1022, the system executes the action output by the policy model at each training step and updates the size of the memory pool based on this action. Specifically, the system uses the memory management interface provided by CUDA or PyTorch. In one specific implementation, the memory management interface can be torch.cuda.set_per_process_memory_fraction or a custom memory pool API to set the actual capacity of the memory pool. Subsequently, the system collects the updated GPU running state information and uses it as the input to the model for the next state, forming a complete reinforcement learning loop.

[0057] This step implements action execution and environmental feedback during reinforcement learning training. Based on the action instructions output by the policy model, the system dynamically adjusts the memory pool capacity through the underlying GPU memory management interface and collects the adjusted GPU state data in real time as new environmental observations, thus forming a complete closed-loop control scheme for reinforcement learning.

[0058] The closed-loop control method achieves dynamic expansion and contraction control through three stages: state observation, action decision-making, and environmental feedback.

[0059] In state observation: GPU utilization, memory usage, memory fragmentation rate, memory pool percentage, and memory allocation rate are collected, and these metrics are used to construct a one-dimensional state vector S. t ;

[0060] In action decision-making: the strategy model calculates and determines the decision action a. t As mentioned above, the decision action is a function of the expansion / shrinkage type and the adjustment ratio. Once the expansion / shrinkage type and the adjustment ratio are determined, the expansion / shrinkage action and its magnitude can be determined.

[0061] In the environmental feedback: After performing an action, a new state S is obtained. t+1 and reward r t Training data (S) is generated. t , a t, , r t, ,S t+1 );

[0062] The reward function is designed as a one-dimensional state vector r k The sum of each element's value and its weight, specifically r k Each element includes GPU utilization deviation penalty, memory fragmentation rate reward, memory over-limit penalty, action fluctuation penalty, and execution success reward. The reward r is determined by calculating the reward function. t ;

[0063] The policy network is optimized using the PPO algorithm, where the advantage function A_t is estimated using GAE (λ=0.95), and the shearing parameter ε=0.2. The system sets the state vector dimension to 5, the action ratio range to [0.5, 2.0], and terminates the current training round when the memory usage is greater than 0.95 or the number of training steps is ≥1000.

[0064] In step S1023, the system calculates the reward value of the current action based on a multi-dimensional reward function to guide the learning process of the policy model. This reward function comprehensively considers several key performance indicators: First, it penalizes deviations in GPU utilization; that is, when GPU utilization deviates from the target value (in one implementation, the target value is set at 70%), a negative reward is given. Second, it provides a reverse reward for memory fragmentation rate; that is, the lower the fragmentation rate, the higher the reward. Third, when memory usage exceeds a set threshold of 95%, a memory over-limit penalty is applied; in one implementation, the threshold is set at 70%. Furthermore, it applies action fluctuation penalties to frequent or drastic scaling actions to improve system stability. Finally, if the action is successfully executed, a positive reward is given to encourage the model to select effective scaling strategies.

[0065] This step designs and implements a multi-dimensional reward calculation mechanism. By comprehensively considering multiple key factors such as GPU utilization deviation, memory fragmentation, memory usage exceeding limits, and action fluctuation, a refined reward function is constructed to provide the policy model with a clear learning objective and optimization direction, ensuring that the model can learn a memory management strategy that balances performance and stability.

[0066] In one specific implementation, the system further determines whether the current training round meets the termination conditions. When the memory usage exceeds a preset threshold (in one specific implementation, the threshold is 95%) or the training steps reach the maximum number of steps (in one specific implementation, the maximum number of steps is 1000), the system marks the end of the current training round, resets the environment state, and prepares to enter the next training iteration.

[0067] Through the implementation steps S1021 to S1023 described above, the system completes the entire process of inputting GPU running status information into the pre-trained deep reinforcement learning policy model, providing a solid data foundation and model support for subsequent output of scaling action decisions in the continuous action space.

[0068] In step S103, the system outputs scaling action decisions in the continuous action space through a pre-trained deep reinforcement learning policy model. These scaling action decisions include maintaining, expanding, or shrinking the memory pool size and the corresponding adjustment ratio. This policy model, built on the PPO algorithm, can output stable and efficient scaling control instructions under complex and variable GPU operating conditions, thereby achieving dynamic management of the GPU's private memory pool.

[0069] In step S1031, the system employs an online inference service during dynamic scaling control to continuously acquire the current operating status information of the GPU and output scaling action decisions based on a pre-trained deep reinforcement learning policy model. This online inference service is implemented by running a class instance containing prediction logic. In one specific implementation, a MemoryPoolAutoscaler class instance is selected. This instance periodically performs state acquisition, action prediction, and memory pool adjustment operations to ensure the real-time performance and stability of scaling control.

[0070] An instance of the MemoryPoolAutoscaler class is implemented using the following code:

[0071] class MemoryPoolAutoscaler:

[0072] def __init__(self, drl_model_path: str, gpu_id: int = 0):

[0073] """

[0074] :param drl_model_path:

[0075] :param gpu_id:

[0076] self.model = PPO.load(drl_model_path)

[0077] self.gpu_id = gpu_id

[0078] self.state_buffer = deque(maxlen=60)

[0079] pynvml.nvmlInit()

[0080] self.device_handle = pynvml.nvmlDeviceGetHandleByIndex(gpu_id)

[0081] torch.cuda.set_device(gpu_id)

[0082] def get_gpu_state(self) -> dict:

[0083] util = pynvml.nvmlDeviceGetUtilizationRates(self.device_handle)

[0084] mem_info = pynvml.nvmlDeviceGetMemoryInfo(self.device_handle)

[0085] return {

[0086] 'gpu_util': util.gpu / 100.0,

[0087] 'mem_usage': mem_info.used / mem_info.total,

[0088] 'frag_ratio': max(0, (mem_info.free - torch.cuda.memory_allocated()) / mem_info.total),

[0089] 'pool_ratio': torch.cuda.memory_reserved() / mem_info.total,

[0090] 'alloc_speed': self._calc_alloc_speed()

[0091] }

[0092] def predict_action(self, state: dict) -> dict:

[0093] state_vector = np.array([state[k] for k in sorted(state.keys())])

[0094] action, _ = self.model.predict(state_vector)

[0095] return {'type': int(action[0]), 'ratio': float(action[1])}

[0096] def adjust_pool(self, action: dict):

[0097] current_size = torch.cuda.memory_reserved()

[0098] if action['type'] == 1:

[0099] new_size = current_size * action['ratio']

[0100] elif action['type'] == 2:

[0101] new_size = current_size / action['ratio']

[0102] else:

[0103] return

[0104] torch.cuda.empty_cache()

[0105] torch.cuda.set_per_process_memory_fraction(new_size / pynvml.nvmlDeviceGetMemoryInfo(self.device_handle).total)

[0106] def run_cycle(self, interval: float = 10.0):

[0107] while True:

[0108] state = self.get_gpu_state()

[0109] action = self.predict_action(state)

[0110] self.adjust_pool(action)

[0111] self.state_buffer.append(state)

[0112] time.sleep(interval)

[0113] The periodic execution process specifically includes step S10311, where the system sets the actual capacity of the memory pool through the CUDA or PyTorch interface in each decision cycle, and records the current state, action decision, and adjusted memory pool size as log information. Specifically, the system calls torch.cuda.set_per_process_memory_fraction or a custom memory pool management interface to convert the scaling actions output by the policy model into actual memory pool capacity adjustment operations, and records the complete decision process to the log system. The recorded decision process includes the input state, output action, and execution result, which are used for subsequent performance analysis and model optimization.

[0114] In step S10312, the system executes specific expansion / shrinkage operations by parsing the action type and adjustment ratio output by the strategy model. The action type includes three cases: expansion, shrinkage, or maintenance. When the action type is "expansion," the system multiplies the current memory pool size by the adjustment ratio; in one implementation, the adjustment ratio ranges from 1.0 to 2.0. When the action type is "shrinkage," the system divides the current memory pool size by the adjustment ratio; in one implementation, the value ranges from 1.0 to 2.0. When the action type is "maintain," the memory pool size remains unchanged. This mechanism ensures the continuity and flexibility of expansion / shrinkage control, avoiding the oscillation and lag problems that may occur in traditional threshold control.

[0115] In step S1032, a circular buffer for recording historical states is maintained through class instances in the online inference service, and the maximum storage length of the circular buffer is 60 time units.

[0116] This buffer uses a queue structure, automatically overwriting the oldest data as new data arrives, always maintaining a state record of the 60 most recent time steps. Each state data includes five dimensions of metrics: GPU utilization, memory usage, memory fragmentation rate, memory pool percentage, and memory allocation rate. The system implements this circular buffer using Python's deque data structure and provides interfaces for adding new states and querying the N most recent historical states. During online inference, the policy model can obtain a sequence of historical states over a recent time window, which in one embodiment is 10 time steps, and combine it with the current state to make a comprehensive decision. This design enables the model to perceive short-term trends in GPU load, thereby making more accurate judgments on memory expansion and contraction. The buffer ensures efficient data updates; the time complexity of each state write and read operation is O(1), meeting real-time requirements.

[0117] This buffer stores GPU runtime information over the last 60 decision cycles, providing the policy model with short-term historical state context, enhancing its ability to perceive GPU load trends, and thus improving the accuracy and robustness of scaling decisions.

[0118] Through the implementation steps S1031 to S1032 described above, the real-time state of the GPU (5 metrics) is mapped to continuous expansion and contraction actions of the private memory pool using a deep reinforcement learning model. The specific range is 0.5-2.0. Based on closed-loop optimization of a multi-objective reward function, dynamic and precise allocation of memory resources is achieved. The system implements a continuous action space expansion and contraction decision output mechanism based on a deep reinforcement learning policy model. It can dynamically adjust the size of the private memory pool according to the current operating state of the GPU, effectively balancing memory utilization, fragmentation rate, and OOM risk, thereby improving the overall efficiency of GPU resource utilization and system stability.

[0119] In step S104, the system executes dynamic scaling operations on the GPU memory private pool based on the scaling decisions output by the deep reinforcement learning policy model in S103. This process achieves elastic adjustment of memory resources through an automated control mechanism, ensuring efficient utilization of GPU resources and system stability under different load conditions.

[0120] In step S1041, the system introduces a running mode selection mechanism during the dynamic scaling control process. The running modes include automatic mode and manual mode. In automatic mode, the system relies entirely on the pre-trained deep reinforcement learning policy model for scaling decisions, and all actions are automatically calculated and executed by the model based on the current GPU running state. In manual mode, the system receives external control commands through API or CLI interfaces, allowing maintenance personnel to intervene and adjust the size of the memory pool according to actual needs, thereby achieving flexible switching between policy model and manual control.

[0121] For details on the execution process, please refer to... Figure 7As shown, the system adopts a layered architecture, comprising three core parts: an input layer, a decision layer, and an execution layer. In the input layer, the system receives two types of input data: real-time monitoring metrics, including performance parameters such as GPU utilization and memory usage, and manual intervention commands, which can be input via API or CLI interfaces. This input data is passed to the DRL decision engine for unified processing. In the decision layer, the system selects different decision paths based on the current operating mode, which can be automatic or manual: in automatic mode, it calls a pre-trained DRL model for intelligent inference to generate scaling strategies; in manual mode, it directly executes the manual commands received via API / CLI. Finally, in the execution layer, the system performs specific private memory pool adjustments through the memory management interface and re-inputs the execution results as feedback signals to the system, forming a closed-loop control. The entire process achieves seamless collaboration between automated decision-making and manual intervention, ensuring that the system can make optimal memory resource allocation decisions in different scenarios.

[0122] In step S1042, after each memory expansion / shrinkage operation, the system collects the adjusted memory pool feedback signal and returns it to the decision engine to update the model input state and optimize subsequent action decisions. The feedback signal includes key indicators such as the adjusted memory pool size, GPU utilization, memory usage rate, memory fragmentation rate, and memory allocation rate, forming a closed-loop control mechanism to ensure the system can dynamically adjust its strategy based on the latest state. The following figure shows an example of the feedback signal values ​​after one adjustment:

[0123]

[0124] This step implements a complete closed-loop feedback control mechanism. After each scaling operation, the system immediately collects multi-dimensional feedback signals, including: the adjusted actual memory pool size (in GB); real-time GPU utilization (range 0-100%); current memory usage (the proportion of used memory to total memory); the latest memory fragmentation rate (range 0-1); and memory allocation rate (in MB / s). This feedback data is processed by a standardized state encoder and updated to the DRL decision engine's state input queue as a 5-dimensional vector. The system uses a time-series database to temporarily store the most recent 10 state change records, providing short-term historical context for the policy model. In automatic mode, this feedback data is directly input into the pre-trained PPO model, driving the model to generate the next round of optimization decisions. In manual mode, the feedback data is recorded in the operation log for operations personnel to analyze the effects of manual intervention. Simultaneously, the system calculates the rate of change of key indicators between two states, specifically the GPU utilization fluctuation. When abnormal fluctuations are detected, a policy rollback mechanism is automatically triggered to ensure system stability. This closed-loop mechanism enables the quantitative evaluation of the effect of each expansion and contraction operation, and continuously optimizes subsequent decisions through continuous status feedback, ultimately achieving precise dynamic control of video memory resources.

[0125] Through the implementation steps S1041 to S1042 described above, the system realizes dynamic expansion and contraction control of the GPU memory private pool based on expansion and contraction action decisions, supports automatic and manual dual-mode operation, and continuously optimizes the decision-making process through a feedback signal mechanism, thereby improving the intelligence level and operating efficiency of GPU resource management.

[0126] Please see Figure 8 , Figure 8 This is a schematic diagram of a GPU memory privatization pool dynamic expansion and contraction control device based on deep reinforcement learning, provided as an embodiment of this application. Figure 8 As shown, the control device 200 includes:

[0127] The video memory status acquisition module 210 is used to acquire the current running status information of the GPU, including GPU utilization, video memory usage, video memory fragmentation rate, video memory pool size, and video memory allocation rate.

[0128] The strategy model inference module 220 is used to input the running state information into the pre-trained deep reinforcement learning strategy model and output the scaling action decision in the continuous action space. The scaling action decision includes maintaining, expanding or shrinking the memory pool size and the corresponding adjustment ratio.

[0129] The memory pool adjustment module 230 creates and manages process-private memory pools by calling the NVIDIA memory pool management interface. It dynamically adjusts the memory pool capacity using memory pool attribute settings and forcibly reclaims unused memory through a memory pool reclamation interface. The module integrates an NVIDIA hardware monitoring interface to monitor memory status in real time and uses an asynchronous memory allocation interface to achieve efficient memory block allocation. It also ensures memory isolation between multiple processes through device visibility control and memory pool access permission settings. In abnormal situations, it triggers a forced memory pool reclamation function to roll back to a safe configuration, and ensures the atomicity of the adjustment through device synchronization. This module directly operates on the GPU's physical memory, supporting dynamic adjustments from 4GB to 90% of the total memory, with latency below 50 milliseconds and fragmentation rate controlled within 5%.

[0130] Furthermore, the control device 200 also includes a model training module (not shown in the figure), which is used to construct a reinforcement learning environment. The reinforcement learning environment includes an action space and a state space. The action space includes scaling types and adjustment ratios, where the scaling types include maintaining, expanding, or shrinking, and the adjustment ratio ranges from 0.5 to 2.0. The state space is a continuous state vector containing five dimensions, representing GPU utilization, memory usage, memory fragmentation rate, current memory pool size, and memory allocation rate, respectively. During each training step, the memory pool size is updated after the action is executed, and the actual capacity of the memory pool is set through the CUDA or PyTorch interface. The updated state information is obtained, and the reward value of the current action is calculated based on a multi-dimensional reward function. The reward function includes a penalty for deviation in GPU utilization, a reverse reward for memory fragmentation rate, a penalty for memory over-limit, a penalty for action fluctuation, and a reward for successful execution.

[0131] Furthermore, the model training module marks the end of the current training round when the memory usage exceeds a preset threshold or the number of training steps reaches the maximum number of steps.

[0132] Furthermore, when the policy model inference module 220 outputs scaling action decisions from the policy model in the continuous action space, the policy model inference module 220 is configured to: continuously acquire GPU current running state information using an online inference service during the dynamic scaling control process, and output scaling action decisions based on a pre-trained deep reinforcement learning policy model; the online inference service periodically performs state acquisition, action prediction, and memory pool adjustment operations by running class instances containing prediction logic; the class instances contain a circular buffer for recording historical states, and the maximum storage length of the circular buffer is 60 time units.

[0133] Furthermore, when the memory pool adjustment module 230 is used to dynamically expand or shrink the GPU memory private pool based on the expansion / shrinkage action decision, the memory pool adjustment module 230 is used to: set the actual capacity of the memory pool through the CUDA or PyTorch interface in each decision cycle, and record the current state, action decision, and adjusted memory pool size as log information; the execution of the action decision is achieved by parsing the action type and adjustment ratio, wherein the action type includes expansion, shrinkage, or maintenance, the expansion operation adjusts the memory pool size through multiplication, and the shrinkage operation adjusts the memory pool size through division.

[0134] Furthermore, when the memory status acquisition module 210 is used to acquire the current running status information of the GPU, the memory status acquisition module 210 is used to: acquire the GPU device handle through the built-in library, and acquire the GPU utilization, total memory, used memory, and free memory based on the GPU device handle; acquire the currently allocated memory through the CUDA or PyTorch interface, and calculate the memory fragmentation rate based on the difference between the free memory and the allocated memory; the calculation result of the memory fragmentation rate is compared with zero by the maximum function to ensure that the memory fragmentation rate is non-negative; the current running status information also includes the ratio of the memory pool size to the total memory, and the memory allocation rate calculated based on the memory allocation situation in the last 60 seconds.

[0135] Furthermore, when the memory pool adjustment module 230 is used to dynamically expand or shrink the GPU memory private pool based on the expansion / shrinkage action decision, the memory pool adjustment module 230 is configured to: set a running mode selection mechanism during the dynamic expansion / shrinkage control process, the running modes including automatic mode and manual mode; in automatic mode, the action decision is executed by a pre-trained deep reinforcement learning policy model; in manual mode, external instructions are received through API or CLI interface to execute the memory pool adjustment operation; after each expansion / shrinkage action is executed, the feedback signal after the memory pool adjustment is collected, and the feedback signal is returned to the decision engine for use in subsequent action decisions; the feedback signal includes the adjusted memory pool size, GPU utilization, memory usage rate, memory fragmentation rate, and memory allocation rate.

[0136] The GPU memory private pool dynamic scaling control device based on deep reinforcement learning provided in this application constructs a private memory pool through a GPU memory management interface and automatically scales the private memory pool using a deep reinforcement learning model. It acquires the current operating status information of the GPU and inputs this information into a pre-trained deep reinforcement learning policy model. The policy model outputs scaling action decisions in a continuous action space, and dynamically scales the GPU memory private pool based on these decisions. This enables intelligent scheduling and dynamic optimization of GPU memory resources, improving GPU resource utilization, reducing memory fragmentation, enhancing system stability, and increasing training efficiency.

[0137] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0138] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the method embodiment shown in the illustration, which is a dynamic scaling control method for GPU memory privatization pool based on deep reinforcement learning, can be found in the method embodiment for specific implementation details, which will not be repeated here.

[0139] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the method embodiment shown in the illustration, which is a dynamic scaling control method for GPU memory privatization pool based on deep reinforcement learning, can be found in the method embodiment for specific implementation details, which will not be repeated here.

[0140] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0144] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for dynamically expanding and shrinking a GPU memory privatization pool, characterized in that, A private memory pool is constructed through the GPU memory management interface. This private memory pool is created and managed independently for each process to avoid different processes sharing the GPU. A deep reinforcement learning model is used to automatically control the scaling of this private memory pool. The control method includes: Obtain the current running status information of the GPU, which includes GPU utilization, memory usage, memory fragmentation rate, memory pool size, and memory allocation rate. The running state information is input into the pre-trained deep reinforcement learning policy model, which is constructed based on the PPO algorithm. The reinforcement learning environment defines the state space and action space. The strategy model outputs scaling action decisions in the continuous action space, and the scaling action decisions include maintaining, expanding or shrinking the size of the private memory pool and the corresponding adjustment ratio. Based on the aforementioned scaling up / down action decision, the GPU memory private pool is dynamically scaled up or down. The pre-training process of the deep reinforcement learning model includes constructing a reinforcement learning environment, which includes the action space and the state space of the environment. The action space of the environment includes expansion / shrinkage types and adjustment ratios, wherein the expansion / shrinkage types include maintaining, expanding, or shrinking, and the adjustment ratio ranges from 0.5 to 2.0; the state space of the environment is a continuous state vector containing five dimensions, representing GPU utilization, memory usage, memory fragmentation rate, current memory pool size, and memory allocation rate, respectively. During each training step, the memory pool size is updated after the action is executed, and the actual capacity of the memory pool is set through the CUDA or PyTorch interface. The updated state information is obtained, and the reward value of the current action is calculated based on a multi-dimensional reward function. The reward function includes a deviation penalty for GPU utilization, a reverse reward for memory fragmentation, a memory over-limit penalty, an action fluctuation penalty, and a successful execution reward.

2. The method for dynamically expanding and shrinking the GPU memory private pool according to claim 1, characterized in that, When the memory usage exceeds the preset threshold or the number of training steps reaches the maximum, mark the end of the current training round.

3. The method for dynamically expanding and shrinking the GPU memory privatization pool according to claim 1, characterized in that, The scaling action decision output by the strategy model in the continuous action space includes: During the dynamic scaling control process, an online inference service is used to continuously acquire the current running status information of the GPU and output scaling action decisions based on the pre-trained deep reinforcement learning policy model. The online inference service periodically performs state acquisition, action prediction, and memory pool adjustment operations by running class instances containing prediction logic. The class instance contains a circular buffer for recording historical states, and the maximum storage length of the circular buffer is 60 time units.

4. The method for dynamically expanding and shrinking the GPU memory privatization pool according to claim 3, characterized in that, In each decision cycle, the actual capacity of the video memory pool is set through the CUDA or PyTorch interface, and the current status, action decision and adjusted video memory pool size are recorded as log information. The execution of the action decision is achieved by parsing the action type and adjusting the ratio, with the adjustment ratio ranging from 1.0 to 2.

0. The action type includes expansion, contraction, or maintenance. The expansion operation adjusts the memory pool size through multiplication, and the contraction operation adjusts the memory pool size through division.

5. The method for dynamically expanding and shrinking the GPU memory privatization pool according to claim 1, characterized in that, The process of obtaining the current running status information of the GPU includes: The GPU device handle is obtained through the built-in library, and the GPU utilization, total video memory, used video memory, and free video memory are obtained based on the GPU device handle. The currently allocated video memory is obtained through the CUDA or PyTorch interface, and the video memory fragmentation rate is calculated based on the difference between the free video memory and the allocated video memory. The memory fragmentation rate is calculated by comparing the result with zero using a maximum function to ensure that the memory fragmentation rate is non-negative. The current operating status information also includes the ratio of the memory pool size to the total memory size, and the memory allocation rate calculated based on the memory allocation situation in the last 60 seconds.

6. The method for dynamically expanding and shrinking the GPU memory privatization pool according to claim 1, characterized in that, The dynamic scaling up and down of the GPU memory private pool based on the scaling up and down action decision includes: In the dynamic expansion and contraction control process, an operation mode selection mechanism is set, and the operation modes include automatic mode and manual mode; In automatic mode, action decisions are made by a pre-trained deep reinforcement learning policy model; in manual mode, external instructions are received via API or CLI interface to perform memory pool adjustment operations. After each expansion or contraction operation is performed, the feedback signal after the adjustment of the video memory pool is collected and the feedback signal is returned to the decision engine for use in subsequent action decisions; The feedback signals include the adjusted memory pool size, GPU utilization, memory usage, memory fragmentation rate, and memory allocation rate.

7. A dynamic scaling control device for a GPU memory privatization pool based on deep reinforcement learning, using the control method described in any one of claims 1-6, characterized in that, include: The video memory status acquisition module is used to acquire the current running status information of the GPU, including GPU utilization, video memory usage, video memory fragmentation rate, video memory pool size, and video memory allocation rate. The policy model inference module is used to input the running state information into the pre-trained deep reinforcement learning policy model and output the scaling action decision in the continuous action space. The scaling action decision includes maintaining, expanding or shrinking the memory pool size and the corresponding adjustment ratio. The memory pool adjustment module is used to dynamically expand or shrink the GPU memory private pool based on the expansion / shrinkage action decision. The memory pool adjustment module constructs a private memory pool through the GPU memory management interface and performs automated expansion and contraction control on the private memory pool.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the GPU memory privatization pool dynamic expansion and contraction control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the GPU memory privatization pool dynamic expansion and contraction control method as described in any one of claims 1 to 6.

Images