A method, device and storage medium for allocating video memory

Abstract

Embodiments of the present application provide a graphics memory allocation method and device and a storage medium to reduce the risk of graphics memory shortage caused by graphics memory fragmentation. The method comprises: obtaining a first graphics memory event corresponding to a first process from a driver layer, obtaining a second graphics memory event corresponding to a plurality of requests triggered in a computing task execution process from a user state, determining an overall fragmentation rate of a first graphics memory space using a first allocation strategy according to the first graphics memory event and the second graphics memory event corresponding to the plurality of requests, determining a second allocation strategy if the overall fragmentation rate meets a first preset condition, the overall fragmentation rate corresponding to the second allocation strategy being smaller than the overall fragmentation rate corresponding to the first allocation strategy, and performing graphics memory allocation on the first graphics memory space based on the second allocation strategy. Adjusting the allocation strategy in the direction of reducing the overall fragmentation rate helps to reduce the risk of graphics memory shortage caused by graphics memory fragmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a video memory allocation method, device and storage medium. Background Technology

[0002] In deep learning model training and inference, memory management is a key factor affecting performance. Traditional memory allocation methods mainly rely on the memory allocation and reclamation mechanisms of the Graphics Processing Unit (GPU) or General-Purpose Graphics Processing Unit (GPGPU) driver layer, as well as the memory pool management strategies of the application layer. These methods typically employ static allocation or fixed algorithms (such as Best-Fit, Worst-Fit, etc.) for memory allocation. These methods often use static allocation modes or preset allocation rules, lacking real-time monitoring capabilities and dynamic optimization mechanisms for memory fragmentation, leading to prominent memory fragmentation problems.

[0003] In model training or inference scenarios, for computational tasks initiated by a single process, pre-allocated memory space is obtained from the GPU's memory by calling memory allocation functions. During the execution of the computational task, the pre-allocated memory space is managed by a user-space memory manager, for example, allocating corresponding memory blocks from the pre-allocated memory space for each request initiated within the process. Due to the randomness of requests (such as varying sizes, frequency fluctuations, and high address jumps), external fragmentation is unavoidable. For example, when a process frequently requests small blocks of memory and cannot merge them after release, the remaining free blocks may not be able to meet the allocation needs of large blocks, ultimately leading to an Out-of-Memory (OOM) error.

[0004] In related technologies, the overall usage of GPU memory is indirectly measured through interfaces provided by the driver layer or the CUDA toolkit. These tools can only provide macroscopic data such as total memory usage and remaining memory, and cannot track the fragmentation state within the pre-allocated space of a single process. Existing tools lack the ability to distinguish between internal fragmentation (underutilized allocated blocks) and external fragmentation (scattered free blocks), and cannot dynamically adjust allocation strategies based on ongoing behavior (such as allocation patterns and address locality), resulting in a passive management approach that relies on manual tuning by developers or default framework strategies (such as PyTorch's torch.cuda.empty_cache()), making it difficult to adapt to complex task loads. In addition, while static allocation strategies (such as fixed-size memory pools) can reduce fragmentation, they have resource waste issues: reserving redundant memory to cope with peak demand leads to low utilization under low load; at the same time, they have poor adaptability and cannot adapt to diverse scenarios such as high-frequency small-scale allocation (such as Attention operations in Transformers) or periodic allocation (such as convolution kernel parameter updates). In long-term running tasks, the accumulation of fragmentation may lead to sudden allocation failures, interrupting the training process. Summary of the Invention

[0005] This invention provides a video memory allocation method, device, and storage medium to reduce the risk of insufficient video memory caused by video memory fragmentation.

[0006] In a first aspect, this application provides a video memory allocation method, comprising: obtaining a first video memory event corresponding to a first process from a driver layer, wherein the first video memory event is used to request a pre-allocated first video memory space from video memory for a computing task initiated by the first process; obtaining second video memory events corresponding to multiple requests triggered during the execution of the computing task from user space, wherein the second video memory event corresponding to each request is used to allocate or release a video memory block from the first video memory space for the request; determining an overall fragmentation rate corresponding to the first video memory space based on the first video memory event and the multiple corresponding second video memory events; the overall fragmentation rate is generated after allocating video memory to the first video memory space based on a first allocation strategy; determining a second allocation strategy when the overall fragmentation rate corresponding to the first video memory space meets a first preset condition; the overall fragmentation rate corresponding to the second allocation strategy is less than the overall fragmentation rate corresponding to the first allocation strategy; and allocating video memory to the first video memory space based on the second allocation strategy.

[0007] The above scheme obtains information about the pre-allocated first video memory space by acquiring the first video memory event corresponding to the first process from the driver layer. By acquiring the second video memory events corresponding to multiple requests triggered during the execution of the computing task from the user space, the allocation and release status of video memory blocks in the pre-allocated first video memory space can be obtained. This allows for the accurate determination of the overall fragmentation rate of the first video memory space. Then, if the overall fragmentation rate of the first video memory space meets the first preset condition, the allocation strategy is adjusted in the direction of reducing the overall fragmentation rate. Finally, video memory is allocated to the first video memory space according to the adjusted second allocation strategy, thereby helping to reduce the risk of insufficient video memory caused by video memory fragmentation.

[0008] In one possible implementation, determining the second allocation strategy includes: extracting memory allocation feature information corresponding to the first process based on the second memory event; identifying the memory allocation mode of the first process based on the memory allocation feature information; and using the allocation strategy corresponding to the memory allocation mode as the second allocation strategy.

[0009] In one possible implementation, the memory allocation characteristic information includes at least one of the following: the allocation frequency and standard deviation of the memory block, the distribution entropy and standard deviation of the memory block allocation capacity, the address jump index of the memory block, and the lifetime and standard deviation of the memory block; determining the memory allocation mode of the first process based on the memory allocation characteristic information corresponding to the first process includes: if the allocation frequency of the memory block corresponding to the first process is greater than a first preset value, and the distribution entropy of the memory block allocation capacity corresponding to the first process is less than a second preset value, then the memory allocation mode is identified as a high-frequency small-scale allocation mode; if the address jump index of the memory block corresponding to the first process is greater than a third preset value, then the memory allocation mode is identified as a distributed allocation mode; if the standard deviation of the memory block allocation frequency is less than a fourth preset value, and the standard deviation of the distribution entropy of the memory block allocation capacity corresponding to the first process is less than a fifth preset threshold, the standard deviation of the distribution entropy of the memory block allocation capacity is less than the fifth preset value, and the lifetime standard deviation is less than a sixth preset value, then the memory allocation mode is identified as a periodic allocation mode.

[0010] In one possible implementation, the allocation strategy corresponding to the high-frequency small-scale allocation mode is to adopt a memory pool allocation strategy, wherein the memory pool includes a set of video memory blocks with a preset granularity; the allocation strategy corresponding to the distributed allocation mode is to adopt a free block merging strategy based on a buddy system; and the allocation strategy corresponding to the periodic allocation mode is a pre-allocation and cache reuse strategy.

[0011] In one possible implementation, determining the overall fragmentation rate of the first video memory space based on the first video memory event and the second video memory events corresponding to the multiple requests respectively includes: determining the total video memory amount of the first video memory space based on the first video memory event. The total allocated capacity of the allocated memory blocks is calculated based on the second memory event. and the actual usage of each memory block The internal fragmentation rate (IFR) is calculated using the following formula:

[0012] IFR= ;

[0013] The external fragmentation rate is calculated using the following formula based on the address difference between adjacent memory blocks in the second memory event. :

[0014] ,in M represents the total free video memory, and M is the maximum size of a contiguous free block.

[0015] The overall fragmentation rate is calculated using the following weighted formula. :

[0016] ,in As the weight of the external fragmentation rate, As the weight of the internal fragmentation rate, .

[0017] In one possible implementation, the first preset condition is that the overall fragmentation rate (OFR) is greater than or equal to a first threshold. ,in The value range is 0.3≤ ≤0.7.

[0018] In one possible implementation, after allocating video memory to the first video memory space based on the second allocation strategy, the method further includes: monitoring the video memory usage of the first video memory space and calculating a new overall fragmentation rate OFR′; if the difference between OFR′ and the overall fragmentation rate OFR before optimization is less than a preset optimization effect threshold... If the third allocation strategy is determined, it is different from the second allocation strategy. Based on the third allocation strategy, the first video memory space is allocated.

[0019] In one possible implementation, any video memory event is collected by an eBPF program set in kernel mode, which captures the event by hooking the video memory allocation / release API or the driver layer interface.

[0020] In a second aspect, this application provides a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, wherein the processor chip, when executing the computer program, implements the method in the first aspect or any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including computer-executable instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method in the first aspect or any possible implementation thereof.

[0023] The technical effects that can be achieved by any of the second to fifth aspects mentioned above can be referred to the description of the beneficial effects in the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of a heterogeneous computing system to which the embodiments of this application apply;

[0025] Figure 2 A schematic diagram of the structure of key software components provided in the embodiments of this application;

[0026] Figure 3 A flowchart illustrating a video memory allocation method provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] like Figure 1This is a schematic diagram illustrating the architecture of a heterogeneous computing system to which this application is exemplarily applicable. The heterogeneous computing system includes a host and devices. Exemplarily, the host may be a Central Processing Unit (CPU), and the devices may be a Graphics Processing Unit (GPU) or a General-purpose Computing on Graphics Processing Unit (GPGPU). In a heterogeneous computing system, the number of devices can be one or more; this application does not specifically limit the number of devices. It should be understood that the heterogeneous computing system may also include other modules or components; this application does not specifically limit these. For ease of description, the following embodiments use one device as an example.

[0030] The host is used to handle non-computationally intensive tasks in artificial intelligence applications (such as task scheduling and data preprocessing). The host may include multiple processing cores, which can execute multiple tasks simultaneously, significantly improving the computing power and performance of servers, including heterogeneous computing systems.

[0031] The device is used to handle large-scale computational tasks (such as model training or inference) in artificial intelligence applications. The device enables computations on AI models, which can be implemented using operators in a computation graph. A computation graph is a multi-graph structure used to represent the computational tasks and data flow processes of an AI model, and operators refer to various operations performed on tensors at different layers of the AI ​​model.

[0032] The device includes a hardware resource pool and video memory. The hardware resource pool includes multiple computing resources for executing tasks of operators in the model or tasks of pre- and post-processing operators in the model. Each computing resource may include one or more programmable multiprocessors, such as a stream processor cluster (SPC). It should be understood that the device may also include other modules or components, and this application does not limit this. This application also does not limit the number of computing resources included in the hardware resource pool.

[0033] It should be understood that the aforementioned host and equipment may include more or fewer components, which will not be elaborated here.

[0034] Currently, in model training or inference scenarios, for computational tasks initiated by a single process, pre-allocated GPU memory space is obtained from the GPU's memory by calling memory allocation functions. During the execution of the computational task, the pre-allocated memory space is managed by a user-space memory manager, for example, allocating corresponding memory blocks from the pre-allocated memory space for each request initiated within the process. In related technologies, the overall usage of GPU memory is indirectly measured through interfaces provided by the driver layer or the CUDA toolkit. These tools can only provide macroscopic data such as total memory usage and remaining memory, but cannot track the specific usage of the pre-allocated memory space for a single process, nor can they obtain fragmentation information corresponding to the pre-allocated memory space. Consequently, it is difficult to cope with the risk of Out of Memory (OOM) caused by memory fragmentation.

[0035] In view of this, this application is based on Figure 1 The diagram of the heterogeneous computing system illustrates a memory allocation method to mitigate the risk of insufficient video memory due to memory fragmentation. This application provides a memory allocation method that can be... Figure 1 The host shown can be executed, or it can be executed by including Figure 1 The server in the heterogeneous computing system shown executes the commands.

[0036] The server also includes, for example, Figure 2 The key software components shown include user-space worker processes, an optimization engine, and a visualization monitoring module, as well as kernel-space GPU drivers, eBPF probes, and buffers. These buffers, for example, use a ring buffer (BPF_MAP_TYPE_RINGBUF) as a data channel between kernel and user space. These key software components can reside on or run on the host machine.

[0037] The workflow includes training / inference programs, deep learning frameworks, and GPU runtime APIs. The training / inference programs include the code for training or inferring deep learning models, and deep learning frameworks such as PyTorch or TensorFlow provide APIs and computation graph management functionality. The GPU runtime APIs are the CUDA APIs called by the deep learning frameworks, such as runtime memory allocation functions (cudaMalloc) and runtime memory deallocation functions (cudaFree).

[0038] The optimization engine includes modules for calculating fragmentation rate, generating optimization strategies, executing optimization strategies, and evaluating optimization effects. The fragmentation rate calculation module accurately calculates the internal, external, and overall fragmentation rates based on collected event data. The optimization strategy generation module automatically generates optimized allocation strategies based on the overall fragmentation rate and memory allocation pattern. The optimization strategy execution module feeds the generated allocation strategies back to the deep learning framework via environment variables or APIs. The deep learning framework may also include a memory manager to allocate pre-allocated memory space according to the allocation strategy. The optimization effect evaluation module monitors the effect of the optimized allocation strategy after execution, forming a closed-loop feedback loop to continuously improve the allocation strategy.

[0039] The visualization monitoring module includes a data dashboard (e.g., Grafana) and a monitoring database (e.g., Prometheus). The monitoring database stores memory status metrics data collected from the optimization engine. The data dashboard visually displays the memory status metrics data from the monitoring database in graphical form.

[0040] GPU drivers act as a bridge between the operating system and GPU hardware, translating API calls into hardware instructions.

[0041] The eBPF probe is set in kernel mode to capture memory allocation / deallocation events in kernel or user mode. It records event data such as process ID, thread ID, and memory allocation / deallocation events, and stores the event data in a buffer. The optimization engine can then retrieve the event data from the buffer for processing.

[0042] Based on the above description, the memory allocation method provided in this application will be described in detail below. For example... Figure 3 As shown, this video memory allocation method includes the following steps:

[0043] Step 301: Obtain the first video memory event corresponding to the first process from the driver layer. The first video memory event is used to request pre-allocated first video memory space from video memory for the computing task initiated by the first process.

[0044] In this embodiment of the application, a probe program runs on the host in the server. The probe program can be set in kernel mode, and step 301 can be executed by the probe program.

[0045] The first process can be a program instance used to perform computational tasks. Each process has its own independent address space, system resources (such as memory, file descriptors, etc.), and at least one execution thread. The computational task can be, for example, a training task or an inference task.

[0046] The first memory space can be a large memory block in the GPU's memory, which is a memory block pre-allocated from the GPU's memory for the computing task initiated by the first process.

[0047] In step 301 above, the first video memory event can be a video memory allocation event or a video memory release event obtained by the probe program from the driver layer.

[0048] For example, the probe program is an eBPF program, which implements event capture by hooking the memory allocation / release API or driver layer interface. The eBPF function entry probes and function return probes mechanism can be used to set eBPF programs at the entry and return points of the driver layer memory allocation function cuMemAlloc, the driver layer memory release function cuMemFree, the runtime memory allocation function cudaMalloc, and the runtime memory release function cudaFree. When a deep learning framework calls memory allocation functions, such as `cuMemAlloc` and `cudaMalloc`, the eBPF program can capture memory allocation events. A probe at the entry point of the memory allocation function can capture the amount of memory requested, while a probe at the return point can capture the actual amount of memory allocated and its corresponding memory address. Similarly, when a deep learning framework calls memory release functions, such as `cuMemFree` and `cudaFree`, the eBPF program can capture memory release events. A probe at the entry point of the memory release function can capture the amount of memory to be released, while a probe at the return point can capture the actual amount of memory released and its corresponding memory address. Memory monitoring can be achieved through eBPF programs without modifying application code, avoiding the intrusive problems of traditional methods (such as inserting `printf` statements).

[0049] Through step 301 above, relevant information about the first video memory space pre-allocated for the first process in the GPU's video memory can be obtained, such as the total video memory amount of the first video memory space, which can be understood as the size of the first video memory space, or the total capacity, or the total storage amount.

[0050] After acquiring the first video memory space pre-allocated for the first process, the user-mode video memory manager manages the first video memory space, such as dividing the first video memory space into video memory blocks. For example, when a video memory block in the first video memory space needs to be used during the execution of a computing task, a request will be triggered. This request is used to request a smaller video memory block from the first video memory space. For example, if the total capacity of the first video memory space is 100M, during the execution of the computing task, request 1 requests a 5M video memory block, and request 2 requests a 10M video memory block. The video memory manager can allocate video memory blocks from the first video memory space for each request of the first process. The actual capacity of the video memory block actually allocated to each request is greater than or equal to the requested capacity in the request.

[0051] Step 302: Obtain the second video memory events corresponding to the multiple requests triggered during the execution of the computing task from the user space. The second video memory event corresponding to a request is used to request the allocation of video memory blocks or release video memory blocks from the first video memory space for a request.

[0052] Step 302 can also be performed by the aforementioned probe program.

[0053] The second memory event can be a memory allocation event or a memory deallocation event obtained by the probe program from user space. For example, it can be obtained from the user-space memory manager. This second memory event can be triggered when the memory manager calls a memory management function in the deep learning framework. For instance, when a memory allocation function in the deep learning framework is called, the probe program can obtain a memory allocation event from the memory manager; similarly, when a memory deallocation function in the deep learning framework is called, the probe program can obtain a memory deallocation event from the memory manager.

[0054] Through step 302 above, the allocation status of the first video memory space can be obtained, such as whether the video memory blocks of the first video memory space have been allocated, and the actual usage of the video memory blocks.

[0055] After acquiring the first memory event and multiple second memory events, the eBPF probe records the event data corresponding to each memory event. Optionally, the eBPF probe can pass the event data to the optimization engine after acquisition, or it can store the event data in a buffer and retrieve it from the buffer when needed by the optimization engine. Event data includes, for example, the timestamp of the event occurrence, process ID (PID), thread ID (TID), allocation address, allocation unit size, process name, and CUDA stream information. For example, when the allocation unit is the first memory space, the allocation unit size can be understood as the size of the first memory space, or the total capacity or total storage of the first memory space; similarly, when the allocation unit is a memory block obtained by dividing the first memory space, the allocation unit size can be understood as the size of the memory block, or the capacity or storage of the memory block.

[0056] Step 303: Determine the overall fragmentation rate of the first video memory space based on the first video memory event and the second video memory events corresponding to the multiple requests respectively; the overall fragmentation rate is generated after allocating video memory to the first video memory space based on the first allocation strategy.

[0057] For example, step 303 can be performed by the aforementioned Figure 1 The optimization engine in the process can obtain the event data corresponding to the first video memory event and multiple second video memory events from the buffer, and then determine the overall fragmentation rate corresponding to the first video memory space based on the event data.

[0058] The first allocation strategy can be a preset allocation strategy or an allocation strategy obtained from the previous adjustment.

[0059] In one possible implementation, the internal fragmentation rate and external fragmentation rate of the first memory space are determined based on the first memory event and the second memory events corresponding to the multiple requests, respectively. Then, the overall fragmentation rate of the first memory space is determined based on the internal fragmentation rate and external fragmentation rate of the first memory space.

[0060] The internal fragmentation rate of the first video memory space can be achieved by determining the total video memory volume of the first video memory space based on the first video memory event. The total allocated capacity of the allocated memory blocks is calculated based on the second memory event. and the actual usage of each memory block Calculate the internal fragmentation rate IFR, where IFR = .

[0061] For example, memory block A corresponds to 1024kb of memory, with 1000kb actually used and 24kb wasted; memory block B corresponds to 2048kb of memory, with 2000kb actually used and 48kb wasted; memory block C corresponds to 512kb of memory, with 500kb actually used and 12kb wasted; the total allocated capacity of memory blocks A, B, and C is 3584kb, the total wasted capacity of the allocated memory blocks is 84kb, and the internal fragmentation rate is the ratio of the total wasted capacity to the total allocated capacity, which is 2.34%.

[0062] The external fragmentation rate of the first memory space can be achieved by calculating the external fragmentation rate based on the address difference between adjacent memory blocks in the second memory event. ,in ,in M represents the total free video memory, and M represents the maximum size of a contiguous free block.

[0063] After determining the internal and external fragmentation rates of the first memory space, the overall fragmentation rate can be determined by a weighted sum of the internal and external fragmentation rates. For example, this can be achieved using a weighted formula. Calculate the overall fragmentation rate ,in As the weight of the external fragmentation rate, As the weight of the internal fragmentation rate, .

[0064] in and The value can be dynamically adjusted based on the degree of impact of fragmentation type. If the external fragmentation rate has a greater impact on the computation task, then it can be adjusted accordingly. Increase it; if the internal fragmentation rate has a greater impact on the computing task, then you can... If the value is increased, for example, if the internal fragmentation rate has a greater impact on training performance, then the value can be increased. Increase the value.

[0065] In this embodiment, not only can multiple fragmentation types be distinguished, but also a comprehensive evaluation of video memory usage can be provided by calculating the internal fragmentation rate (waste caused by memory alignment) and the external fragmentation rate (waste caused by address gaps).

[0066] Step 304: If the overall fragmentation rate of the first video memory space meets the first preset condition, determine the second allocation strategy; the overall fragmentation rate of the second allocation strategy is less than the overall fragmentation rate of the first allocation strategy.

[0067] Step 304 can be derived from the aforementioned Figure 1 The optimization engine in the process is executed.

[0068] In some embodiments, the first preset condition includes: the overall fragmentation rate (OFR) is greater than or equal to a first threshold. ,in The value range is 0.3≤ ≤0.7.

[0069] Taking a first threshold of 0.3 as an example, if the overall fragmentation rate corresponding to the first video memory space is greater than or equal to 0.3, it means that the overall fragmentation rate corresponding to the first video memory space meets the first preset condition and the allocation strategy needs to be adjusted, that is, continue to execute step 304; otherwise, if the overall fragmentation rate corresponding to the first video memory space is less than 0.3, no video memory optimization is performed, that is, the first allocation strategy is still used to allocate video memory to the first video memory space.

[0070] In this embodiment, the fragmentation rate can also be graded for optimization. For example, an overall fragmentation rate greater than 0.7 indicates a high fragmentation level; an overall fragmentation rate greater than 0.3 and less than 0.7 indicates a medium fragmentation level; and an overall fragmentation rate less than 0.3 indicates a low fragmentation level. Different allocation strategies can be set for different fragmentation levels.

[0071] In one method for implementing step 304 above, if the overall fragmentation rate corresponding to the first video memory space meets the first preset condition, a second allocation strategy corresponding to the current video memory allocation mode can be determined. The specific process includes the following steps:

[0072] S1, based on the second video memory event, extract the video memory allocation feature information corresponding to the first process.

[0073] The memory allocation characteristic information includes at least one of the following: the allocation frequency of memory blocks and its standard deviation, the distribution entropy of memory block allocation capacity and its standard deviation, the address jump index gap distribution of memory blocks, and the lifetime of memory blocks and its standard deviation. The allocation frequency of memory blocks is the ratio between the number of events and the time interval. For example, in a time interval of t seconds, N second memory events are collected, where the allocation frequency of memory blocks is the ratio of N to t. The distribution entropy of memory block allocation capacity describes the uniformity of memory block allocation capacity distribution. The more uniform the distribution of memory blocks of different allocation capacities, the higher the distribution entropy; conversely, the more uneven the distribution, the lower the distribution entropy. For example, if the probability of 10MB, 20MB, and 30MB memory blocks appearing is the same, the distribution entropy is 1. However, if the probability of a 10MB memory block appearing is 60%, a 20MB block 30%, and a 30MB block 10%, then the distribution entropy is less than 1. The gap distribution of the jump index is determined by calculating the variance of the difference between consecutive allocated addresses. The memory block lifetime is the time span from successful allocation to complete release of a single memory block. The standard deviation of any of the above parameters (such as the allocation frequency of memory blocks, the distribution entropy of memory block allocation capacity, or the lifetime of memory blocks) is an indicator of the degree of dispersion of the parameter values. Taking the standard deviation of the lifetime of memory blocks as an example, the standard deviation of the lifetime of memory blocks is an indicator of the degree of dispersion of the lifetime of all memory blocks. The larger the standard deviation, the greater the difference in the lifetime of each memory block, and the smaller the standard deviation, the smaller the difference in the lifetime of each memory block.

[0074] S2, determine the memory allocation mode of the first process based on the memory allocation characteristic information corresponding to the first process.

[0075] In one possible implementation, the following three judgments can be made sequentially based on the memory block allocation frequency, the memory block allocation capacity distribution entropy, and the memory block address gap distribution:

[0076] (1) If the allocation frequency of the video memory block corresponding to the first process is greater than the first preset value, and the distribution entropy of the allocated capacity of the video memory block corresponding to the first process is less than the second preset value, then the video memory allocation mode is identified as a high-frequency small-scale allocation mode.

[0077] (2) If the address jump index of the video memory block corresponding to the first process is greater than the third preset value, the video memory allocation mode will be identified as a distributed allocation mode.

[0078] (3) If the standard deviation of the memory block allocation frequency corresponding to the first process is less than the fourth preset value, and the standard deviation of the distribution entropy of the memory block allocation capacity corresponding to the first process is less than the fifth preset value, and the standard deviation of the memory block lifetime is less than the sixth preset value, then the memory allocation mode is identified as a periodic allocation mode.

[0079] If none of the above three conditions are met, the memory allocation mode of the first process will be identified as an unknown mode, and general optimization can be enabled. The order of the above three conditions can also be adjusted, for example, the conditions can be judged in the order of (1), (2), (3), or in the order of (2), (1), (3). This application does not limit the order of the conditions.

[0080] In another possible implementation, the memory allocation model of the first process can be determined based on a machine learning model, which can be obtained by learning the relationship between different allocation patterns and the overall fragmentation rate through historical data.

[0081] S3 uses the allocation strategy corresponding to the memory allocation mode of the first process as the second allocation strategy.

[0082] For example, the allocation strategy corresponding to the high-frequency, small-scale allocation mode is a memory pool allocation strategy, where the memory pool includes a set of video memory blocks with a preset granularity. For instance, if the first video memory space includes 1000MB, and the preset granularity in the first allocation strategy is 10MB, the first video memory space is divided into 100 video memory blocks, meaning the memory pool includes 100 10MB video memory blocks. Since the internal fragmentation rate is high when executing the first allocation strategy, and considering the characteristics of the high-frequency, small-scale allocation mode where each request requires a relatively small number of video memory blocks, the maximum value of the video memory block can be adjusted to 5MB in the second allocation strategy. Then, the first video memory space is divided into 200 video memory blocks, meaning the memory pool includes 200 5MB video memory blocks. This reduces the internal fragmentation rate and improves video memory utilization.

[0083] For example, the allocation strategy corresponding to the distributed allocation mode is to adopt a free block merging strategy based on the buddy system. In the distributed allocation mode, the memory requirements of multiple requests corresponding to a process vary greatly, which can easily lead to memory fragmentation. Some large-capacity memory blocks and some small-capacity memory blocks can be pre-allocated in batches in the memory pool. The large-capacity memory blocks can be obtained by merging adjacent small-capacity free blocks, thereby increasing the available contiguous memory addresses, avoiding large block allocation failures, and reducing the external fragmentation rate.

[0084] The allocation strategy corresponding to the periodic allocation mode is a pre-allocation and cache reuse strategy. The characteristics of the periodic allocation mode are relatively regular memory allocation and release, and multiple requests within the same process for memory blocks of the same capacity (or size). For example, in iterative training scenarios, the memory needed for the entire cycle can be pre-allocated before the start of iterative training, eliminating allocation latency during the process. Alternatively, frequently used memory blocks can be cached and reused after the cycle ends, thus reducing allocation overhead.

[0085] In this embodiment, a targeted and optimized allocation strategy is generated based on the characteristics of the video memory allocation mode (high frequency small scale, distributed, periodic, etc.) and the fragmentation rate threshold. Compared with manual judgment and adjustment of the allocation strategy, this embodiment can realize intelligent video memory management.

[0086] Step 305: Based on the second allocation strategy, allocate video memory to the first video memory space.

[0087] For example, the second allocation strategy can be updated in the memory manager so that the memory manager can manage the first memory space according to the second allocation strategy.

[0088] In this embodiment, by obtaining the first memory event corresponding to the first process from the driver layer, the pre-allocated first memory space can be obtained. By obtaining the second memory events corresponding to multiple requests triggered during the execution of the computing task from the user space, the allocation and release status of memory blocks in the pre-allocated first memory space can be obtained. This allows for the accurate determination of the overall fragmentation rate of the first memory space. Then, if the overall fragmentation rate of the first memory space meets a first preset condition, the allocation strategy is adjusted in the direction of reducing the overall fragmentation rate. Finally, memory allocation is performed on the first memory space according to the adjusted second allocation strategy, thereby helping to reduce the risk of insufficient memory caused by memory fragmentation. Moreover, this embodiment can effectively reduce the memory fragmentation rate and improve memory utilization and training performance.

[0089] After step 305, the effect of using the second allocation strategy can also be monitored. In some embodiments, the video memory usage corresponding to the first video memory space is obtained; based on the video memory usage, the optimization effect of video memory fragmentation corresponding to the first video memory space is determined; for example, before using the second allocation strategy, video memory status indicators are collected, including but not limited to at least one of the following: internal fragmentation rate, external fragmentation rate, overall fragmentation rate, OOM count, video memory utilization, etc. After executing the second allocation strategy, the video memory status indicators are collected again, and then the video memory status indicators before and after using the second allocation strategy are compared. If the optimization effect of video memory meets the second preset condition, the second allocation strategy continues to be used, and video memory is allocated to the first video memory space; for example, the second preset condition includes: the difference between the video memory status indicators after using the second allocation strategy and the video memory status indicators before using the second allocation strategy is greater than or equal to the preset optimization effect threshold corresponding to the video memory status indicators. If the optimization effect of the video memory does not meet the second preset condition, for example, if the difference between the video memory status index after using the second allocation strategy and the video memory status index before using the second allocation strategy is less than the preset optimization effect threshold corresponding to the video memory status index, then a third allocation strategy is determined. The third allocation strategy is different from the second allocation strategy. According to the third allocation strategy, video memory is allocated to the first video memory space.

[0090] In some embodiments, a new overall fragmentation rate OFR′ is calculated based on the monitored memory usage of the first memory space; if the difference between OFR′ and the original overall fragmentation rate OFR is greater than or equal to a preset optimization effect threshold... If the difference between OFR′ and the overall fragmentation rate OFR before optimization is less than the preset optimization effect threshold, then the second allocation strategy will continue to be used to allocate memory in the first memory space. Then, the third allocation strategy is determined. The overall fragmentation rate (OFR) before optimization can be the overall fragmentation rate determined in step 303, that is, the overall fragmentation rate after allocating the first memory space using the first allocation strategy, and also the overall fragmentation rate before allocating the first memory space using the second allocation strategy.

[0091] The method for determining the third allocation strategy can refer to the method for determining the second allocation strategy in steps 301 to 304, and will not be repeated here.

[0092] In some embodiments, taking the internal fragmentation rate as the memory status indicator and the second preset condition as an improvement of 20% in the internal fragmentation rate, the internal fragmentation rate before using the second allocation strategy is 20%, and the internal fragmentation rate after using the second allocation strategy is 10%, indicating that the internal fragmentation rate has been improved by 50%, indicating that the memory optimization effect meets the second preset condition, and the second allocation strategy is continued to be used to allocate memory in the first memory space.

[0093] Optionally, the aforementioned memory status indicators can also be visualized, for example, by displaying at least one memory status indicator on the user interface.

[0094] The memory allocation method provided in this application embodiment can integrate eBPF monitoring into the training process of various deep learning frameworks, such as PyTorch and TensorFlow.

[0095] The following example demonstrates the integration of eBPF monitoring into the PyTorch framework training process. The steps are as follows: First, training begins, eBPF monitoring is enabled, and the first memory event corresponding to the first process and multiple second memory events are collected. Second, the PyTorch memory allocator parameters are configured. Third, the training loop begins, training steps are executed, and intelligent monitoring intervenes. The training function is the original training code. Intelligent monitoring integrates real-time acquisition of the current overall fragmentation rate. If the overall fragmentation rate is greater than 30%, an optimization process is triggered. The allocation strategy is optimized based on the current memory usage, and the optimized allocation strategy is executed. Fourth, training continues. During each training cycle, the system automatically monitors memory usage, analyzes fragmentation rate changes, triggers optimization strategies when optimization conditions are met, and verifies the optimization effect.

[0096] It should be understood that the training process of integrating eBPF monitoring into the TensorFlow framework is similar to the training process of integrating eBPF monitoring into the PyTorch framework described above, and will not be repeated here. The difference is that the second step is to configure the TensorFlow memory allocator parameters.

[0097] Based on the same inventive concept, this application provides a computer device, such as... Figure 4 The computer device 400 shown includes at least one processor 401 and a memory 402, in which programs or data are stored.

[0098] Memory 402 may be volatile memory, such as random access memory; memory may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 402 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 402 may be a combination of the above-described memories.

[0099] Optionally, processor 401 may be one or more central processing units.

[0100] Optionally, the process executed by the processor 401 can refer to the method flow described in the foregoing method embodiments, and will not be repeated here.

[0101] Optionally, although Figure 4 As not shown in the diagram, the computer device 400 may also include at least one input / output interface, which can be used to connect a mouse, keyboard, etc. The input / output interface is an optional component and may or may not be present; it is not limited here.

[0102] Based on the same inventive concept, this application provides a computer device, which includes a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip. When the processor chip executes the computer program, it implements the method steps in any of the above method embodiments.

[0103] Based on the same technical concept, embodiments of this application provide a computer-readable storage medium including computer-executable instructions, which, when executed on a computer, cause the computer to perform the method steps in any of the above method embodiments.

[0104] Based on the same technical concept, this application provides a computer program product that stores a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer performs the method steps in any of the above method embodiments.

[0105] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), systems, chips, computer-readable storage media, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems".

[0106] This application is described with reference to at least one of the following diagrams of the method, apparatus (device) or system of this application: flowchart, block diagram. It should be understood that at least one of the following can be implemented by computer program instructions: each step in the flowchart, each block in the block diagram, and a combination of steps in the flowchart and blocks in the block diagram.

[0107] These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing at least one of the following: in the process Figure 1 Devices or blocks that specify functions in one or more processes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means that performs at least one of the following: in the process Figure 1 The specified function or box in one or more processes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing at least one of the following: steps for a function specified in one or more processes in a flowchart, or steps for a function specified in one or more blocks in a block diagram.

[0110] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for allocating video memory, characterized in that, include: Obtain the first video memory event corresponding to the first process from the driver layer. The first video memory event is used to request a pre-allocated first video memory space from the video memory for the computing task initiated by the first process. The second video memory events corresponding to multiple requests triggered during the execution of the computing task are obtained from the user space. The second video memory event corresponding to a request is used to request the allocation of a video memory block or release a video memory block from the first video memory space for the request. Based on the first video memory event and the second video memory events corresponding to the multiple requests, the overall fragmentation rate corresponding to the first video memory space is determined; the overall fragmentation rate is generated after allocating video memory to the first video memory space based on the first allocation strategy. If the overall fragmentation rate of the first video memory space meets the first preset condition, a second allocation strategy is determined; the overall fragmentation rate of the second allocation strategy is less than the overall fragmentation rate of the first allocation strategy. Based on the second allocation strategy, memory allocation is performed on the first memory space.

2. The method as described in claim 1, characterized in that, The determination of the second allocation strategy includes: Based on the second video memory event, extract the video memory allocation feature information corresponding to the first process; Based on the memory allocation feature information corresponding to the first process, the memory allocation mode of the first process is determined. The allocation strategy corresponding to the video memory allocation mode of the first process is used as the second allocation strategy.

3. The method as described in claim 2, characterized in that, The memory allocation characteristic information includes at least one of the following: memory block allocation frequency and its standard deviation, memory block allocation capacity distribution entropy and its standard deviation, memory block address jump index, and memory block lifetime and its standard deviation. The step of determining the video memory allocation mode of the first process based on the video memory allocation feature information corresponding to the first process includes: If the allocation frequency of the video memory block corresponding to the first process is greater than the first preset value, and the distribution entropy of the video memory block allocation capacity is less than the second preset value, then the video memory allocation mode is identified as a high-frequency small-scale allocation mode. If the address jump index of the video memory block corresponding to the first process is greater than the third preset value, then the video memory allocation mode is identified as a distributed allocation mode. If the standard deviation of the allocation frequency of the memory block corresponding to the first process is less than the fourth preset value, the standard deviation of the distribution entropy of the memory block allocation capacity is less than the fifth preset value, and the standard deviation of the memory block lifetime is less than the sixth preset value, then the memory allocation mode is identified as a periodic allocation mode.

4. The method as described in claim 3, characterized in that, The allocation strategy corresponding to the high-frequency small-scale allocation mode is to adopt a memory pool allocation strategy, wherein the memory pool includes a set of video memory blocks with a preset granularity; the allocation strategy corresponding to the distributed allocation mode is to adopt a free block merging strategy based on a buddy system; and the allocation strategy corresponding to the periodic allocation mode is a pre-allocation and cache reuse strategy.

5. The method as described in claim 1, characterized in that, The step of determining the overall fragmentation rate of the first video memory space based on the first video memory event and the second video memory events corresponding to the plurality of requests includes: The total video memory size of the first video memory space is determined based on the first video memory event. ; According to the statistics of the second memory event, the total allocated capacity of the allocated memory blocks has been counted. and the actual usage of each memory block The internal fragmentation rate (IFR) is calculated using the following formula: IFR= ; Based on the address difference between adjacent memory blocks in the second memory event, the external fragmentation rate is calculated using the following formula. : ,in M represents the total free video memory, and M represents the maximum capacity of a contiguous free block. The overall fragmentation rate is calculated using the following weighted formula: ,in For the overall fragmentation rate, As the weight of the external fragmentation rate, As the weight of the internal fragmentation rate, .

6. The method as described in claim 1, characterized in that, The first preset condition is that the overall fragmentation rate is greater than or equal to a first threshold. ,in The value range is 0.3≤ ≤0.

7.

7. The method according to any one of claims 1 to 6, characterized in that, After allocating video memory to the first video memory space based on the second allocation strategy, the method further includes: Monitor the memory usage of the first memory space and calculate the new overall fragmentation rate OFR′; If the difference between OFR′ and the overall fragmentation rate OFR before optimization is less than the preset optimization effect threshold If so, a third allocation strategy is determined, which is different from the second allocation strategy; According to the third allocation strategy, the first video memory space is allocated.

8. The method according to any one of claims 1 to 6, characterized in that, Any video memory event is collected by an eBPF program set in kernel mode. The eBPF program captures events by hooking the video memory allocation / release API or the driver layer interface.

9. A computer device, comprising a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip, characterized in that, When the processor chip executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It includes computer-executable instructions that, when executed on a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 8.

Images