Access-aware gpu memory latency allocation method, apparatus
By using an access-aware, delayed memory allocation method, the problem of low GPU memory utilization efficiency is solved, achieving efficient utilization of memory resources and improving system performance. This method is suitable for CPU-GPU heterogeneous computing systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-06-22
- Publication Date
- 2026-07-24
Smart Images
Figure CN122450684A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer memory allocation technology, and for example to an access-aware GPU memory latency allocation method and device. Background Technology
[0002] With the rapid development of high-performance computing applications such as deep learning, graph computing, and large-scale data processing, graphics processing units (GPUs) have become the main computing acceleration platform due to their high parallel computing capabilities. However, despite the continuous improvement of GPU computing power, the growth of its on-device memory capacity is relatively limited, especially in embedded and edge computing scenarios, making memory resources one of the key bottlenecks restricting system performance and scalability.
[0003] To address the issue of GPU memory constraints, existing technologies primarily optimize memory usage by reducing memory requirements or improving memory utilization, but they still lack a systematic improvement at the level of memory allocation timing. Specifically, existing technologies can be broadly categorized into three types: the first type is memory requirement optimization methods, which reduce memory consumption through model compression, low-precision computation, or operator-level optimization; the second type is memory capacity expansion methods, which expand available memory in time or space through data exchange, cross-device migration, or recomputation; and the third type is runtime memory management methods, which improve memory utilization efficiency through memory pools, paging mechanisms, or dynamic allocation strategies.
[0004] However, most of the above methods are based on the traditional assumption of "allocation equals occupancy," assuming that device memory allocation is completed when a memory allocation request is issued, without considering the timing characteristics of actual access. In a typical GPU execution flow, memory allocation usually occurs first, followed by data copying or computation operations, resulting in a time interval between memory allocation and actual access—the allocation-access gap. During this interval, memory is occupied but not yet accessed, causing resource idleness, reducing memory utilization, and potentially increasing peak memory usage. Even with optimizations through adjusting the execution order, existing technologies struggle to fundamentally eliminate this problem. Therefore, in multi-task concurrency or memory-constrained scenarios, existing technical solutions suffer from low GPU memory utilization efficiency and high system memory pressure.
[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application. Summary of the Invention
[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.
[0007] The access-aware GPU memory latency allocation method and device provided in this disclosure can solve the problems of low GPU memory utilization efficiency and high system memory pressure in multi-task concurrency or memory-constrained scenarios.
[0008] This disclosure provides an access-aware GPU memory latency allocation method, which may include: When a memory allocation request is received from an application, the allocation metadata related to the memory allocation request is recorded. However, no actual device memory allocation operation is performed, thus completing a logical pseudo-allocation and putting the buffer corresponding to the memory allocation request in a logically allocated but physically unallocated state. When the first data access request for the buffer is detected, the on-demand allocation mechanism is triggered, and the actual device memory allocation operation is performed according to the recorded allocation metadata to allocate physical memory for the buffer; After the actual allocation operation is completed, an address backfilling operation is performed, which combines the allocated physical base address with the logical offset corresponding to the data access request to generate the final physical access address in order to execute the data access request.
[0009] In some embodiments, the above-mentioned recording of allocation metadata related to memory allocation requests includes: Create a metadata record for each memory allocation request, which includes at least the allocation command identifier, device identifier, process identifier, and requested memory size information, and store the metadata record in the memory management structure on the driver side.
[0010] In some embodiments, triggering the on-demand allocation mechanism upon detecting the first data access request for the buffer includes: When the GPU-side command processing unit processes a data access request, it checks the actual allocation status of the requested target buffer. If the status is unallocated, the GPU-side command processing unit suspends data access requests and sends an allocation trigger signal to the CPU-side driver. The CPU-side driver responds to the allocation trigger signal, performs the actual device memory allocation operation based on the corresponding allocation metadata, and returns the allocated base address to the GPU side.
[0011] In some embodiments, the above-described address backfilling operation includes: The GPU-side command processing unit receives the base address returned by the CPU-side driver; For each data access request to be executed, the offset representing the relative position is combined with the base address to generate the final physical access address and update the corresponding request structure; Once the address has been filled, the data access request is placed in the execution queue, and the DMA engine performs the data copying.
[0012] In some embodiments, the above method further includes assigning a unique identifier to each memory allocation command and data access command, and establishing a mapping relationship between the two; Maintain a command status table on the GPU side. The command status includes at least: a non-triggered status indicating that allocation has not yet been triggered, a triggered allocation status indicating that an allocation request has been sent and is waiting for completion, and a completed allocation status indicating that allocation has been completed and can be directly accessed.
[0013] In some embodiments, the above method further includes unified control of data access requests based on a command status table; This includes unified control of data access requests based on command status tables, including: When a data access request arrives for the first time and the corresponding status is not triggered, the status is updated to the triggered allocation status, and the on-demand allocation mechanism is triggered. When other data access requests arrive and the corresponding status is "allocation triggered", keep the request in a suspended waiting state; When the status is "allocation completed", the address backfilling operation and data transmission are performed directly.
[0014] In some embodiments, the data access request described above is a data copy request from the host to the device. By delaying the actual allocation of physical memory to align with the data copy request, the idle time gap between memory allocation and the first access in the traditional process is eliminated.
[0015] In some embodiments, during the pseudo-allocation phase, no device memory request is performed, no page table is updated, and no physical page frames are allocated; only a logical handle or virtual placeholder is returned to the upper-layer application.
[0016] In some embodiments, before detecting the first data access request for the buffer, the method further includes: The data access request is split into multiple sub-requests, and each sub-request is configured with a logical offset of its relative position in the target buffer. These sub-requests are sent to the GPU-side command processing unit, enabling the construction and distribution of data access requests to be completed during the pseudo-allocation phase, without relying on the completed physical address.
[0017] This disclosure provides an electronic device that includes at least one processor; and memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the aforementioned access-aware GPU memory latency allocation method.
[0018] The access-aware GPU memory latency allocation method and device provided in this disclosure can achieve the following technical effects: This disclosure presents an access-aware GPU memory latency allocation method, constructing a "placement first, allocation later" latency memory management logic. This scheme overturns the traditional "allocation equals occupancy" instant mapping mechanism, reconstructing the timing relationship between memory allocation and data access to solve the GPU memory idleness problem. Specifically, when the application layer initiates a memory allocation request, the system only performs a "pseudo-allocation" operation, that is, only records metadata and returns a logical handle, without requesting physical memory. Only when the system detects the first actual data access request for that memory region does this access behavior drive the actual physical memory allocation, and then the newly generated physical base address is dynamically updated to the access instruction to be executed using an address backfilling mechanism. This technical solution precisely shifts the actual physical memory occupancy time window to align with the first access time, fundamentally eliminating the allocation-access gap existing in the traditional process. The technical effects achieved are: while ensuring complete transparency of program execution semantics and introducing only a very slight performance overhead, it significantly reduces memory unutilization time and significantly compresses peak memory usage in multi-task concurrent scenarios, thereby greatly improving GPU memory utilization efficiency and system concurrent processing capabilities.
[0019] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description
[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart illustrating an access-aware GPU memory latency allocation method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of an allocation-access gap and delay mechanism provided in an embodiment of this disclosure; Figure 3 This disclosure provides a CPU-GPU system architecture and memory path diagram. Figure 4 This is a schematic diagram of the overall process of access-aware GPU memory latency allocation provided by an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an access-aware GPU memory latency allocation device provided in an embodiment of this disclosure. Detailed Implementation
[0021] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.
[0022] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.
[0023] Unless otherwise stated, the term "multiple" means two or more.
[0024] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.
[0025] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.
[0026] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.
[0027] To address the aforementioned issues, this disclosure provides an access-aware GPU memory latency allocation method, device, and storage medium.
[0028] The following description, in conjunction with the accompanying drawings, illustrates the access-aware GPU memory latency allocation method, device, and storage medium provided in the embodiments of this disclosure.
[0029] Figure 1 This is a flowchart illustrating an access-aware GPU memory latency allocation method provided in an embodiment of this disclosure.
[0030] Combination Figure 1 As shown, the access-aware GPU memory latency allocation method can specifically include: S101, when a memory allocation request is received from an application, the allocation metadata related to the memory allocation request is recorded, and no actual device memory allocation operation is performed, thereby completing a logical pseudo-allocation, so that the buffer corresponding to the memory allocation request is in a state of logical allocation but physical unallocated. S102, upon detecting the first data access request for the buffer, triggers the on-demand allocation mechanism, performs the actual device memory allocation operation based on the recorded allocation metadata, and allocates physical memory for the buffer; S103 After the actual allocation operation is completed, an address backfilling operation is performed, which combines the allocated physical base address with the logical offset corresponding to the data access request to generate the final physical access address in order to execute the data access request.
[0031] In some embodiments, the above-mentioned recording of allocation metadata related to memory allocation requests includes: Create a metadata record for each memory allocation request, which includes at least the allocation command identifier, device identifier, process identifier, and requested memory size information, and store the metadata record in the memory management structure on the driver side.
[0032] In some embodiments, triggering the on-demand allocation mechanism upon detecting the first data access request for the buffer includes: When the GPU-side command processing unit processes a data access request, it checks the actual allocation status of the requested target buffer. If the status is unallocated, the GPU-side command processing unit suspends data access requests and sends an allocation trigger signal to the CPU-side driver. The CPU-side driver responds to the allocation trigger signal, performs the actual device memory allocation operation based on the corresponding allocation metadata, and returns the allocated base address to the GPU side.
[0033] In some embodiments, the above-described address backfilling operation includes: The GPU-side command processing unit receives the base address returned by the CPU-side driver; For each data access request to be executed, the offset representing the relative position is combined with the base address to generate the final physical access address and update the corresponding request structure. Once the address backfilling is complete, the data access request is placed in the execution queue, and the Direct Memory Access (DMA) engine performs the data copying.
[0034] In some embodiments, Figure 1 The method also includes assigning a unique identifier to each memory allocation command and data access command, and establishing a mapping relationship between the two; Maintain a command status table on the GPU side. The command status includes at least: a non-triggered status indicating that allocation has not yet been triggered, a triggered allocation status indicating that an allocation request has been sent and is waiting for completion, and a completed allocation status indicating that allocation has been completed and can be directly accessed.
[0035] In some embodiments, Figure 1 The method also includes unified control of data access requests based on command status tables; This includes unified control of data access requests based on command status tables, including: When a data access request arrives for the first time and the corresponding status is not triggered, the status is updated to the triggered allocation status, and the on-demand allocation mechanism is triggered. When other data access requests arrive and the corresponding status is "allocation triggered", keep the request in a suspended waiting state; When the status is "allocation completed", the address backfilling operation and data transmission are performed directly.
[0036] In some embodiments, the data access request described above is a data copy request from the host to the device. By delaying the actual allocation of physical memory to align with the data copy request, the idle time gap between memory allocation and the first access in the traditional process is eliminated.
[0037] In some embodiments, during the pseudo-allocation phase described above, no device memory request is performed, no page table is updated, and no physical page frame is allocated; only a logical handle or virtual placeholder is returned to the upper-layer application.
[0038] In some embodiments, prior to detecting the first data access request for the buffer as described above, the method further includes: The data access request is split into multiple sub-requests, and each sub-request is configured with a logical offset of its relative position in the target buffer. These sub-requests are sent to the GPU-side command processing unit, enabling the construction and distribution of data access requests to be completed during the pseudo-allocation phase, without relying on the completed physical address.
[0039] Figure 2 This is a schematic diagram of an allocation-access gap and delay mechanism provided in an embodiment of this disclosure. Figure 3 This disclosure provides a CPU-GPU system architecture and memory path diagram. Figure 4 This is a schematic diagram of the overall process of access-aware GPU memory latency allocation provided in this disclosure embodiment, combined with... Figures 2 to 4 ,right Figure 1 The access-aware GPU memory latency allocation method will be described further.
[0040] Specifically, this disclosure proposes an access-aware GPU memory latency allocation method. This method's corresponding memory latency allocation mechanism identifies the temporal misalignment between memory allocation and actual access during GPU program execution, transforming the traditional "allocation equals occupancy" memory management approach into a dynamic "access-triggered allocation" mechanism. This achieves precise alignment between device memory residency time and actual usage time. For example... Figure 2 As shown, in the traditional GPU operating mechanism, memory allocation operations are executed immediately upon receiving an allocation request, while actual data writing or computational access often occurs in subsequent stages. This creates a significant time interval between memory allocation completion and the first access, known as the allocation-access gap. During this interval, memory is allocated but not yet accessed, resulting in resource idle residency. This disclosure eliminates this time gap by delaying the actual memory allocation timing, aligning it with the first access event, thereby significantly reducing memory idle time. Furthermore, this disclosure refactors the memory allocation and access triggering logic, ensuring that memory is only actually allocated when it is about to be accessed. This effectively shortens memory idle residency time, reduces peak memory usage, and improves overall memory utilization efficiency without altering the original program execution semantics.
[0041] Unlike the traditional execution path of "first completing memory allocation, then performing data copying or computation access," this disclosure introduces an access-driven mechanism into the execution path, allowing data access requests to drive the memory allocation process in reverse. When an access request arrives but the target memory has not yet been actually allocated, the system dynamically triggers the allocation operation and continues the original access process after allocation is completed. This mechanism avoids the problem of memory remaining unused for a long time before access, thereby significantly optimizing the temporal behavior of GPU memory management.
[0042] Regarding the allocation-access timing reconstruction mechanism, during GPU execution, there is typically a non-zero time interval between memory allocation and data access. This time interval is composed of factors such as driver processing latency, command queuing, data transfer, and DMA execution. This disclosure models and analyzes this time interval, defining it as the "allocation-access gap," and proposes a timing reconstruction mechanism to shift the memory allocation operation backward on the timeline, making it closer to the first actual access event.
[0043] Specifically, in traditional mechanisms, page table mapping is established and physical memory resources are occupied immediately after memory allocation, but no access requests have yet been generated on the device side. In this disclosure, however, the system delays the actual allocation operation, triggering physical memory allocation only when the first data access request (e.g., a host-to-device data copy or a device-side write request) is detected as imminent. This eliminates the idle time that previously existed between allocation and access, allowing memory resource occupancy to highly overlap with the actual data writing process, thereby significantly reducing memory unutilization time.
[0044] To address the pseudo-allocation and metadata management mechanism and support the delayed allocation strategy, this disclosure introduces a pseudo-allocation mechanism. Upon receiving a memory allocation request, it does not immediately execute the actual device memory allocation operation, but instead only records metadata information related to the allocation request. This metadata includes, but is not limited to, device identifier, process identifier, requested memory size, and allocation command identifier. This information is uniformly stored in a driver-side data structure for subsequent querying and retrieval when the actual allocation operation is triggered.
[0045] Under this mechanism, the system logically considers memory to have been "allocated," but physically, it does not occupy actual device memory, nor does it establish a mapping relationship between virtual and physical addresses. This "logically allocated, physically unallocated" state allows the system to achieve delayed memory allocation without disrupting the program execution flow. Simultaneously, this mechanism provides the necessary contextual information for subsequent access-triggered allocations, enabling the delayed allocation process to complete smoothly without needing to re-initiate allocation requests.
[0046] Regarding the access-triggered allocation mechanism, after a pseudo-allocation is completed, when the system executes a data access operation involving that memory region, this invention dynamically performs the actual memory allocation through the access-triggered mechanism. Specifically, when the GPU-side command processing unit receives an access request for a certain buffer, it first checks whether the buffer has already been actually allocated. If the detection result is an unallocated state, it immediately sends an allocation trigger signal to the central processing unit (CPU)-side driver, which then performs the actual device memory allocation operation based on pre-recorded allocation metadata.
[0047] During the allocation process, the driver requests the corresponding number of physical pages from the device's memory resource pool and establishes a mapping relationship between virtual addresses and physical addresses. The allocated base address is then returned to the GPU-side execution unit. This mechanism enables allocation operations to be driven by access behavior, making memory allocation timing entirely dependent on actual access needs, thus avoiding the premature allocation problem present in traditional mechanisms.
[0048] Regarding the address backfilling mechanism, since the actual physical address has not yet been generated during the pseudo-allocation phase, the target address of the access request cannot directly point to a specific location in the device memory when it is initially generated. This disclosure introduces an address backfilling mechanism to dynamically update the target address in the access request after the actual allocation is completed.
[0049] Specifically, when a data access request is generated, the system only allocates an offset relative to the start of the buffer; this offset serves as a placeholder for the logical address. Once the actual allocation is complete, the system retrieves the allocated base address and combines it with the offset in the request to generate the final physical access address. This address is then filled back into the original access request, and the request is re-entered into the execution flow.
[0050] Through this mechanism, the system can construct access requests without generating physical addresses in advance, and quickly resume request execution after allocation, thereby ensuring the correctness and continuity of the entire execution process.
[0051] To ensure the correctness of the delayed allocation mechanism in complex execution environments, this disclosure further introduces a command association and state control mechanism to uniformly manage the dependency relationship between memory allocation commands and access commands. Specifically, the system assigns a unique identifier to each allocation command and access command and establishes a mapping relationship between them, enabling access requests to accurately locate their corresponding allocation metadata.
[0052] Meanwhile, the system maintains status information for each access command to describe its current execution stage, including an untriggered state, a triggered allocation state, and a completed allocation state. In the untriggered state, the access request will trigger an allocation operation; in the triggered state, the request enters a waiting state; and in the completed allocation state, the request can directly execute data access. Through this status control mechanism, access requests can be suspended, resumed, and executed sequentially, thereby ensuring the stability and consistency of the delayed allocation process in a concurrent environment.
[0053] In one specific embodiment, the access-aware GPU memory latency allocation method proposed in this disclosure can be applied to CPU-GPU heterogeneous computing systems. For example... Figure 3As shown, the system includes a CPU-side driver, a GPU-side command processor (CP), a DMA engine, and dynamic random access memory (DRAM). These modules communicate with each other via Peripheral Component Interconnect Express (PCIe) or a high-speed interconnect bus. In this system architecture, memory allocation operations are primarily performed by the CPU-side driver, while memory copy operations are initiated by the CPU and completed by the GPU. These two operations are separated in their execution paths, thus providing the implementation basis for the delayed allocation mechanism described in this disclosure.
[0054] Regarding the overall implementation process based on access-driven mechanisms, this embodiment implements an access-triggered delayed allocation mechanism by constructing a unified allocation and access coordination execution process. For example... Figure 4 As shown, the method as a whole includes an input phase, a pseudo-allocation phase, a copy preparation phase, a delayed allocation phase, a data copy phase, and an output phase.
[0055] Specifically, in the input phase, the application submits memory allocation and data access requests to the CPU-side driver through a unified interface; in the pseudo-allocation phase, the driver only records allocation metadata without performing actual allocation; in the copy preparation phase, data access requests are split into multiple sub-requests and sent to the GPU side; when the GPU side detects that the target memory has not yet been allocated, the actual allocation operation is triggered in the delayed allocation phase; subsequently, the actual data writing is performed in the data copy phase; finally, the allocated and initialized data buffer is obtained in the output phase. Through the above process, the transformation from "allocation-driven execution" to "access-driven allocation" is realized.
[0056] Regarding the refactoring of allocation-access timing, in this embodiment, to refactor the memory allocation and access timing, the CPU-side driver only performs logical processing and does not allocate physical resources when handling memory allocation requests. Specifically, the driver calculates the required number of pages based on the requested memory size and maintains an internal allocation record table. This table records the identifier information, target device, required memory size, and process information of each allocation request. Simultaneously, access requests enter the command queue according to the original execution order and are gradually sent to the GPU side.
[0057] When processing an access request, the GPU-side command processing unit checks the corresponding allocation status. If it finds that the target buffer has not yet been actually allocated, it pauses the access request and sends an allocation trigger signal to the CPU-side driver. Upon receiving this signal, the CPU performs the actual allocation operation, thus delaying the allocation process that would normally be executed earlier until the access phase, achieving timing shifting and alignment.
[0058] Regarding the pseudo-allocation and metadata management mechanism, during the pseudo-allocation phase, the CPU-side driver creates an allocation metadata record for each memory allocation request. This record includes at least the allocation command identifier (AllocCmdID), device identifier (DeviceID), process identifier (PID), and requested bytes (ByteSize). This record is stored in a memory management structure maintained by the driver, such as a hash table or linked list structure, to support fast subsequent lookups.
[0059] During this phase, no device memory allocation operations are performed, page tables are not updated, physical page frames are not allocated, and the actual device address is not returned. Only a logical handle or virtual placeholder identifier is returned to the upper layer. In this way, the system avoids occupying device memory resources before the access occurs, while preserving complete allocation context information to provide the necessary basis for subsequent allocation triggers.
[0060] Regarding the access-triggered allocation mechanism, during the copy preparation phase, when the GPU-side command processing unit receives a data access request, it first queries the corresponding allocation status table based on the copy command identifier (CopyCmdID) in the request. For example... Figure 4 As shown, a triggering relationship is established between the copy preparation phase and the delayed allocation phase. When the current copy request is detected to be in an unallocated state, the command processing unit sends a memory allocation trigger request (MemAllocTrig) to the CPU-side driver.
[0061] During the delayed allocation phase, after receiving the trigger request, the CPU-side driver performs the actual device memory allocation operation based on the metadata information recorded in the pseudo-allocation phase, and returns the allocated base address to the GPU-side command processing unit through the memory allocation reply (MemAllocRsp), thereby completing the memory allocation process driven by access behavior.
[0062] Regarding the address backfilling mechanism, during the copy preparation phase, when a memory copy command is split into multiple memory copy requests, since actual memory allocation has not yet been completed, the target address field in each access request is represented in offset form, indicating the relative position of the request in the target buffer. For example... Figure 4As shown, after the delay allocation is completed, the GPU-side command processing unit performs address backfilling operations on all pending requests based on the returned base address.
[0063] Specifically, the offset in each request is combined with the base address to generate the final physical access address, and the request structure is updated. Subsequently, the requests that have completed address backfilling are put back into the execution queue, and the DMA engine performs the data copy operation, thereby ensuring the correctness of data access under the delayed allocation mechanism.
[0064] To ensure the correctness of the entire delay allocation process, this implementation maintains a command state table (MemCopyCmdState Table) on the GPU side for command association and state control mechanisms. This table is indexed by the copy command identifier, and each record contains the current state of the command and its corresponding base address information. The command state includes at least the untriggered state (UR), the triggered allocation state (TR), and the allocated state (AL).
[0065] When an access request arrives for the first time, if the status is "not triggered," the status is updated to "triggered," and an allocation trigger request is sent. If the status is "triggered," the request remains suspended. If the status is "allocation completed," address backfilling is performed directly, and the data transmission phase begins. This state machine mechanism enables unified control in the event of multiple concurrent requests, avoiding repeated triggering of allocation operations while ensuring the orderly execution of requests.
[0066] Regarding the experimental setup and results: (1) Environment: In this embodiment, the above-mentioned access-aware latency memory allocation mechanism can be implemented and verified based on the open-source GPU simulation platform (Multi-GPU Simulator, MGPUSim). The simulation architecture refers to the existing third-generation graphics core architecture (Advanced Micro Devices Graphics Core Next 3, AMD GCN3) of semiconductor companies and is benchmarked against typical discrete GPU device configurations. The simulation system contains 64 computing units (CUs), equipped with a multi-level cache hierarchy and high-bandwidth memory (HBM) device memory, with a page size set to 4KB, and data transfer between the CPU and GPU is realized through PCIe interconnect. At the same time, driver-level memory management logic is extended on the CPU side, and the command processing unit and state control module are modified collaboratively on the GPU side to support the complete execution of the latency allocation and access triggering mechanism.
[0067] (2) Benchmarks: In this embodiment, several representative GPU computing workloads are selected as test benchmarks, including typical applications such as dense linear algebra computation, sparse matrix operations, deep neural network inference and training operators, and graph computation. These benchmarks cover various workload types, including Advanced Encryption Standard (AES), matrix multiplication and its transpose multiplication (ATAX), Fast Fourier Transform (FFT), Sparse Matrix-Vector Multiplication (SPMV), PageRank, Rectified Linear Unit (ReLU), and Generative Pre-trained Transformer (GPT). These benchmark programs cover different data access modes, including continuous access, random access, and hash-aggregated access, and can comprehensively reflect the applicability and stability of the latency allocation mechanism under different memory behavior scenarios.
[0068] (3) Performance: Under the experimental environment and baseline conditions described above, compared with the traditional original allocation mechanism and the optimized allocation-copy pairing mechanism, the access-aware latency allocation mechanism proposed in this invention shows a significant advantage in memory utilization efficiency. Specifically, the average memory unutilized time is reduced by approximately 99.67%, significantly eliminating the idle residency phase between allocation and access; in complex multi-task execution scenarios, the peak memory usage is reduced by an average of approximately 53.53%, effectively alleviating device memory pressure; at the same time, in terms of overall execution time, only a slight overhead of about 2.5% is introduced, indicating that the mechanism improves memory efficiency without having a significant negative impact on system performance. In addition, under the condition of concurrent execution of multiple workloads, the method can still maintain stable performance, verifying its feasibility and scalability in actual high-load environments.
[0069] The access-aware GPU memory latency allocation method disclosed herein constructs a "placement first, allocation later" latency memory management logic. This scheme overturns the traditional "allocation equals occupancy" instant mapping mechanism, solving the GPU memory idle problem by reconstructing the timing relationship between memory allocation and data access. Specifically, when the application layer initiates a memory allocation request, the system only performs a "pseudo-allocation" operation, that is, only records metadata and returns a logical handle, without requesting physical memory; only when the system detects the first actual data access request for that memory region, the access behavior drives the actual physical memory allocation, and then uses an address backfilling mechanism to dynamically update the newly generated physical base address to the access instruction to be executed. This technical solution precisely shifts the actual physical memory occupancy time window to be aligned with the first access time, fundamentally eliminating the allocation-access gap existing in the traditional process. The technical effect achieved is: while ensuring complete transparency of program execution semantics and introducing only a very slight performance overhead, it significantly reduces the memory unutilization time and significantly compresses the peak memory occupation in multi-task concurrent scenarios, thereby greatly improving the GPU memory utilization efficiency and system concurrent processing capabilities.
[0070] Combination Figure 5 As shown in the illustration, this disclosure also provides an access-aware GPU memory latency allocation device 500, including a processor 504 and a memory 501. Optionally, the system may further include a communication interface 502 and a bus 503. The processor 504, communication interface 502, and memory 501 can communicate with each other via the bus 503. The communication interface 502 can be used for information transmission. The processor 504 can invoke logical instructions in the memory 501 to execute the access-aware GPU memory latency allocation method described in the above embodiment.
[0071] Furthermore, the logic instructions in the aforementioned memory 501 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.
[0072] The memory 501, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 504 executes functional applications and data processing by running the program instructions / modules stored in the memory 501, thereby implementing the access-aware GPU memory latency allocation method in the above embodiments.
[0073] The memory 501 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 501 may include high-speed random access memory and may also include non-volatile memory.
[0074] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured as an access-aware GPU memory latency allocation method.
[0075] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.
[0076] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.
Claims
1. A GPU memory latency allocation method based on access awareness, characterized in that, The method includes: When a memory allocation request is received from an application, the allocation metadata related to the memory allocation request is recorded, but no actual device memory allocation operation is performed, thereby completing a logical pseudo-allocation, so that the buffer corresponding to the memory allocation request is in a state of logical allocation but physical unallocated. Upon detecting the first data access request for the buffer, an on-demand allocation mechanism is triggered, and the actual device memory allocation operation is performed according to the recorded allocation metadata to allocate physical memory for the buffer. After the actual allocation operation is completed, an address backfilling operation is performed. The allocated physical base address is combined with the logical offset corresponding to the data access request to generate the final physical access address, so as to execute the data access request.
2. The method according to claim 1, characterized in that, The record contains allocation metadata related to the memory allocation request, including: A metadata record is created for each memory allocation request, which includes at least the allocation command identifier, device identifier, process identifier, and requested memory size information, and the metadata record is stored in the memory management structure on the driver side.
3. The method according to claim 1, characterized in that, The step of triggering the on-demand allocation mechanism upon detecting the first data access request for the buffer includes: When the GPU-side command processing unit processes a data access request, it checks the actual allocation status of the requested target buffer. If the status is unallocated, the GPU-side command processing unit suspends the data access request and sends an allocation trigger signal to the CPU-side driver. The CPU-side driver responds to the allocation trigger signal, performs the actual device memory allocation operation according to the corresponding allocation metadata, and returns the allocated base address to the GPU side.
4. The method according to claim 3, characterized in that, The execution address backfilling operation includes: The GPU-side command processing unit receives the base address returned by the CPU-side driver; For each data access request to be executed, the offset representing the relative position is combined with the base address to generate the final physical access address and update the corresponding request structure; Once the address has been filled, the data access request is placed in the execution queue, and the DMA engine performs the data copying.
5. The method according to claim 1, characterized in that, The method also includes assigning a unique identifier to each memory allocation command and data access command, and establishing a mapping relationship between the two; A command status table is maintained on the GPU side. The command status includes at least: a non-triggered status indicating that allocation has not yet been triggered, a triggered allocation status indicating that an allocation request has been sent and is waiting for completion, and a completed allocation status indicating that allocation has been completed and can be directly accessed.
6. The method according to claim 5, characterized in that, The method also includes unified control of data access requests based on the command status table; The unified control of data access requests based on the command status table includes: When a data access request arrives for the first time and the corresponding status is untriggered, the status is updated to triggered allocation status, and the on-demand allocation mechanism is triggered. When other data access requests arrive and the corresponding status is the triggered allocation status, the request remains in the suspended waiting state. When the status is "allocation completed", the address backfilling operation and data transmission are executed directly.
7. The method according to claim 1, characterized in that, The data access request is a data copy request from the host to the device. By delaying the actual allocation of physical memory to align with the data copy request, the idle time gap between memory allocation and the first access in the traditional process is eliminated.
8. The method according to claim 2, characterized in that, During the pseudo-allocation phase, no device memory request is performed, no page table is updated, and no physical page frames are allocated. Only a logical handle or virtual placeholder is returned to the upper-layer application.
9. The method according to claim 1, characterized in that, Prior to detecting the first data access request for the buffer, the method further includes: The data access request is split into multiple sub-requests, and each sub-request is configured with a logical offset of its relative position in the target buffer. These sub-requests are sent to the GPU-side command processing unit, enabling the construction and distribution of data access requests to be completed during the pseudo-allocation phase, without relying on the completed physical address.
10. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.