Cross-process memory overcommit management method and system for ascendent device
By deploying a shared library to manage video memory on Ascend devices, the problem of scarce video memory resources was solved, cross-process video memory over-allocation was achieved, and the efficiency of video memory usage and the flexibility of multi-process deployment were improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-05-20
- Publication Date
- 2026-07-21
Smart Images
Figure CN122431897A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of device memory management technology in artificial intelligence inference deployment, and specifically to a cross-process video memory over-allocation management method and system for Ascend devices. Background Technology
[0002] The Ascend computing platform has become a mainstream domestic hardware provider in the field of artificial intelligence due to its outstanding performance in deep learning inference, operator acceleration, and multi-model hybrid deployment. In conventional development models, business logic typically calls the Ascend computing language runtime interface directly, implementing computational tasks through standard processes such as memory allocation, model loading, data organization, and execution submission. However, with the rapid iteration of model technologies, the demand for memory from inference tasks has exploded, and traditional memory management methods are increasingly showing their limitations: the size of the model workspace and input / output buffers increases exponentially with the increase in model complexity. When the limited physical memory capacity of a single GPU cannot support the inference load of multiple models, the system is often forced to abort tasks due to allocation failures.
[0003] To address the shortage of video memory resources, academia has proposed a system-level unified virtual memory management approach. This scheme uses unified virtual address space technology to achieve data migration between the host and the device, thereby providing oversubscription of device memory. However, this scheme has stringent technical prerequisites, requiring the underlying hardware to have advanced page fault interrupt response and page fault instruction replay capabilities, and relying on drivers and operating systems to provide complete page table management, page migration, and other supporting support. Therefore, it can only be applied on certain specific hardware and systems and lacks universality.
[0004] In summary, how to design a memory over-allocation scheme to solve the problem of memory shortage in multi-model, multi-process hybrid deployment scenarios while taking into account both versatility and ease of use has become an urgent technical challenge in this field. Summary of the Invention
[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a cross-process video memory over-allocation management method and system for Ascend devices.
[0006] A cross-process video memory over-allocation management method for Ascend devices, provided by the present invention, includes: Step S1: Deploy the shared library between the application and the underlying ACL runtime to take over the relevant interface calls; Step S2: When the application requests device memory, it does not occupy the device's physical video memory, reserves the device's virtual address space, and returns the device's virtual address as a stable logical device address to the application. At the same time, it creates managed memory block metadata. Step S3: Construct the set of participating addresses for the current call, and fix the managed memory blocks in the set of participating addresses as objects that cannot be swapped out during the current call cycle; Step S4: Before calling the underlying ACL raw interface, perform a residency check on each managed memory block in the participating address set: if it is already resided, reuse it directly; if it is not yet resided, apply for physical video memory on the device as needed and establish a mapping; when local video memory is insufficient, prioritize performing local swapping out and recovery within the current process. Step S5: If the physical video memory required for this stay is still insufficient after partial swapping out, then access the cross-process shared state at the device level, request global video memory quota, and initiate a reclamation request to other active processes to complete cross-process video memory coordination. Step S6: After the target managed memory block has completed its residency, translate the address actually used by the underlying runtime into the current valid address; repair the address entries in the high-level object before the call and restore them after the call; and perform mapping release, physical handle release, global quota return and state cleanup when the model is unloaded, the video memory is released or the process exits.
[0007] Preferably, step S1 includes: Deployed as a shared library between the application and the underlying ACL runtime, it takes over the interface calls related to memory allocation, model loading, model execution, operator execution, tensor creation, and tensor address setting by exporting symbols with the same name and cooperating with symbol redirection or interception.
[0008] Preferably, step S3 includes: Before model execution, operator execution, or tensor execution, the participating address set for the current call is constructed based on the model binding relationship, dataset object, tensor object, tensor list object, and executor object, and the managed memory block in the participating address set is fixed as an object that cannot be swapped out during the current call cycle.
[0009] Preferably, a delayed initialization strategy is adopted, in which device allocation granularity detection, local video memory capacity detection, and cross-process global coordination environment initialization are performed only when the current thread is detected to have a valid ACL context; The interfaces that have been taken over include: The video memory management interface is used for device video memory allocation and release, data transfer, and memory initialization. The model interface is used for model loading and inference. Operator interface, used for single operator compilation, invocation, and handle execution; Tensor and executor configuration interface for creating and destroying data structures.
[0010] Preferably, during the device memory allocation phase, if the allocation granularity recommended by the underlying hardware fails to be obtained, it automatically degenerates to the minimum allocation granularity preset by the system; the memory allocation size is aligned upwards according to the determined allocation granularity; for each managed memory block, the core metadata maintained by the system includes the logical virtual address returned to the application layer, the memory size originally requested by the user, the aligned actual block size, the current lifecycle state of the memory block, the mapping handle of the physical video memory on the device side, the pointer of the backup area used for swapping out on the host side, the ledger record marker used for cross-process quota coordination, the fixed reference count used to prevent accidental reclamation during execution, and the set of active execution flows associated with the memory block; the current lifecycle state of the memory block includes the unmapped state of unallocated physical video memory, the resident state of allocated physical video memory on the device side, and the swapped-out state of data temporarily rolled back to the host side.
[0011] Preferably, when partially swapping out a managed memory block, all active execution flows related to that managed memory block are first synchronized. If the host-side backup area has not yet been created, a backup area of the same size as the managed memory block is requested on the host side and aligned to the specified bytes. Then, the device contents of the managed memory block are copied to the host-side backup area, the mapping between the virtual address and the physical video memory is released, the device physical video memory handle is released, and the local ledger and global ledger are updated. The block status is then updated to SWAPPED. During recovery, the device physical video memory handle is re-requested and mapped to the previously reserved virtual address. If a host-side backup area exists, the backup data is backfilled to the device side, and the block status is then updated to RESIDENT. The partial swapping strategy includes the best-fit swapping strategy and the sequential swapping strategy. The best-fit swapping strategy prioritizes the managed memory block whose size is not smaller than the current gap and is closest to the current gap. If no candidate block meets the conditions, the swappable block is selected according to the principle of prioritizing larger blocks.
[0012] Preferably, the cross-process shared state is maintained at the device level and stored in shared memory; the cross-process shared state includes the global used physical video memory value, the global physical video memory capacity limit, the number of active processes, and a process registration table; the process registration table records at least the process identifier pid, the local listening address, and the activity status; the shared state is protected by a process sharing robust lock that supports the recovery of abnormal processes, and old process records that have exited but not properly deregistered are cleaned up before new process registration and global quota application, and the shared state area of the corresponding device is destroyed when the last process exits.
[0013] Preferably, before calling the underlying ACL original interface, check whether the address stored in the data buffer descriptor or executor points to the managed memory block; if an explicit update is required, temporarily rewrite the address in the object to the current valid address, and restore it to the original logical address on the application side after the original interface call is completed.
[0014] A cross-process video memory over-allocation management system for Ascend devices, provided by the present invention, includes: The interface interception module is used to intercept the application's runtime interface calls to the underlying ACL in a shared library manner; The runtime initialization module is used to detect the ACL context, probe the allocation granularity and video memory capacity, and initialize the cross-process coordination environment; The virtual address reservation module is used to reserve the device's virtual address space and return a stable logical device address; The memory block management module is used to create and maintain the metadata, block status, physical handle, host-side backup area, fixed count, and active execution flow for each returned logical device address of the managed memory block; The binding relationship maintenance module is used to maintain the binding relationships between models, tensors, tensor lists, executor objects, and underlying managed addresses; The residency control module is used to ensure that managed memory blocks in the participating address set are in an accessible state at the execution boundary, and to perform residency checks on each memory block; The partial swap-out recovery module performs stream synchronization, host backup, demapping, release of physical video memory, and on-demand recovery on swappable managed memory blocks within the current process when the resident check finds insufficient local video memory. When the physical video memory required for this resident application is still insufficient after a local swap-out, the cross-process coordination module accesses the cross-process shared state at the device level, requests global video memory quota, and initiates a reclamation request to other active processes. The address repair module translates the address actually used at the underlying runtime into the current valid address after the managed memory block is successfully resident, and repairs the address entries in the higher-level object before the call and restores them after the call. The cleanup and recycling module is used to complete mapping removal, quota return, and status cleanup when resources are released or processes exit. The policy control module is used to configure the swap-out policy, capacity control parameters, and prefetch policy.
[0015] Preferably, the application completes access by linking to a shared library; the shared library interacts with the Ascend device through the ACL raw interface; the cross-process coordination module exchanges quota reclamation requests and release results with other processes through shared memory and inter-process communication mechanisms.
[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention adopts a pure user-space implementation method, which does not rely on page fault interrupts, instruction replay and system-level unified virtual memory support, has a low deployment threshold and a wide range of applicability.
[0017] 2. This invention provides a stable logical device address to the application by "first reserving the virtual address, then allocating physical memory as needed, and repeatedly establishing mappings on the original virtual address", so that upper-layer services can access the application without modifying the original ACL interface call logic.
[0018] 3. This invention identifies the real working set before API boundaries such as model execution and operator execution, and combines fixing, resident checks, local swapping out and recovery mechanisms to avoid the waste of video memory caused by coarse-grained static reservation and improve video memory utilization efficiency.
[0019] 4. This invention achieves unified memory coordination among multiple independent processes on the same device through a shared memory ledger and inter-process recycling requests. It can break the process-level memory resource silos and improve memory utilization, capacity elasticity and deployment flexibility in multi-process inference scenarios. Attached Figure Description
[0020] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of the overall process of the method of the present invention.
[0021] Figure 2 This is a flowchart illustrating the residency, swapping out, and recovery of managed memory blocks according to the present invention.
[0022] Figure 3 This is a flowchart illustrating the cross-process memory coordination process of this invention. Detailed Implementation
[0023] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0024] like Figure 1 As shown, this illustrates the relationship between shared library interface takeover, device virtual address reservation, participation address set construction, residency checks, partial swap-out recovery, cross-process coordination, and address patching and resource cleanup. like Figure 2 and Figure 3 As shown, a cross-process video memory over-allocation management method for Ascend devices includes: Step S1: Deploy the shared library between the application and the underlying ACL runtime. By exporting symbols with the same name and cooperating with symbol redirection or interception, take over the interface calls related to memory allocation, model loading, model execution, operator execution, tensor creation, and tensor address setting.
[0025] Step S2: When the application requests device memory, it does not immediately occupy the device's physical video memory. Instead, it reserves the device's virtual address space according to the device's recommended allocation granularity and returns the reserved device virtual address as a stable logical device address to the application. At the same time, it creates managed memory block metadata for this address.
[0026] Step S3: Before model execution, operator execution, or tensor execution, construct the participating address set for the current call based on the model binding relationship, dataset object, tensor object, tensor list object, and executor object, and fix the managed memory blocks in the participating address set as objects that cannot be swapped out during the current call cycle.
[0027] Step S4: Before actually calling the underlying ACL raw interface, perform a residency check on each managed memory block in the participating address set; for managed memory blocks that have already resided, reuse them directly; for managed memory blocks that have not yet resided, request physical video memory from the device as needed and establish a mapping; when local video memory is insufficient, prioritize performing local swapping out and recovery within the current process.
[0028] Step S5: If the physical video memory required for this stay is still insufficient after the partial swap-out is completed in the current process, then the cross-process shared state is accessed at the device level, a global video memory quota is requested, and a reclamation request is initiated to other active processes to complete the cross-process video memory coordination.
[0029] Step S6: After the target managed memory block completes its residency, the address actually used by the underlying runtime is translated into the current valid address; when the managed memory address stored in the higher-level object is inconsistent with the current valid address, the address entries in the higher-level object are repaired before the call and restored after the call; and when the model is unloaded, the video memory is released or the process exits, the mapping is unloaded, the physical handle is released, the global quota is returned and the state is cleaned up.
[0030] The method of this invention employs a delayed initialization strategy, performing device allocation granularity detection, local video memory capacity detection, and cross-process global coordination environment initialization only when a valid ACL context is detected in the current thread. The managed interfaces include at least a video memory management interface for device video memory allocation and release, data transmission, and memory initialization; a model interface for model loading and inference; an operator interface for single operator compilation, invocation, and handle execution; and a tensor and executor configuration interface for data structure creation and destruction.
[0031] During the device memory allocation phase, if the allocation granularity recommended by the underlying hardware fails to be obtained, it automatically degenerates to the minimum allocation granularity preset by the system. The memory allocation size needs to be upward aligned according to the determined allocation granularity. For each managed memory block, the system maintains at least the following core metadata: the logical virtual address returned to the application layer, the original memory size requested by the user, the aligned actual block size, the current lifecycle state of the memory block, the mapping handle of the physical video memory on the device side, the pointer to the backup area on the host side used for swapping out, the ledger record marker for cross-process quota coordination, the fixed reference count to prevent accidental reclamation during execution, and the set of active execution flows associated with the memory block. Among them, the current lifecycle state of the memory block includes at least: the unmapped state of unallocated physical video memory, the resident state of allocated memory on the device side, and the swapped-out state of data temporarily rolled back to the host side.
[0032] The construction of the address set is based on multi-layered object dependencies and binding relationships. Specifically, this includes: the dependency relationship between the model instance and its runtime working area and weights; the mapping relationship between a single tensor object and its underlying physical data; the association relationship between a tensor set object and its internal member tensors; and the reference relationship between the computation executor object and various tensors and input / output address items. The system dynamically extracts and collects the real working set addresses necessary within the current call lifecycle by recursively or by traversing the above binding relationships. This includes at least: the runtime working area and model weight addresses required for model execution; the underlying buffer addresses encapsulated by high-level data containers; the physical memory addresses actually pointed to by each level of tensor structure; the input / output overlay addresses explicitly declared in the executor; and the dynamic working area addresses required by the underlying neural network computation library at runtime.
[0033] When partially swapping out a managed memory block, all active execution flows related to that managed memory block are first synchronized. If the host-side backup area has not yet been created, a backup area of the same size as the managed memory block is requested on the host side and aligned to the specified bytes. Then, the device contents of the managed memory block are copied to the host-side backup area, the mapping between the virtual address and the physical video memory is released, the device physical video memory handle is released, and the local ledger and global ledger are updated. The block status is then updated to SWAPPED. During recovery, the device physical video memory handle is re-requested and mapped to the previously reserved virtual address. If a host-side backup area exists, the backup data is backfilled to the device side, and the block status is then updated to RESIDENT. The partial swap-out strategy includes the best fit swap-out strategy and the sequential swap-out strategy. The best fit swap-out strategy prioritizes the managed memory block whose size is not smaller than the current gap and is closest to the current gap. If no candidate block meets the conditions, the swappable block is selected according to the principle of prioritizing larger blocks.
[0034] Cross-process shared state is maintained at the device level and stored in shared memory. The cross-process shared state includes at least the global used physical video memory value, the global physical video memory capacity limit, the number of active processes, and a process registration table. The process registration table records at least the process identifier (PID), the local listening address, and the activity status. The shared state is protected by a process sharing robust lock that supports abnormal process recovery. Before new process registration and global quota application, old process records that have exited but not properly deregistered are cleaned up. The shared state area of the corresponding device is destroyed when the last process exits.
[0035] Before the underlying ACL original interface is called, check whether the address stored in the data buffer descriptor or executor points to the managed memory block; if an explicit update is required, temporarily rewrite the address in the object to the current valid address, and restore it to the original logical address on the application side after the original interface call is completed.
[0036] A cross-process video memory over-allocation management system for Ascend devices, the system is deployed on a computer system that includes at least a CPU, host memory, Ascend devices and a shared memory region, and is located as a shared library between the application and the underlying ACL runtime.
[0037] The system includes at least an interface interception module, a runtime initialization module, a virtual address reservation module, a memory block management module, a binding relationship maintenance module, a resident control module, a partial swap-out recovery module, a cross-process coordination module, an address patching module, a cleanup and recycling module, and a policy control module.
[0038] in: The interface interception module is used to take over the interfaces related to ACL memory, models, operators, tensors, and executors; The runtime initialization module is used to detect the ACL context, probe the allocation granularity and video memory capacity, and initialize the cross-process coordination environment; The virtual address reservation module is used to reserve the device's virtual address space and return a stable logical device address; The memory block management module is used to maintain the metadata, block status, physical handle, host-side backup area, fixed count, and active execution flow of managed memory blocks; The binding relationship maintenance module is used to maintain the binding relationships between models, tensors, tensor lists, and executor objects and the underlying managed addresses. The residency control module is used to ensure that managed memory blocks in the participating address set are in an accessible state at the execution boundary; The partial swap-out recovery module is used to perform stream synchronization, host backup, demapping, release physical video memory, and on-demand recovery; The cross-process coordination module is used to share the global memory ledger and initiate and respond to peer reclamation requests; The address patching module is used to patch address entries in high-level objects before they are called and restore them after they are called. The cleanup and recycling module is used to complete mapping removal, quota return, and status cleanup when resources are released or processes exit. The policy control module is used to configure the swap-out policy, capacity control parameters, and prefetch policy.
[0039] The application completes access by linking to the shared library, which interacts with the Ascend device through the ACL raw interface; the cross-process coordination module exchanges quota reclamation requests and release results with other processes through shared memory and inter-process communication mechanisms.
[0040] Example 1 This embodiment provides a cross-process video memory over-allocation management method for Ascend devices. The method runs on a computer system that includes at least a CPU, host memory, Ascend devices, and a shared memory region, and is accessed through a shared library to run an existing ACL.
[0041] Step S101: Runtime Takeover and Delayed Initialization. The system exports symbols with the same names as the original ACL interfaces and, in conjunction with symbol redirection or interception, connects the video memory management interface, model interface, operator interface, tensor and executor configuration interface, and related workspace interfaces to the virtual video memory management layer. The shared library can be connected to the business process through preloading or linking replacement. To avoid premature initialization before the ACL context is established, the system only performs device allocation granularity detection, local video memory capacity detection, and cross-process global coordination environment initialization after detecting that the current thread has a valid ACL context; preferably, each process initializes only once for each device and caches runtime parameters such as device number, recommended granularity, capacity limit, shared state name, and listening address; when no valid ACL context is detected, the system directly passes through the original interface call.
[0042] Step S102: Managed memory block creation and registration. When an application calls the device memory request interface, the system first obtains the recommended allocation granularity of the device; if obtaining the recommended allocation granularity fails, it degenerates into the minimum allocation granularity. The system aligns the user's requested size upwards according to the allocation granularity to obtain the aligned block size. Subsequently, it calls the device virtual address reservation interface to reserve the corresponding device virtual address space and directly returns the reserved device virtual address to the application as the logical device address; then, it registers the metadata of the managed memory block in the local memory block table. Thus, the upper-layer application sees a stable logical device address from beginning to end, while the existence and mapping of the underlying physical video memory are dynamically determined by this invention at the execution boundary.
[0043] Step S103: To accurately identify the actual memory range accessed by a single computation task (such as model or operator execution), the system establishes a multi-level binding relationship covering model instances, tensor objects, tensor sets, and executor objects to maintain references to underlying data and address overlays for each object. Before triggering computation execution, the system automatically constructs the set of participating addresses following the hierarchical order of "running workspace and weights, data buffer descriptors, underlying tensor addresses, tensor set member addresses, executor explicit input / output addresses, and computation library dynamic workspace."
[0044] This collection uses efficient data structures for deduplication and merges overlapping logical address ranges to optimize residency checking efficiency. For managed memory blocks within the collection, the system increments their fixed reference count at the start of execution, locking the resource to ensure it is not swapped out locally or reclaimed across processes within the current call cycle; after the corresponding execution flow completes synchronization, the fixed reference count is decremented, thereby restoring resource liquidity. Step S104: Resident check, partial swap-out and recovery. For each managed memory block in the participating address set, the system performs a resident check before actually calling the underlying ACL raw interface: if the managed memory block is already in the RESIDENT state, it is directly reused; if the managed memory block has not yet been mapped to physical video memory, it first determines whether the sum of the local used physical video memory and the space required this time exceeds the local capacity limit.
[0045] The local capacity limit can be set to 90%–98% of the device's physical video memory capacity, or to a configurable fixed number of bytes. To reduce critical jitter, the system can also reserve an additional 1–2 allocation granularities as a safety margin. When the local capacity limit is exceeded, the system selects manageable memory blocks that can be swapped out from local candidate blocks and performs a partial swap. Preferably, the candidate blocks to be swapped out satisfy the condition that the reference count is zero and there are no incomplete active execution flows. In this embodiment, the partial swap strategy can adopt a best-fit swap strategy or a sequential swap strategy. The best-fit swap strategy prioritizes candidate blocks whose size is not smaller than the current gap and are closest to the current gap. If no suitable block is found, the selection continues according to the principle of prioritizing larger blocks. When swapping out a candidate block, the system first synchronizes all active execution flows related to that candidate block to ensure that device-side computation has been completed. If the host-side backup area has not yet been created, a backup area of the same size as the block is requested on the host side, preferably aligned to 64 bytes. Then, the current device content of the block is copied to the host-side backup area, the mapping relationship between virtual addresses and physical memory is released, the device physical memory handle is released, and the local and global ledgers are updated. Finally, the block status is updated to SWAPPED. The target number of bytes for a single partial reclamation is the current gap plus the aforementioned safety margin. When the swapped-out block is accessed again, the system re-requests the device physical memory handle, maps the new physical memory to the previously reserved virtual address, and if a host-side backup area exists, the backup data is copied from the host back to the device side, and the block status is updated to RESIDENT.
[0046] Step S105: Cross-process global video memory coordination. If the physical video memory required for the current residency cannot be met after performing a local swap-out within the current process, the system switches to cross-process coordination. The system maintains the cross-process shared state at the device level, preferably stored in shared memory; the shared state includes at least the global used physical video memory value, the global physical video memory capacity limit, the number of active processes, and a process registry table, wherein the process registry table records at least the PID, local listening address, and activity status. The global capacity limit is measured using byte granularity consistent with the allocation granularity and can be set to 90% to 98% of the device's physical video memory capacity. The shared state is protected using a process-shared robust lock that supports abnormal process recovery. When a requesting process needs to request physical video memory for a managed memory block locally, it first attempts to reserve the required number of bytes in the shared state; if the global used value plus the current request exceeds the global capacity limit, the missing byte count is calculated, and a reclamation request is sent to other active processes according to the process registry table, requesting them to release some swappable blocks locally. Reclamation requests are transmitted via Unix domain sockets, shared memory message queues, or other inter-process communication mechanisms. The reclamation process can be configured with a retry threshold of 1 to 8 times, and the timeout for each round of reclamation can range from 1ms to 100ms. Upon receiving a reclamation request, the peer process executes its swapping logic locally and returns the number of bytes actually released to the requesting process. The requesting process then retryes retaining the global quota based on the peer's reclamation result until successful or the retry threshold is reached. Before registering new processes and requesting global quotas, the system prioritizes cleaning up old process records that have exited but not properly deregistered. When the last process exits, the system destroys the shared state area of the corresponding device.
[0047] Step S106: Address Translation, Call Execution, and Resource Cleanup. For ordinary managed pointers, as long as their corresponding managed memory block resides, the logical device address can be directly passed to the underlying runtime. For address entries stored in data buffer descriptors or executors, before calling the underlying ACL original interface, the system checks whether the address stored inside the object points to the managed memory block; if an explicit update is required, the address in the object is temporarily rewritten to the current valid address, and the correspondence between "original logical address - current valid address - associated object" is recorded in the address repair table of this call. After the original interface call is completed, the address is restored to the original logical address on the application side in reverse order, thereby ensuring that the address remains stable from the application's perspective and preventing higher-level objects from holding invalid underlying addresses for a long time. After an execution commit is completed, the system records the execution flow participating in the collection association in activeStreams, and reduces the reference count of the corresponding managed memory block after the execution flow completes the event trigger or explicit synchronization. When a model is unloaded, video memory is released, or a process exits, the system unmaps the data, releases the physical handle, returns the global quota, and cleans up the cross-process state. In case of abnormal exit, the system can also perform device synchronization or wait for a configurable recycling timeout before performing forced cleanup.
[0048] Example 2 This embodiment provides a cross-process video memory over-allocation management system for Ascend devices. The system can be implemented by executing the process steps of the method described in Embodiment 1, and is preferably deployed in a computer that includes at least a CPU, host memory, Ascend devices, and a shared memory region. The system includes an interface interception module, a runtime initialization module, a virtual address reservation module, a memory block management module, a binding relationship maintenance module, a resident control module, a partial swap-out recovery module, a cross-process coordination module, an address patching module, a cleanup and recycling module, and a policy control module.
[0049] The interface interception module is used to take over the interfaces related to ACL memory, models, operators, tensors, and executors; the runtime initialization module is used to complete allocation granularity detection, memory capacity detection, and cross-process coordination environment initialization after a valid ACL context is established; the virtual address reservation module is used to reserve the device's virtual address space and return a stable logical device address when an application requests device memory; the memory block management module is used to maintain the metadata, block status, and host-side backup area of managed memory blocks; the binding relationship maintenance module is used to maintain the relationship between models, tensors, tensor lists, executor objects, and underlying managed addresses; and the resident control module is used to ensure that managed memory blocks in the participating address set are within the execution boundary. Before actual execution, it is in an accessible state; the partial swap-out recovery module is used to perform stream synchronization, host backup, demapping, release physical video memory, and on-demand recovery; the cross-process coordination module is used to coordinate video memory quotas between multiple processes based on the shared memory ledger and inter-process reclamation request mechanism; the address patching module is used to temporarily patch and restore address entries in high-level objects before and after the underlying ACL raw interface call; the cleanup and reclamation module is used to complete mapping unmapping and state cleanup when resources are released or processes exit; the policy control module is used to configure runtime parameters such as allocation granularity, local capacity limit, global capacity limit, reclamation safety margin, reclamation timeout, retry threshold, swap-out policy, and prefetch window.
[0050] The application accesses the system by linking to the shared library, which calls the underlying runtime and drives the Ascend device through the ACL raw interface; the cross-process coordination module interacts with other active processes through shared memory and inter-process communication mechanisms to exchange recycling requests and release results.
[0051] In addition to being implemented purely in software, the above modules can also be implemented in hardware, such as logic circuits, application-specific integrated circuits, programmable logic devices, or embedded controllers, to achieve the same functionality. The message granularity of the inter-process communication mechanism is consistent with the allocation granularity, and the strategy control module supports configuring operating parameters by device, by process, or by model instance.
[0052] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0053] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A cross-process video memory over-allocation management method for Ascend devices, characterized in that, include: Step S1: Deploy the shared library between the application and the underlying ACL runtime to take over the relevant interface calls; Step S2: When the application requests device memory, it does not occupy the device's physical video memory, reserves the device's virtual address space, and returns the device's virtual address as a stable logical device address to the application. At the same time, it creates managed memory block metadata. Step S3: Construct the set of participating addresses for the current call, and fix the managed memory blocks in the set of participating addresses as objects that cannot be swapped out during the current call cycle; Step S4: Before calling the underlying ACL raw interface, perform a residency check on each managed memory block in the participating address set: if it is already resided, reuse it directly; if it is not yet resided, apply for physical video memory on the device as needed and establish a mapping; when local video memory is insufficient, prioritize performing local swapping out and recovery within the current process. Step S5: If the physical video memory required for this stay is still insufficient after partial swapping out, then access the cross-process shared state at the device level, request global video memory quota, and initiate a reclamation request to other active processes to complete cross-process video memory coordination. Step S6: After the target managed memory block has completed its residency, translate the address actually used by the underlying runtime into the current valid address; repair the address entries in the high-level object before the call and restore them after the call; and perform mapping release, physical handle release, global quota return and state cleanup when the model is unloaded, the video memory is released or the process exits.
2. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, Step S1 includes: Deploying the shared library between the application and the underlying ACL runtime allows for the control of interface calls related to memory allocation, model loading, model execution, operator execution, tensor creation, and tensor address setting by exporting symbols with the same name and using symbol redirection or interception.
3. The cross-process video memory over-allocation management method for Ascend devices according to claim 2, characterized in that, Step S3 includes: Before model execution, operator execution, or tensor execution, the participating address set for the current call is constructed based on the model binding relationship, dataset object, tensor object, tensor list object, and executor object, and the managed memory block in the participating address set is fixed as an object that cannot be swapped out during the current call cycle.
4. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, A delayed initialization strategy is adopted, and device allocation granularity detection, local video memory capacity detection, and cross-process global coordination environment initialization are performed only when the current thread is detected to have a valid ACL context. The interfaces that have been taken over include: The video memory management interface is used for device video memory allocation and release, data transfer, and memory initialization. The model interface is used for model loading and inference. Operator interface, used for single operator compilation, invocation, and handle execution; Tensor and executor configuration interface for creating and destroying data structures.
5. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, During the device memory allocation phase, if the allocation granularity recommended by the underlying hardware fails to be obtained, it will automatically degenerate into the minimum allocation granularity preset by the system; the memory allocation size will be aligned upwards according to the determined allocation granularity. For each managed memory block, the core metadata maintained by the system includes the logical virtual address returned to the application layer, the memory size originally requested by the user, the aligned actual block size, the current lifecycle state of the memory block, the mapping handle of the physical video memory on the device side, the pointer to the backup area on the host side used for swapping out, the ledger record marker for cross-process quota coordination, the fixed reference count to prevent accidental reclamation during execution, and the set of active execution flows associated with the memory block; the current lifecycle state of the memory block includes the unmapped state of unallocated physical video memory, the resident state of allocated physical video memory on the device side, and the swapped-out state of data temporarily rolled back to the host side.
6. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, When swapping out a managed memory block, first synchronize all active execution flows related to that managed memory block; if the host-side backup area has not yet been created, then request a backup area of the same size as the managed memory block on the host side and align it to the specified bytes. The device contents of the managed memory block are then copied to the host-side backup area, the mapping between the virtual address and the physical video memory is released, the device physical video memory handle is released, and the local and global ledgers are updated. The block status is then updated to SWAPPED. During recovery, the device physical video memory handle is re-allocated and mapped to the previously reserved virtual address. If a host-side backup area exists, the backup data is backfilled to the device side, and the block status is updated to RESIDENT. The local swap-out strategy includes the best-fit swap-out strategy and the sequential swap-out strategy. The best-fit swap-out strategy prioritizes managed memory blocks whose size is not smaller than the current gap and are closest to the current gap. If no candidate block meets the conditions, the swappable block is selected according to the principle of prioritizing larger blocks.
7. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, Cross-process shared state is maintained at the device level and stored in shared memory; the cross-process shared state includes the global used physical video memory value, the global physical video memory capacity limit, the number of active processes, and a process registration table; the process registration table records at least the process identifier (pid), local listening address, and activity status; the shared state is protected by a process sharing robust lock that supports abnormal process recovery, and old process records that have exited but not properly deregistered are cleaned up before new process registration and global quota application, and the shared state area of the corresponding device is destroyed when the last process exits.
8. The cross-process video memory over-allocation management method for Ascend devices according to claim 1, characterized in that, Before the underlying ACL original interface is called, check whether the address stored in the data buffer descriptor or executor points to the managed memory block; if an explicit update is required, temporarily rewrite the address in the object to the current valid address, and restore it to the original logical address on the application side after the original interface call is completed.
9. A cross-process video memory over-allocation management system for Ascend devices, characterized in that, include: The interface interception module is used to intercept the application's runtime interface calls to the underlying ACL in a shared library manner; The runtime initialization module is used to detect the ACL context, probe the allocation granularity and video memory capacity, and initialize the cross-process coordination environment; The virtual address reservation module is used to reserve the device's virtual address space and return a stable logical device address; The memory block management module is used to create and maintain the metadata, block status, physical handle, host-side backup area, fixed count, and active execution flow for each returned logical device address of the managed memory block; The binding relationship maintenance module is used to maintain the binding relationships between models, tensors, tensor lists, executor objects, and underlying managed addresses; The residency control module is used to ensure that managed memory blocks in the participating address set are in an accessible state at the execution boundary, and to perform residency checks on each memory block; The partial swap-out recovery module performs stream synchronization, host backup, demapping, release of physical video memory, and on-demand recovery on swappable managed memory blocks within the current process when the resident check finds that local video memory is insufficient. When the physical video memory required for this resident application is still insufficient after a local swap-out, the cross-process coordination module accesses the cross-process shared state at the device level, requests global video memory quota, and initiates a reclamation request to other active processes. The address repair module translates the address actually used at the underlying runtime into the current valid address after the managed memory block is successfully resident, and repairs the address entries in the higher-level object before the call and restores them after the call. The cleanup and recycling module is used to complete mapping removal, quota return, and status cleanup when resources are released or processes exit. The policy control module is used to configure the swap-out policy, capacity control parameters, and prefetch policy.
10. The cross-process video memory over-allocation management system for Ascend devices according to claim 9, characterized in that, The application completes access by linking to the shared library; the shared library interacts with the Ascend device through the ACL raw interface; the cross-process coordination module exchanges quota reclamation requests and release results with other processes through shared memory and inter-process communication mechanisms.