The application discloses a runtime
memory management optimization method special for
model inference scenes. In the application, through a hierarchical
memory pool pre-allocation and dynamic shared area reuse mechanism, the problem of GPU memory fragmentation in the multi-task concurrent
inference scene is fundamentally solved. Before the task is started, the hot area and the non-hot area are accurately divided according to historical data, and a continuous GPU
memory block is pre-allocated to avoid address
discretization caused by dynamic allocation. In the
inference process, the hot area range and capacity are adjusted in real
time based on
access frequency, so that high-frequency access data always resides in the continuous GPU memory space, and the GPU
memory address translation overhead and
data migration delay are greatly reduced. At the same time, in combination with an arbitration mechanism for dynamically adjusting priorities according to task SLA requirements, it is ensured that high-priority tasks can temporarily occupy the idle memory of low-priority tasks when the GPU memory is tight, so that the real-time requirements of key tasks are guaranteed, the overall utilization of GPU memory resources is maximized, and the waste of resources in the traditional static allocation mode is avoided.