Video memory management method and device, and electronic equipment

Abstract

The application provides a display memory management method and device and electronic equipment, and relates to the technical field of computers.The method comprises the following steps: in response to a display memory allocation request of an application program, selecting a target display memory resource from a display memory resource pool based on a hook function; allocating the target display memory resource to the application program and returning a virtual address of the target display memory resource to the application program; the display memory resource pool comprises a local display memory layer, a first display memory layer and a second display memory layer, the local display memory layer comprises local display memory resources of a local device, the first display memory layer comprises first display memory resources of a same-machine device, the second display memory layer comprises second display memory resources of a device used by a remote node, and the target display memory resource is determined from the local display memory resources, the first display memory resources and the second display memory resources based on a layered resource search strategy by the hook function.Through the above method, cross-device and cross-node display memory resource sharing is achieved, the expansibility of the display memory resources is improved, and the display memory management effect is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a display memory management method and device and electronic equipment. BACKGROUND

[0002] In large-scale parallel computing scenarios such as deep learning and scientific computing, especially in memory-intensive application scenarios such as trillion-parameter-level large model training and large-scale molecular dynamics simulation, the allocation and management of device memory resources are crucial.

[0003] Currently, the management of device memory resources mainly relies on unified virtual memory (UVM) technology, memory pooling management technology, and data parallelism and model parallelism technology.

[0004] The core of the memory pooling management technology is to pre-construct a device memory resource pool, which includes multiple memory blocks of the device. When an application needs to use memory resources, a memory allocation request can be initiated, and the memory management system can determine and allocate an idle memory block from the device memory resource pool and mark the allocated memory block as active (Active) state. This memory management method is suitable for scenarios using a single device, and it is difficult to break through the physical limitations of single-card memory, and the resource scalability is limited.

[0005] Since memory resources are usually tight, UVM technology can be introduced, which allows an application to use virtual memory space that exceeds the physical memory capacity. When the memory resources are insufficient, the host memory block can be moved into the device memory resource pool as a supplement, and both the host memory block and the memory block use virtual addresses uniformly. The memory management system can locate the real physical address of the host memory block or the memory block through address mapping based on the virtual address of the host memory block or the memory block, so that the application can use it. This memory management method also cannot break through the physical limitations of single-card memory, and it needs to use host memory. Since the bandwidth of the host memory is much lower than that of the device memory, this method will also affect the running performance of the application, resulting in increased latency.

[0006] Based on this, in some related technologies, data parallelism and model parallelism technology is selected. The core of this technology is to set up multiple devices and distribute data or models to multiple devices, thereby bypassing the physical limitations of single-card memory. However, this memory management method requires a large number of modifications to the application code, has high development costs, poor compatibility, and limited scalability.

[0007] In summary, the existing memory management methods are limited by the physical limitations of single-card memory and development costs, resulting in limited resource scalability and difficulty in meeting the business needs of memory-intensive application scenarios. SUMMARY

[0008] The application provides a display memory management method, device and electronic equipment, to solve the defects that the existing display memory management method is limited by the physical limitation of single-card display memory and the development cost, resulting in limited resource expansion and difficulty in meeting the business requirements of display memory intensive application scenarios.

[0009] The application provides a display memory management method, comprising: in response to a display memory allocation request of an application program, selecting a target display memory resource from a display memory resource pool based on a hook function; allocating the target display memory resource to the application program and returning a virtual address of the target display memory resource to the application program; wherein the display memory resource pool comprises a local display memory layer, a first display memory layer and a second display memory layer, the local display memory layer comprises local display memory resources of a local device, the first display memory layer comprises first display memory resources of a same-machine device, and the second display memory layer comprises second display memory resources of a device used by a remote node, the local device is a device currently used by a server, the same-machine device is a device deployed on the same server as the local device, and the target display memory resource is determined from the local display memory resources, the first display memory resources and the second display memory resources based on a layered resource search strategy by the hook function.

[0010] According to the display memory management method provided by the application, the target display memory resource comprises a target display memory block, the second display memory layer comprises a plurality of remote nodes, the second display memory resources of each remote node comprise at least one second display memory block, and the selection of the target display memory resource from the display memory resource pool based on the hook function comprises: detecting whether the local display memory resources are sufficient based on the hook function; if the local display memory resources are not sufficient, detecting whether the first display memory resources are sufficient; if the first display memory resources are not sufficient, detecting whether the second display memory resources are sufficient; if the second display memory resources are sufficient, selecting a target remote node from all the remote nodes; and selecting a second display memory block meeting the display memory allocation request from the target remote node as the target display memory block.

[0011] According to the display memory management method provided by the application, the target remote node is determined based on the following steps: performing node screening on all the remote nodes to obtain an available node set; the available node set comprises at least two candidate nodes meeting the display memory allocation request; respectively determining a comprehensive score of each candidate node based on node information of each candidate node; the node information comprises network delay, bandwidth, node available display memory proportion and load; determining the target remote node based on each comprehensive score; and the target remote node is the candidate node with the highest comprehensive score.

[0012] The application provides a display memory management method, the display memory resource pool further comprises a system memory layer, the system memory layer comprises a plurality of memory blocks of a local host, and the local host is a host currently used by a server; before detecting whether the first display memory resource is sufficient, the method further comprises: performing network health state detection according to a first preset period; if the heartbeat time of each remote node is greater than a preset time threshold, or the network connection of all remote nodes is interrupted, or the second display memory resource has been exhausted, it is determined that the network health state is a fault state, and a memory block meeting a display memory allocation request is selected from the local host as a target display memory block.

[0013] The application provides a display memory management method, the target display memory resource comprises a target display memory block, the first display memory layer comprises a plurality of same machine devices, and the first display memory resource of each same machine device comprises at least one first display memory block; based on the hook function, the target display memory resource is selected from the display memory resource pool, comprising: based on the hook function, detecting whether the local display memory resource is sufficient; if the local display memory resource is insufficient, detecting whether the first display memory resource is sufficient; if the first display memory resource is sufficient, selecting a first display memory block meeting the display memory allocation request from the first display memory layer as the target display memory block.

[0014] The application provides a display memory management method, the target display memory resource comprises a target display memory block, the local display memory layer comprises at least one local display memory block; based on the hook function, the target display memory resource is selected from the display memory resource pool, comprising: based on the hook function, detecting whether the local display memory resource is sufficient; if the local display memory resource is sufficient, selecting a local display memory block meeting the display memory allocation request from the local display memory layer as the target display memory block.

[0015] The application provides a display memory management method, the second display memory resource of the remote node comprises a second display memory block; after the target display memory resource is allocated to the application program and the virtual address of the target display memory resource is returned to the application program, the method further comprises: in response to a display memory access request of the application program, based on the hook function, detecting whether a to-be-accessed display memory block required to be accessed by the application program is a second display memory block; if the to-be-accessed display memory block required to be accessed by the application program is a second display memory block, detecting whether the to-be-accessed display memory block can be hit in a local buffer area; the local buffer area comprises a plurality of cache display memory blocks; if the to-be-accessed display memory block cannot be hit in the local buffer area, and the current available cache capacity of the local buffer area is insufficient, based on the LRU algorithm, releasing at least one cache display memory block, and writing the data of the second display memory block of the remote node into the local buffer area to generate a to-be-accessed cache display memory block; determining a local cache address of the to-be-accessed cache display memory block in the local buffer area; and redirecting the display memory access request to the local cache address.

[0016] According to the video memory management method provided by the application, after responding to the video memory access request of the application program, the method further comprises: obtaining historical access records according to a second preset period; the historical access records comprise access information of each cache video memory block in a preset time period; performing time locality analysis on the historical access records to determine a first candidate set; the first candidate set comprises at least one first candidate video memory block; performing space locality analysis on the historical access records to determine a second candidate set; the second candidate set comprises at least one second candidate video memory block; performing access mode analysis on the historical access records to determine a third candidate set; the third candidate set comprises at least one third candidate video memory block; selecting a plurality of target candidate video memory blocks from the first candidate set, the second candidate set and the third candidate set, and writing data of each target candidate video memory block into a backup buffer.

[0017] The application further provides a video memory management device, comprising: a resource selection module, configured to select a target video memory resource from a video memory resource pool based on a hook function in response to a video memory allocation request of an application program; and a resource allocation module, configured to allocate the target video memory resource to the application program and return a virtual address of the target video memory resource to the application program; wherein the video memory resource pool comprises a local video memory layer, a first video memory layer and a second video memory layer, the local video memory layer comprises local video memory resources of a local device, the first video memory layer comprises first video memory resources of a same-machine device, and the second video memory layer comprises second video memory resources of a device used by a remote node, the local device is a device currently used by a server, the same-machine device is a device deployed on the same server as the local device, and the target video memory resource is determined by the hook function based on a layered resource search strategy from the local video memory resources, the first video memory resources and the second video memory resources.

[0018] The application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements any of the above video memory management methods when executing the computer program.

[0019] The GPU memory management method, device and electronic equipment provided by the application introduce a multi-level GPU memory resource pool architecture on the basis of the prior art, the GPU memory resource pool is a three-level physical GPU memory pool, which includes a local GPU memory layer, a first GPU memory layer and a second GPU memory layer, the local GPU memory layer includes local GPU memory resources of a local device, the first GPU memory layer includes first GPU memory resources of a same-machine device, and the second GPU memory layer includes second GPU memory resources of a device used by a remote node, the local device is a device currently used by a server, and the same-machine device is a device deployed on the same server as the local device, unlike a traditional device GPU memory resource pool, the first GPU memory layer and the second GPU memory layer are arranged to break through the physical limitation of a single card GPU memory, realize GPU memory resource sharing across devices and nodes, and improve the expansibility of GPU memory resources, when an application program initiates a GPU memory allocation request, a hook function can be used to select target GPU memory resources meeting the GPU memory allocation request from the GPU memory resource pool according to a hierarchical resource search strategy, the selection range of the GPU memory resources is widened, the problem of insufficient GPU memory resources can be effectively solved, the application program code does not need to be modified, the use cost is low, and the business demand of a GPU memory intensive application scenario can be met, meanwhile, the advantages of the UVM technology are fully utilized, after the target GPU memory resources are allocated to the application program, the virtual address of the target GPU memory resources is returned to the application program, which is beneficial to subsequent positioning of the real physical address of the target GPU memory resources through address mapping based on the virtual address, so that the application program can be used, the problem of physical address management confusion that may occur in the multi-level GPU memory resource pool architecture can be avoided, and the effect of further optimizing GPU memory management is beneficial. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0021] Figure 1 is one of the flowcharts of the GPU memory management method provided by the application.

[0022] Figure 2 is a structural schematic diagram of the multi-level GPU memory intelligent extension and management system provided by the application.

[0023] Figure 3 is the second flowchart of the GPU memory management method provided by the application.

[0024] Figure 4 is the third flowchart of the GPU memory management method provided by the application.

[0025] Figure 5Figure 1 is a schematic diagram of a screening process of a target remote node according to the present application.

[0026] Figure 6 Figure 2 is another schematic diagram of a screening process of a target remote node according to the present application.

[0027] Figure 7 Figure 3 is a schematic diagram of a monitoring agent mechanism according to the present application.

[0028] Figure 8 Figure 4 is a schematic diagram of a network health state detection process according to the present application.

[0029] Figure 9 Figure 5 is a schematic diagram of a structure of a video memory block management module according to the present application.

[0030] Figure 10 Figure 6 is a schematic diagram of a defragmentation mechanism according to the present application.

[0031] Figure 11 Figure 7 is a schematic diagram of a video memory access process according to the present application.

[0032] Figure 12 Figure 8 is a schematic diagram of a working process of an RDMA transmission engine according to the present application.

[0033] Figure 13 Figure 9 is a schematic diagram of a first intelligent prefetch mechanism according to the present application.

[0034] Figure 14 Figure 10 is a schematic diagram of a second intelligent prefetch mechanism according to the present application.

[0035] Figure 15 Figure 11 is a schematic diagram of a batch merging mechanism according to the present application.

[0036] Figure 16 Figure 12 is a schematic diagram of a double buffering mechanism according to the present application.

[0037] Figure 17 Figure 13 is a schematic diagram of a Hook library initialization and address virtualization process according to the present application.

[0038] Figure 18 Figure 14 is a schematic diagram of an address virtual mapping process according to the present application.

[0039] Figure 19 Figure 15 is a schematic diagram of a structure of a video memory management device according to the present application.

[0040] Figure 20 Figure 16 is a schematic diagram of a structure of an electronic device according to the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0042] Please refer to Figures 1 to 18 , Figure 1 is one of flowcharts of the video memory management method provided by the present application, Figure 2 is a structural schematic diagram of the multi-level GPU video memory intelligent extension and management system provided by the present application, Figure 3 is another flowchart of the video memory management method provided by the present application, Figure 4 is a third flowchart of the video memory management method provided by the present application, Figure 5 is one of flowcharts of the target remote node screening provided by the present application, Figure 6 is another flowchart of the target remote node screening provided by the present application, Figure 7 is a principle schematic diagram of the monitoring proxy mechanism provided by the present application, Figure 8 is a flowchart of the network health state detection provided by the present application, Figure 9 is a structural schematic diagram of the video memory block management module provided by the present application, Figure 10 is a principle schematic diagram of the fragment consolidation mechanism provided by the present application, Figure 11 is a flowchart of the video memory access provided by the present application, Figure 12 is a working flowchart of the RDMA transmission engine provided by the present application, Figure 13 is one of flowcharts of the intelligent prefetching mechanism provided by the present application, Figure 14 is another flowchart of the intelligent prefetching mechanism provided by the present application, Figure 15 is a flowchart of the batch merging mechanism provided by the present application, Figure 16 is a flowchart of the double buffering mechanism provided by the present application, Figure 17 is a flowchart of the Hook library initialization and address virtualization provided by the present application, Figure 18 is a flowchart of the address virtual mapping provided by the present application.

[0043] In the embodiment, the display memory management method is applied to a multi-level GPU (Graphics Processing Unit) display memory intelligent expansion and management system based on GPU Direct RDMA (GPU Direct Remote Direct Memory Access) technology (hereinafter referred to as "the system"). The system realizes dynamic expansion and intelligent scheduling of device display memory capacity through hierarchical modular design and transparent Hook mechanism without modifying the application program code.

[0044] The GPU Direct RDMA technology is a zero-copy data transmission technology that allows the RDMA (Remote Direct Memory Access) network card to bypass the CPU (Central Processing Unit) and system memory and directly read and write GPU display memory, realizing end-to-end direct transmission between GPU display memories, and the end-to-end delay can be reduced to microseconds.

[0045] Specifically, as shown in Figure 2 The system adopts a four-layer vertical design, which includes an application layer, a transparent interception layer based on a transparent Hook mechanism (also referred to as a "driver Hook layer"), a core management layer, and a resource layer. The transparent interception layer and the core management layer are used to realize intelligent scheduling, and the resource layer is constructed with a cross-node display memory resource pool.

[0046] The application program running in the application layer does not need to be modified in any way, and can directly call the standard API (Application Programming Interface) to realize display memory resource allocation. This zero-intrusion design is one of the cores of the system, and the application program can realize display memory resource allocation without sensing the physical location change of the underlying display memory resource, and can reduce the development cost without learning new programming interfaces.

[0047] The transparent interception layer is the core of realizing the transparency of the system, which can accurately intercept memory allocation functions, data transmission functions and memory release functions through the Linux dynamic linker preloading mechanism, and provide intervention points for subsequent expansion logic. The transparent interception layer can intercept memory allocation functions, data transmission functions and memory release functions through the LD_PRELOAD dynamic link hijacking technology when the application program is running. It can not only forward the request initiated by the application program, but also build a continuous virtual address space for the upper application program, and maintain the mapping relationship between the virtual address and the physical address of the display memory resource. The actual display memory resource may be distributed in the local device, the same machine device, the remote node device, or even the system memory.

[0048] It should be noted that the memory allocation function, the data transmission function and the memory release function are three memory operation functions: the memory allocation function is used to allocate a specified size of memory block on the device; the data transmission function is used to realize data copying between the host (CPU) and the device (GPU) or between devices; and the memory release function is used to release the memory resource allocated by the memory allocation function. When the application program initiates a memory allocation request, the memory allocation function in the standard API can be directly called to apply for the memory resource satisfying the memory allocation request from the system; when the application program initiates a memory release request, the memory release function in the standard API can be directly called to release the memory resource of the system; and when the application program initiates a memory access request, the data transmission function in the standard API can be directly called to obtain the data of the corresponding memory block.

[0049] The core management layer includes three intelligent scheduling engines, namely, a memory block management module, a state monitoring and coordination module and a degradation decision engine. In addition, the core management layer also includes a local memory exchange space. The core management layer is mainly used to abstract all memory resources into a unified logical pool to realize global perspective optimization scheduling.

[0050] The memory block management module is used to maintain a global memory state table and uses an optimistic locking protocol to process concurrent requests of application programs.

[0051] The local memory exchange space can be used as a cache layer of data of remote nodes. The cache pool is managed by an LRU (Least Recently Used) algorithm. The capacity of the cache pool can be adjusted according to actual needs. For example, the capacity of the cache pool is 1 GB (Gigabyte).

[0052] The state monitoring and coordination module can maintain millisecond-level state synchronization with all remote nodes based on gRPC (a high-performance RPC framework) bidirectional flow and receive state feedback data of the resource layer (i.e., only the changed fields or data are transmitted when the state monitoring and coordination module receives state data from the memory layer and the memory layer, without full data transmission) by using an incremental update strategy, so as to minimize the communication overhead of cross-node.

[0053] The degradation decision engine is used to complete the switching of the resource layer according to the network health condition of the system within a millisecond time period.

[0054] The resource layer adopts a multi-level GPU resource pool architecture, and provides four levels of physical storage resources, which constructs three levels of GPU layers and one level of memory layer. The three levels of GPU layers include a local GPU layer (Level 0), a first GPU layer (Level 1), and a second GPU layer (Level 2), and the one level of memory layer includes a system memory layer (Level 3). The system constructs a unified virtual address space for the resource layer, and the local GPU layer, the first GPU layer, the second GPU layer, and the system memory layer all use virtual addresses uniformly.

[0055] The local GPU layer includes local GPU resources of a local device. The local device is a device currently used by the server, has low delay and high bandwidth, and has a delay of microseconds and a bandwidth of 900 GB / s. It is the preferred data residence for all application computing tasks.

[0056] The first GPU layer includes first GPU resources of a same-machine device. The same-machine device is a device deployed on the same server as the local device. The same-machine device and the local device can be connected and communicated through PCIe P2P technology, and have good delay and bandwidth performance, and are suitable for on-machine expansion of device GPU resources.

[0057] The PCIe P2P (Point to Point) technology is a point-to-point communication technology using a general open standard (based on the PCIe bus), which allows PCIe devices (such as GPUs) to exchange data directly without going through the CPU.

[0058] The second GPU layer includes second GPU resources of a device used by a remote node. The device used by the remote node can be connected to the system through an RDMA network. Although the delay is relatively long, the bandwidth is high, and the resource layer can provide large-scale cross-node GPU resource expansion capability.

[0059] The system memory layer includes memory resources of a local host. The local host is a host currently used by the server. The system memory layer can lock the memory by allocating memory pages of the local host through a host memory allocation function, and can be used as a degraded backup resource in case of network failure.

[0060] When allocating GPU resources for an application, if the GPU resources of the three levels of GPU layers are exhausted or a network failure occurs, the system can automatically degrade to call the system memory layer to ensure the continuous running of the application. The core innovation of the multi-level GPU resource pool architecture is to extend the device GPU management from a single-machine driver level to a distributed coordination level, and to realize the global perspective scheduling of GPU resources.

[0061] The core principle of the system is to realize the transparent video memory resource expansion service for the upper application program by constructing a three-level video memory hierarchy including a local video memory layer, a first video memory layer and a second video memory layer, combining the GPU Direct RDMA technology and the Hook interception mechanism of the transparent interception layer, and automatically degrading to the system memory layer when the network fails, thereby guaranteeing the continuity of the application program running.

[0062] As shown in Figure 1 The video memory management method includes steps S110 to S120, and each step is as follows:

[0063] S110: In response to the video memory allocation request of the application program, the target video memory resource is selected from the video memory resource pool based on the hook function.

[0064] S120: The target video memory resource is allocated to the application program, and the virtual address of the target video memory resource is returned to the application program.

[0065] The video memory resource pool includes a local video memory layer, a first video memory layer and a second video memory layer, the local video memory layer includes local video memory resources of a local device, the first video memory layer includes first video memory resources of a same-machine device, and the second video memory layer includes second video memory resources of a device used by a remote node, the local device is a device currently used by the server, the same-machine device is a device deployed on the same server as the local device, and the target video memory resource is determined by the hook function based on a layered resource search strategy from the local video memory resources, the first video memory resources and the second video memory resources.

[0066] It should be noted that the device in the present application includes an artificial intelligence chip, which includes but is not limited to GPU, GPGPU (General-Purpose Graphics Processing Unit, general-purpose graphics processor), TPU (Tensor Processing Unit, tensor processor) and the like; the host usually refers to the CPU.

[0067] For the server, since the server architecture usually adopts the CPU-GPU architecture, for the convenience of understanding, in the following, the device is taken as an example of GPU, and the host is taken as an example of CPU. For example, the local device refers to the local GPU, the same-machine device refers to the same-machine GPU, the device used by the remote node refers to the GPU used by the remote node, and the local host refers to the local CPU.

[0068] The video memory management method of the embodiment can be roughly divided into six process steps, which are initialization, local evaluation, layered resource search, network health decision, intelligent degradation and asynchronous prefetching.

[0069] Firstly, in the initialization process, the system needs to load the Hook library (the Hook library loading time is less than 100 milliseconds) when the application starts, establish a resource directory, set the heartbeat period (for example, 2 seconds) and the delay threshold (for example, the delay threshold is 15 seconds), and the like.

[0070] Further, as shown in Figure 3 and Figure 4 , when the application program initiates a display memory allocation request, the memory allocation function in the standard API can be directly called to apply to the system for display memory resources that meet the display memory allocation request; at this time, the transparent interception layer based on the transparent Hook mechanism responds to the display memory allocation request of the application program, can intercept the memory allocation function of the standard API based on the hook function (i.e. the Hook function), and through the hierarchical resource search strategy and the network health state perception mechanism, the automatic display memory allocation and degradation decision is made to realize the transparent display memory resource expansion of the application program.

[0071] The display memory resource is usually a display memory block, after the hook function intercepts the memory allocation function of the standard API, the local evaluation process is entered, and the local display memory resource allocation is preferentially tried, so the hook function will first query the local display memory block manager to detect whether the local display memory resource is sufficient, that is, whether there is idle display memory resource in the local display memory layer that meets the display memory allocation request.

[0072] Optionally, the evaluation delay of the local evaluation process should be less than 1 microsecond.

[0073] The local display memory layer includes at least one local display memory block, if the local display memory resource is sufficient, that is, there is idle display memory resource in the local display memory layer that meets the display memory allocation request, then the local display memory block that meets the display memory allocation request can be directly selected from the local display memory layer as the target display memory block.

[0074] Optionally, if the local display memory layer does not have display memory resource that meets the display memory allocation request, it can be considered that the local display memory resource is insufficient.

[0075] Optionally, even if the local display memory layer has display memory resource that meets the display memory allocation request, but the remaining local display memory resource of the local display memory layer is lower than the local alarm threshold (for example, the local alarm threshold is 10% of all display memory resources of the local display memory layer), in order to avoid the local display memory resource being too tense, it can also be considered that the local display memory resource is insufficient.

[0076] If the local display memory resource is insufficient, the transparent interception layer can initiate a remote allocation request to the coordination center in the state monitoring and coordination module.

[0077] It should be noted that for the local GPU, the memory resources of the first memory layer and the memory resources of the second memory layer can be regarded as remote memory resources; wherein the first memory layer includes a plurality of same-machine GPUs, and the first memory resources of each same-machine GPU include at least one first memory block; the second memory layer includes a plurality of remote nodes, and the second memory resources of each remote node include at least one second memory block. The remote allocation request is used to request allocation of memory resources other than the local memory layer.

[0078] At this time, the system enters the network health decision flow, and the coordination center can perform network health state detection according to a first preset period. The network health state detection is the core basis for the degradation decision engine to execute the degradation decision. The coordination center can evaluate the network health state of the system in real time through the heartbeat mechanism (for example, the heartbeat period is 2 seconds) and the RTT (Round-Trip Time, Round-Trip Time) of the data packet. If the remote node continuously times out (for example, 3 times) for a plurality of times, or the round-trip time of the data packet of the remote node exceeds the delay threshold, the degradation decision can be triggered immediately. The key of this process is that the network health state detection and the degradation decision are tightly coupled, which can ensure that the system can complete the switching of the resource layer within 100 milliseconds when the network jitter or the remote node fails, and avoid application program lag or crash.

[0079] Optionally, the detection delay of the network health decision flow should be less than 100 milliseconds.

[0080] Specifically, if the network health state of the system is a healthy state, the remote memory resources will not fail to be allocated due to network failure. At this time, the system will enter the hierarchical resource search flow, and according to the priority of the three-level memory layer defined in the hierarchical resource search strategy, the hook function will preferentially attempt to allocate the same-machine memory resources. It is detected whether the first memory resources are sufficient, that is, whether there is idle memory resource in the first memory layer that satisfies the memory allocation request.

[0081] Optionally, the search duration of the hierarchical resource search flow is usually less than 10 milliseconds. If the search duration of the hierarchical resource search flow is greater than or equal to 50 milliseconds, it is considered that the search is timed out.

[0082] If the first memory resources are sufficient, that is, there is idle memory resource in the first memory layer that satisfies the memory allocation request, a first memory block that satisfies the memory allocation request can be directly selected from the first memory layer as a target memory block.

[0083] If the first memory resources are not sufficient, according to the hierarchical resource search strategy, the hook function will further attempt to allocate remote memory resources, and detect whether the second memory resources are sufficient, that is, whether there is idle memory resource in the second memory layer that satisfies the memory allocation request.

[0084] If the second GPU resource is sufficient, the coordination center can select the optimal target remote node from all remote nodes according to a multi-dimensional scoring mechanism, and select a second GPU block that meets the GPU allocation request from the target remote node as the target GPU block by calling the GPU manager of the target remote node through gRPC bidirectional flow.

[0085] Optionally, the multi-dimensional scoring mechanism needs to refer to the node information of each remote node to score each remote node, and the remote node with the highest score is the target remote node.

[0086] Optionally, the node information of each remote node includes the network delay, bandwidth, available GPU proportion and load of each remote node.

[0087] If the second GPU resource has been exhausted or the network health status of the system is in a failure state, the system enters an intelligent degradation process, and the hook function performs resource allocation in the system memory layer according to a hierarchical resource search strategy: the system memory layer includes multiple memory blocks of the local CPU, and the hook function can select a memory block that meets the GPU allocation request from the local CPU as the target GPU block, ensuring that the application program runs without interruption. This system memory switching mechanism is fast and the application layer is basically unaware.

[0088] Optionally, each time the network health status of the system is detected to be in a failure state, the degradation count is incremented once, and when the degradation count is greater than a degradation count threshold (for example, 5 times), the degradation decision can be directly triggered to directly perform resource allocation in the system memory layer.

[0089] Optionally, the switching time of the system from the second GPU layer to the system memory layer should be less than 100 milliseconds.

[0090] Optionally, when the GPU resources of the three-level GPU layer are exhausted, the system can manage the cache resources in the local GPU swap space through an eviction mechanism based on the LRU algorithm, the cache resources in the local GPU swap space include multiple cache GPU blocks, and the system can sort the cache GPU blocks according to the time stamps of the last access of each cache GPU block, and release the cache GPU block that has not been used for the longest time, thereby reserving more cache space for use.

[0091] Further, after determining the target GPU block to be allocated, the system can allocate the target GPU block to the application program and return the virtual address of the target GPU resource to the application program, and the returned virtual address is transparent to the application program, that is, the application program itself cannot distinguish whether the target GPU block allocated by it is a local GPU resource or a remote GPU resource, and the real physical address of the target GPU block is managed internally by the system.

[0092] Optionally, if the application needs to access data in the video memory block, it can initiate a video memory access request; if the video memory block that the application needs to access is the video memory block of a remote node, the system will enter an asynchronous prefetch process, start RDMA network transmission, perform asynchronous intelligent prefetching of the data in the video memory block of the remote node, and return the data to the application.

[0093] Optionally, the size of the asynchronous intelligent prefetch memory block should be at least 20MB (Megabyte), and the latency masking rate should be greater than 90%.

[0094] The video memory management method provided in this embodiment introduces a multi-level video memory resource pool architecture based on existing technologies. The video memory resource pool is a three-level physical video memory pool, including a local video memory layer, a first video memory layer, and a second video memory layer. The local video memory layer includes the local video memory resources of the local device; the first video memory layer includes the first video memory resources of devices on the same machine; and the second video memory layer includes the second video memory resources of devices used by remote nodes. The local device is the device currently used by the server, and the devices on the same machine are devices deployed on the same server as the local device. Unlike traditional device video memory resource pools, the setup of the first and second video memory layers allows the video memory resource pool to break through the physical limitations of a single card's video memory, realizing cross-device and cross-node video memory resource sharing, improving the scalability of video memory resources. When the application... When a video memory allocation request is initiated, hook functions can be used to select target video memory resources from the video memory resource pool based on a hierarchical resource search strategy. This broadens the range of video memory resources available and effectively addresses the problem of insufficient video memory resources. It requires no modification to the application code, has low usage costs, and can meet the business needs of video memory-intensive application scenarios. Simultaneously, by fully leveraging the advantages of UVM technology, after allocating the target video memory resource to the application, the virtual address of the target video memory resource is returned to the application. This facilitates subsequent location of the target video memory resource's actual physical address through address mapping, enabling the application to use it. This avoids the potential physical address management chaos that can occur with multi-level video memory resource pool architectures and further optimizes video memory management.

[0095] In some embodiments, the target video memory resource includes a target video memory block, the second video memory layer includes multiple remote nodes, and the second video memory resource of each remote node includes at least one second video memory block; selecting the target video memory resource from the video memory resource pool based on a hook function includes: detecting whether the local video memory resource is sufficient based on the hook function; if the local video memory resource is insufficient, detecting whether the first video memory resource is sufficient; if the first video memory resource is insufficient, detecting whether the second video memory resource is sufficient; if the second video memory resource is sufficient, selecting a target remote node from all remote nodes; selecting a second video memory block from the target remote node that satisfies the video memory allocation request as the target video memory block.

[0096] In the embodiment, the target graphics memory resource includes a target graphics memory block, and the second graphics memory layer includes a plurality of remote nodes, and the second graphics memory resource of each remote node includes at least one second graphics memory block.

[0097] Specifically, when the application program initiates a graphics memory allocation request, a memory allocation function in a standard API can be directly called to apply to the system for graphics memory resources meeting the graphics memory allocation request; at this time, the transparent interception layer based on the transparent Hook mechanism can respond to the graphics memory allocation request of the application program, intercept the memory allocation function of the standard API based on the Hook function (i.e., the Hook function), and make automatic graphics memory allocation and degradation decisions through a hierarchical resource search strategy and a network health state perception mechanism, so as to realize transparent graphics memory resource expansion for the application program.

[0098] Further, after the Hook function intercepts the memory allocation function of the standard API, the system will enter a local evaluation process to preferentially attempt local graphics memory resource allocation, and therefore the Hook function will first query the local graphics memory block manager to detect whether the local graphics memory resource is sufficient, i.e., whether there is idle graphics memory resource meeting the graphics memory allocation request in the current local graphics memory layer.

[0099] If the local graphics memory resource is not sufficient, the transparent interception layer can initiate a remote allocation request to the coordination center in the state monitoring and coordination module, at this time, the system enters a network health decision process, and the coordination center can perform network health state detection according to a first preset period.

[0100] If the network health state of the system is a healthy state, the remote graphics memory resource will not fail to be allocated due to network failure, at this time, the system will enter a hierarchical resource search process, and the Hook function will preferentially attempt in-machine graphics memory resource allocation according to the priority of the three-level graphics memory layer defined in the hierarchical resource search strategy, and detect whether the first graphics memory resource is sufficient, i.e., whether there is idle graphics memory resource meeting the graphics memory allocation request in the current first graphics memory layer.

[0101] If the first graphics memory resource is not sufficient, the Hook function will further attempt remote graphics memory resource allocation according to the hierarchical resource search strategy, and detect whether the second graphics memory resource is sufficient, i.e., whether there is idle graphics memory resource meeting the graphics memory allocation request in the current second graphics memory layer.

[0102] If the second graphics memory resource is sufficient, the coordination center can select the optimal target remote node from all remote nodes according to a multi-dimensional scoring mechanism, and select a second graphics memory block meeting the graphics memory allocation request from the target remote node through a gRPC bidirectional stream call to the graphics memory manager of the target remote node, as the target graphics memory block.

[0103] In some embodiments, the target remote node is determined based on the following steps: filtering all remote nodes to obtain an available node set; the available node set includes at least two candidate nodes that meet the video memory allocation request; determining the comprehensive score of each candidate node based on the node information of each candidate node; the node information includes network latency, bandwidth, the proportion of available video memory in the node and load; determining the target remote node based on each comprehensive score; the target remote node is the candidate node with the highest comprehensive score.

[0104] In this embodiment, the coordination center adopts a centralized scheduling architecture and can serve as a core component for global resource management of the system. It can achieve unified scheduling of cross-node memory resources through intelligent scoring algorithms and network health status detection.

[0105] The coordination center includes a scheduling strategy engine and is equipped with a node information table and a global status table.

[0106] The scheduling strategy engine is used to maintain the parameter weights corresponding to different node information. For example, in the scheduling strategy engine, the first weight corresponding to the latency score can be set to 0.4, the second weight corresponding to the bandwidth score to 0.3, the third weight corresponding to the availability score to 0.2, and the fourth weight corresponding to the load score to 0.1.

[0107] The node information table is used to record the node information of all remote nodes. The node information of each remote node includes, but is not limited to, network latency, bandwidth, node available video memory ratio, load, GPU configuration, and last heartbeat time.

[0108] The global state table is used to maintain the memory block identifier (i.e., block ID) of the memory block, realize the location mapping from the memory block identifier to the physical address, and also to maintain the node scoring priority queue, supporting 128K entry management.

[0109] Specifically, such as Figure 5 and Figure 6 As shown, after receiving a remote allocation request, the coordination center can perform node screening on all remote nodes, filtering out remote nodes that do not meet the video memory allocation request or have insufficient video memory resources, and obtain a set of available nodes. The set of available nodes includes at least two candidate nodes that meet the video memory allocation request.

[0110] The remote allocation request carries at least two parameters: the size of the memory block requested by the memory allocation request and the application identifier (i.e., the application ID).

[0111] Furthermore, the coordination center can determine the node information of each candidate node based on the node information table.

[0112] The node information of each candidate node includes, but is not limited to, network delay, bandwidth, available GPU memory ratio, and load.

[0113] Further, based on the node information of each candidate node, a comprehensive score of each candidate node is determined respectively, and based on each comprehensive score, a target remote node is determined, the target remote node being the candidate node with the highest comprehensive score.

[0114] Optionally, if there are two or more candidate nodes with the highest comprehensive score, the candidate node with the smallest load can be selected as the target remote node.

[0115] For example, the highest comprehensive score is 10 points, and if there are three candidate nodes with a comprehensive score of 10 points, the candidate node with the smallest load is selected as the target remote node among the three candidate nodes.

[0116] Optionally, the scheduling strategy engine can recalculate the comprehensive score of each remote node at a fixed period (for example, 1 second), dynamically adjust the resource allocation strategy, and automatically select the optimal GPU memory allocation location according to the comprehensive score, which is beneficial to load balancing and performance optimization.

[0117] Further, after determining the target remote node, the coordination center can send an AllocateBlock RPC request (i.e., an allocation block remote procedure call request) to the target remote node to implement GPU memory allocation of the target remote node, and update the global state table and return the allocation result after the request is sent successfully.

[0118] Optionally, the average delay of the target remote node determination process should be less than 10 microseconds, and the load balancing variance of the system should be less than 15%.

[0119] Optionally, the system is provided with a fault-tolerant mechanism: for each remote node, the system can receive the heartbeat of the remote node at a preset heartbeat period (for example, 2 seconds), if the heartbeat of the remote node is timed out for 3 times in a row, a degraded allocation is triggered; if the system fails to allocate the second GPU memory resource for 5 times in a row or when a network failure occurs, a degraded allocation is triggered, and when the network recovers and the number of idle second GPU memory blocks in the second GPU memory resource is greater than 20%, the system can automatically perform asynchronous data rollback, and switch back to the mode of allocating the second GPU memory resource.

[0120] Optionally, for each remote node, the node can use an incremental update strategy to synchronize heartbeat information to the system synchronizer, which can reduce communication volume, reduce data synchronization delay, and support efficient scheduling of large-scale clusters.

[0121] In some embodiments, the comprehensive score is determined based on a delay score of the candidate node and a first weight corresponding to the delay score, a bandwidth score of the candidate node and a second weight corresponding to the bandwidth score, an availability score of the candidate node and a third weight corresponding to the availability score, and a load score of the candidate node and a fourth weight corresponding to the load score, wherein the delay score is determined based on a network delay of the candidate node, the bandwidth score is determined based on a bandwidth of the candidate node, the availability score is determined based on a node available GPU ratio of the candidate node, and the load score is determined based on a load of the candidate node.

[0122] Optionally, for each candidate node, the node information of the candidate node includes a network delay, a bandwidth, a node available GPU ratio, and a load, and the comprehensive score of the candidate node is determined based on the network delay, the bandwidth, the node available GPU ratio, and the load. The comprehensive score can be expressed by the following formula:

[0123] ;

[0124] wherein, the network delay is denoted as D, the delay score is denoted as Sd, the first weight corresponding to the delay score is denoted as Wd; the bandwidth is denoted as B, the bandwidth score is denoted as Sb, the second weight corresponding to the bandwidth score is denoted as Wb; the node available GPU is denoted as G, the total GPU of the node is denoted as Gt, the node available GPU ratio (i.e. the availability score) is denoted as Ga, the third weight corresponding to the availability score is denoted as Wa; the load is denoted as L, the load score is denoted as Sl, the fourth weight corresponding to the load score is denoted as Wl.

[0125] In some embodiments, the GPU resource pool further includes a system memory layer, the system memory layer including a plurality of memory blocks of a local host, the local host being a host currently used by the server; before determining whether the first GPU resource is sufficient, the method further includes: performing network health state detection at a first preset period; if the heartbeat time of each remote node is greater than a preset time threshold, or the network connection of all remote nodes is interrupted, or the second GPU resource has been exhausted, determining that the network health state is a failure state, and selecting a memory block satisfying the GPU allocation request from the local host as a target GPU block.

[0126] In the present embodiment, the GPU resource pool further includes a system memory layer, the system memory layer including a plurality of memory blocks of a local host, the local host being a host currently used by the server, and the local host including a local CPU, and the present embodiment is described by taking the local CPU as an example.

[0127] Specifically, when an application initiates a video memory allocation request, it can directly call the memory allocation function in the standard API to request video memory resources from the system to satisfy the video memory allocation request. At this time, the transparent interception layer based on the transparent Hook mechanism responds to the application's video memory allocation request. It can intercept the memory allocation function of the standard API based on the hook function (i.e., the Hook function), and through the hierarchical resource search strategy and network health status awareness mechanism, it performs automated video memory allocation and degradation decisions to achieve transparent video memory resource expansion for the application.

[0128] Video memory resources are usually in the form of video memory blocks. After the hook function intercepts the memory allocation function of the standard API, it will enter the local evaluation process and try to allocate local video memory resources first. Therefore, the hook function will first query the local video memory block manager to check whether the local video memory resources are sufficient, that is, check whether the local video memory layer currently has free video memory resources to satisfy the video memory allocation request.

[0129] If local video memory resources are insufficient, the transparent interception layer can initiate a remote allocation request to the coordination center in the status monitoring and coordination module. At this time, the system enters the network health decision process, and the coordination center can perform network health status detection according to the first preset period (e.g., 2 seconds).

[0130] like Figure 7 As shown, in this embodiment, the status monitoring and coordination module is used to collect status data and memory information of each remote node, monitor network health status, and make global scheduling decisions.

[0131] The monitoring agent mechanism of the status monitoring and coordination module adopts a dual-threaded hybrid architecture. The two threads are a periodic sampling thread and an event-driven thread: the periodic sampling thread executes once every 50 milliseconds and can call the memory information acquisition function to obtain the memory information of each remote node, count the number of active memory blocks, cached memory blocks and free memory blocks of each remote node, calculate the cache hit rate of each remote node, and construct compressed data packets for each remote node; the event-driven thread is used to subscribe to page fault events of UVM, connection status events of RDMA network and memory operation events. When the health status of a remote node changes, it immediately triggers event reporting to ensure zero delay for critical events. In addition, the status monitoring and coordination module also runs a heartbeat management thread. The memory information and network health status information reported by each remote node can be packaged and sent to the system in a 2-second cycle through an independent heartbeat management thread. The heartbeat management thread can use Protocol Buffers serialization technology to convert the memory information (denoted as MemoryStats information) and network health status information (denoted as NetworkHealth information) reported by each remote node into a compact binary format, and transmit it to the coordination center in the form of gRPC bidirectional stream, while receiving scheduling instructions from the coordination center.

[0132] Optionally, to reduce communication overhead, the system implements an incremental update strategy, where each remote node only reports the changed fields in the video memory information. At the same time, the update information from multiple (e.g., 100) remote nodes is compressed into a single message to reduce network overhead.

[0133] Optionally, incremental information can be represented by a triplet consisting of the memory block identifier (block_id), a specific field (field), and the field change amount (delta).

[0134] This design achieves a balance between real-time monitoring and low resource consumption, which helps ensure the accuracy of global scheduling decisions.

[0135] Specifically, such as Figure 8 As shown, when the status monitoring and coordination module performs network health status detection, it can perform heartbeat detection on each remote node. If the heartbeat time of any remote node is less than or equal to a preset time threshold (e.g., 15 seconds), it can further determine whether the RDMA network connection of each remote node is interrupted.

[0136] If the RDMA network connection of any remote node is not interrupted, resource availability detection can be performed to determine whether the second video memory resources have been exhausted.

[0137] If the second video memory resource is not exhausted, the network health status of the system can be determined to be healthy.

[0138] Conversely, if the heartbeat time of each remote node is greater than the preset time threshold, or the network connection of all remote nodes is interrupted, or the second video memory resources are exhausted, then the network health status of the system is determined to be a fault state.

[0139] In the event of a failure, the system will trigger a degradation process. The data of the existing remote nodes needs to be asynchronously migrated back to the first video memory layer or the local video memory layer. Based on the video memory allocation request, the system will allocate system memory resources to the application and select a memory block from the local CPU that meets the video memory allocation request as the target video memory block.

[0140] Optionally, the system will continuously monitor the network recovery status and automatically switch back to the second memory layer when the network health status is restored to a healthy state, thereby achieving system self-healing.

[0141] In some embodiments, the target graphics memory resource includes a target graphics memory block, the first graphics memory layer includes a plurality of same-machine devices, the first graphics memory resource of each same-machine device includes at least one first graphics memory block, and the same-machine device includes a same-machine GPU. The present embodiment takes the same-machine GPU as an example for description.

[0142] In the present embodiment, the target graphics memory resource includes a target graphics memory block, the first graphics memory layer includes a plurality of same-machine devices, the first graphics memory resource of each same-machine device includes at least one first graphics memory block, and the same-machine device includes a same-machine GPU. The present embodiment takes the same-machine GPU as an example for description.

[0143] Specifically, when the application program initiates a graphics memory allocation request, a memory allocation function in a standard API can be directly called to apply to the system for graphics memory resources meeting the graphics memory allocation request. At this time, the transparent interception layer based on the transparent Hook mechanism responds to the graphics memory allocation request of the application program, can intercept the memory allocation function of the standard API based on the Hook function, and automatically allocates the graphics memory and makes a downgrade decision through a hierarchical resource search strategy and a network health state perception mechanism, so as to realize transparent graphics memory resource expansion for the application program.

[0144] The graphics memory resource is usually a graphics memory block. After the Hook function intercepts the memory allocation function of the standard API, the local evaluation process is entered, and the local graphics memory resource allocation is preferentially attempted. Therefore, the Hook function first queries the local graphics memory block manager to detect whether the local graphics memory resource is sufficient, that is, whether there is idle graphics memory resource meeting the graphics memory allocation request in the local graphics memory layer.

[0145] If the local graphics memory resource is not sufficient, the transparent interception layer can initiate a remote allocation request to the coordination center in the state monitoring and coordination module. At this time, the system enters the network health decision process, and the coordination center can perform network health state detection according to a first preset period.

[0146] If the network health state of the system is a healthy state, the remote graphics memory resource will not fail to be allocated due to network failure. At this time, the system will enter the hierarchical resource search process, and the Hook function will preferentially attempt the same-machine graphics memory resource allocation according to the priority of the three-level graphics memory layer defined in the hierarchical resource search strategy, that is, whether there is idle graphics memory resource meeting the graphics memory allocation request in the first graphics memory layer.

[0147] If the first GPU memory resource is sufficient, that is, the first GPU memory layer currently has free GPU memory resource satisfying the GPU memory allocation request, a first GPU memory block satisfying the GPU memory allocation request can be directly selected from the first GPU memory layer as the target GPU memory block.

[0148] In some embodiments, the target GPU memory resource includes a target GPU memory block, and the local GPU memory layer includes at least one local GPU memory block; based on the hook function, the target GPU memory resource is selected from the GPU memory resource pool, including: based on the hook function, detecting whether the local GPU memory resource is sufficient; if the local GPU memory resource is sufficient, a local GPU memory block satisfying the GPU memory allocation request is selected from the local GPU memory layer as the target GPU memory block.

[0149] In this embodiment, the target GPU memory resource includes a target GPU memory block, and the local GPU memory layer includes at least one local GPU memory block.

[0150] As shown in Figure 3 and Figure 4 When the application initiates a GPU memory allocation request, a memory allocation function in the standard API can be directly called to apply to the system for GPU memory resource satisfying the GPU memory allocation request; at this time, the transparent interception layer based on the transparent Hook mechanism responds to the GPU memory allocation request of the application, can intercept the memory allocation function of the standard API based on the hook function (that is, the Hook function), and make automatic GPU memory allocation and degradation decision through the hierarchical resource search strategy and the network health state perception mechanism, to realize transparent GPU memory resource expansion for the application.

[0151] The GPU memory resource is usually a GPU memory block, after the hook function intercepts the memory allocation function of the standard API, the local evaluation process will be entered, and the local GPU memory resource allocation is preferentially tried, so the hook function will first query the local GPU memory block manager to detect whether the local GPU memory resource is sufficient, that is, whether the local GPU memory layer currently has free GPU memory resource satisfying the GPU memory allocation request.

[0152] If the local GPU memory resource is sufficient, that is, the local GPU memory layer currently has free GPU memory resource satisfying the GPU memory allocation request, a local GPU memory block satisfying the GPU memory allocation request can be directly selected from the local GPU memory layer as the target GPU memory block.

[0153] It should be noted that the local GPU memory block, the first GPU memory block, the second GPU memory block and the memory block are all managed by the GPU memory block management module of the core management layer, as shown in Figure 9 The GPU memory block management module is used for maintaining the life cycle of all GPU memory blocks, which includes an allocation strategy engine, a fragmentation consolidation engine and an optimistic lock manager, and can maintain a global GPU memory state table, and uses an optimistic lock protocol to process concurrent requests of the application. The core value of the GPU memory block management module lies in supporting high-concurrency allocation requests with lock-free design, while maintaining the consistency of cross-node metadata.

[0154] The allocation strategy engine maintains three priority queues: an active queue, a cached queue, and a free queue. The active queue stores memory blocks marked as active, the cached queue stores memory blocks marked as cached, and the free queue stores memory blocks marked as free. When handling concurrent requests from the application, the allocation strategy engine uses an optimistic locking protocol instead of a traditional mutex mechanism. Each memory block carries a version number, and the consistency of the version number must be verified before and after a read operation. Write operations on memory blocks use CAS (Compare and Swap) atomic instructions for submission. If a conflict occurs in a memory block operation, it will be retried a maximum of three times. If the conflict is not resolved after three retries, a fallback process is triggered.

[0155] The defragmentation engine periodically scans the global memory status table and uses CAS atomic operations to merge free memory blocks across nodes, reducing memory fragmentation. The global memory status table uses a hash table structure, with each entry being 64 bytes, and can manage hundreds of thousands of blocks.

[0156] Optionally, the entries in the global memory status table include, but are not limited to, memory block identifier (block_id), memory block address (location), memory block status (active, cached, and released can be represented by different numbers), memory block size (size), last access timestamp (last_access), access count (access_count), optimistic version number (version), and application identifier (app_id).

[0157] The block state machine in the memory block management module can strictly manage the lifecycle of each memory block: memory blocks marked as free can be allocated and used immediately; memory blocks marked as active need to record the application identifier (i.e., application ID) and increment the version number; memory blocks marked as cached need to retain the corresponding data for fast reuse. When the LRU algorithm's eviction mechanism is triggered, if there is dirty data in the memory block, it needs to be written back to the remote node first.

[0158] Optionally, when a video memory block marked as Free transitions to Active state, the video memory block management module triggers the allocate_block function. At this time, the optimistic version number of the video memory block is incremented, the application identifier app_id is initialized, and the video memory block needs to hold a table-level lock table_mutex. The delay of this state transition process should be less than 1 microsecond.

[0159] Optionally, when a memory block marked as active transitions to cached state, the memory block management module triggers a memory release hook function. At this time, the optimistic version number of the memory block is incremented, and the timestamp of the last access, last_access, is updated. This state transition process is a lock-free operation, and the delay should be less than 0.5 microseconds.

[0160] Optionally, when a memory block marked as cached transitions to free, the memory block management module triggers the evict_lru function. At this time, the optimistic version number of the memory block is incremented, and the metadata is cleared. This state transition process can be executed asynchronously without affecting the system's main thread.

[0161] Optionally, when a memory block marked as cached transitions to active state, the memory block management module reallocates the memory block. At this time, the optimistic version number (version) of the memory block is incremented, the access count (access_count) is incremented, and the LRU algorithm hits the memory block. The delay of this state transition process should be less than 0.5 microseconds.

[0162] Optionally, the defragmentation engine employs a defragmentation mechanism: under long-term system operation, external fragmentation may occur in the video memory resource pool. In this case, the system can manage adjacent free video memory blocks through a doubly linked list to achieve efficient merging of video memory blocks.

[0163] by Figure 10 For example, if there are two or more memory blocks marked as free and their addresses are consecutive, these memory blocks can be merged into one memory block, increasing the size of the merged memory block.

[0164] In some embodiments, the second GPU resource of the remote node comprises a second GPU block; after the target GPU resource is allocated to the application program and the virtual address of the target GPU resource is returned to the application program, the method further comprises: in response to a GPU access request of the application program, detecting, based on the hook function, whether a to-be-accessed GPU block required to be accessed by the application program is the second GPU block; if the to-be-accessed GPU block required to be accessed by the application program is the second GPU block, detecting whether the to-be-accessed GPU block can be hit in the local buffer; the local buffer comprises a plurality of cached GPU blocks; if the to-be-accessed GPU block cannot be hit in the local buffer and the current available cache capacity of the local buffer is insufficient, releasing at least one cached GPU block based on the LRU algorithm, writing data of the second GPU block of the remote node into the local buffer to generate a to-be-accessed cached GPU block, determining a local cache address of the to-be-accessed cached GPU block in the local buffer, and redirecting the GPU access request to the local cache address.

[0165] It can be understood that the application program also has a demand for accessing GPU data.

[0166] Specifically, as shown in Figure 11 When the application program initiates a GPU access request, a data transmission function in the standard API can be directly called to request data of a to-be-accessed GPU block required to be accessed by the system; at this time, the transparent interception layer based on the transparent Hook mechanism can respond to the GPU access request, and the hook function can intercept the data transmission function of the standard API, and the hook function can detect whether the to-be-accessed GPU block required to be accessed by the application program is a remote GPU block (i.e., the second GPU block).

[0167] Optionally, the hook function can detect a source address or a target address carried by the GPU access request, and if the source address or the target address carried by the GPU access request points to the second GPU block, it can be determined that the to-be-accessed GPU block required to be accessed by the application program is the second GPU block.

[0168] Further, if the to-be-accessed GPU block required to be accessed by the application program is the second GPU block, it is detected whether the to-be-accessed GPU block can be hit in the local buffer.

[0169] Specifically, the local GPU exchange space comprises a local buffer, and the local buffer comprises a plurality of cached GPU blocks; when the application program needs to access a to-be-accessed GPU block, the system can first search for a GPU block identifier block_id of the to-be-accessed GPU block in a cache hash table of the local buffer; if the system can search for the GPU block identifier block_id of the to-be-accessed GPU block in the cache hash table of the local buffer, it is considered that the to-be-accessed GPU block can be hit in the local buffer.

[0170] If the to-be-accessed GPU block can hit in the local buffer, the system can directly extract the data corresponding to the to-be-accessed GPU block in the local buffer, and update the last access timestamp last_access of the to-be-accessed GPU block to the current nanosecond value in an atomic operation. The LRU algorithm maintains a cache GPU block access list lru_list. At this time, the to-be-accessed GPU block can be moved to the head of the cache GPU block access list lru_list, and the local cache address of the to-be-accessed GPU block in the local buffer is returned to the application program. The hook function can redirect the GPU access request to the local cache address and perform local data copying. In the scenario where the to-be-accessed GPU block can hit in the local buffer, the hit path delay of the system can be less than 1.2 microseconds, and there is no blocking overhead.

[0171] If the to-be-accessed GPU block cannot hit in the local buffer, and the current available cache capacity of the local buffer is insufficient, at least one cache GPU block is released based on the LRU algorithm, and an asynchronous RDMA prefetch process is triggered, and the current request is suspended. After the data of the second GPU block of the remote node is written into the local buffer, the to-be-accessed cache GPU block is generated, the address mapping table is updated, and the local cache address of the to-be-accessed cache GPU block in the local buffer is determined according to the address mapping table. The hook function can redirect the GPU access request to the local cache address, and perform local data copying. This access mechanism can hide the remote access delay behind the computation, and the delay perceived by the application layer is close to local access.

[0172] Specifically, if the to-be-accessed GPU block cannot hit in the local buffer, the system will trigger a complete prefetch process.

[0173] Firstly, the system will first detect the current available cache capacity current_usage of the local buffer.

[0174] If the current available cache capacity current_usage of the local buffer is insufficient, the system will call the GPU block eviction function evict_lru(), and asynchronously evict a plurality of least recently used cache GPU blocks at the tail of the cache GPU block access list lru_list. At this time, the system can release the local cache space occupied by these evicted cache GPU blocks through the memory release function, until it is determined that the current available cache capacity current_usage of the local buffer is sufficient.

[0175] Optionally, if the GPU block size of the to-be-accessed GPU block exceeds the capacity of the cache pool in the local GPU swap space after being added to the used cache capacity of the local buffer, it is considered that the current available cache capacity current_usage of the local buffer is insufficient.

[0176] Optionally, for each released cache memory block, the system detects the dirty data flag is_dirty of the memory block, and if the dirty data flag indicates that the data of the memory block has been modified, the system initiates the RDMA transmission engine to synchronously write back the modified data to the corresponding remote node, and after the data write back is completed, the local cache space occupied by the evicted cache memory block is released.

[0177] If it is determined that the current available cache capacity current_usage of the local buffer is sufficient, the system can allocate a memory page of the local buffer to the to-be-accessed memory block through a cache allocation function, and then submit an asynchronous transmission request to the RDMA transmission engine. The RDMA transmission engine can initiate a GPU Direct RDMA read operation by using an asynchronous function ibv_post_send, and the local network card hardware can directly read the data of the corresponding second memory block from the GPU memory of the remote node to the local buffer. After the data transmission is completed, the to-be-accessed cache memory block is created and the data of the to-be-accessed cache memory block is filled. At this time, the dirty data flag is_dirty of the to-be-accessed cache memory block is false (the dirty data flag is_dirty being false indicates that the data has not been modified). The system can insert the to-be-accessed cache memory block into the head of the cache memory block index and the cache memory block access list lru_list, and update the address mapping table. Then, the system can determine the local cache address of the to-be-accessed cache memory block in the local buffer according to the address mapping table, the hook function can redirect the memory access request to the local cache address, and perform local data copying. This process does not require the CPU to participate in data transmission. In the scenario where the to-be-accessed memory block fails to hit in the local buffer, the total time of the miss path of the system is about 12 milliseconds, but after the intelligent prefetching process is optimized, the actual waiting time of the application layer can be reduced to the millisecond level.

[0178] Optionally, during the data transmission process of the remote node GPU memory, the application thread of the system can choose to be lightly blocked on the synchronization function poll_cq, or continue to perform non-dependent calculation to realize delay masking.

[0179] It should be noted that the role of the RDMA transmission engine is to realize cross-node GPU memory data zero-copy transmission through the GPU Direct RDMA technology, and its architecture can be divided into four layers: the top layer defines an RDMA Transfer class to provide a data transmission interface; the configuration management layer is used to manage the context, protection domain, completion queue and queue pair of the RDMA device; the memory management layer is used to maintain the local GPU memory registry and the MR (Memory Region) cache of the remote node; and the bottom layer can realize the direct interaction between the GPU and the network card through hardware.

[0180] As Figure 12As shown, the core workflow of the RDMA transmission engine starts from the registration of the display memory: before the allocation of the remote display block (i.e., the second display block), the system can call the ibv_reg_mr function to register the virtual address of the GPU of the remote node as a memory region accessible by the RDMA transmission engine, set the atomic operation permissions such as local write, remote read, and remote write; then, the RDMA transmission engine can obtain the local key lkey of the local GPU, and for the remote node, the RDMA transmission engine can exchange the MR information of the remote node in the form of RPC (Remote Procedure Call), obtain the memory address and remote key rkey of the remote node; when performing data transmission between the remote node and the local, create Scatter elements or Gather elements (Scatter elements and Gather elements are core data structures for efficient processing of non-continuous memory data) to describe the data layout, at this time, the RDMA transmission engine can create and send a work request to the RDMA card, call the ibv_post_send function to submit the work request to the queue pair, and after receiving the work request, the card hardware can directly read the GPU display memory data, and send it to the opposite GPU (i.e., the GPU of the remote node) through the InfiniBand (InfiniBand Architecture) network or the RoCE (RDMA over Converged Ethernet) network. In the above process, the CPU does not participate in data transmission, and only receives the transmission end notification asynchronously through the completion queue. The RDMA transmission engine can effectively improve the bandwidth utilization and reduce the CPU occupancy, and the end-to-end delay of the local CPU and the remote node CPU can be reduced to 1-10 milliseconds, so that the system breaks through the limitation of the traditional PCIe bandwidth, improves the remote display memory access performance to the level close to the local, and lays a foundation for building a distributed GPU memory pool using unified addressing.

[0181] Optionally, to reduce the data transmission overhead, the system can use a batch merging algorithm to merge multiple data transmission requests with consecutive addresses into one request to realize batch transmission and reduce the network round trip times of data transmission requests.

[0182] Optionally, the system uses a double buffering mechanism or a multi-buffering mechanism, and a pre-fetch work thread is used to set at least one standby buffer in the background, while the main thread is performing calculation through the local buffer (i.e., the current buffer), the pre-fetch thread is used to pre-fetch the display memory data of the remote node and write it into the standby buffer, so as to overlap the transmission flow and the calculation flow, and further reduce the delay.

[0183] In some embodiments, in response to the application's memory access request, the method further comprises: obtaining historical access records at a second preset period; the historical access records comprise access information of each cache memory block within a preset time period; performing temporal locality analysis on the historical access records to determine a first candidate set; the first candidate set comprises at least one first candidate memory block; performing spatial locality analysis on the historical access records to determine a second candidate set; the second candidate set comprises at least one second candidate memory block; performing access pattern analysis on the historical access records to determine a third candidate set; the third candidate set comprises at least one third candidate memory block; selecting a plurality of target candidate memory blocks from the first candidate set, the second candidate set, and the third candidate set, and writing data of each target candidate memory block into the standby buffer.

[0184] As shown in Figure 13 and Figure 14 To further reduce the delay of the system, the embodiment introduces an intelligent prefetching mechanism and a double buffering mechanism: the double buffering mechanism allows the system to set two buffers in the local memory exchange space, namely the local buffer (current buffer) and the standby buffer; the core of the intelligent prefetching mechanism is to actively predict the future data needs of the application by analyzing the historical access patterns of the application, and when the application uses the data in the current buffer for calculation, the data that the application may need in the future is written into the standby buffer in advance through an independent transmission stream, realizing the temporal and spatial overlap of the calculation stream and the transmission stream, thereby effectively reducing the remote access delay.

[0185] Specifically, the system can construct a prediction model in advance according to the locality principle of application access, and maintain a recent access record ring queue through a prefetching manager, which is used to continuously track the access time, access frequency, and access type of each cache memory block accessed by the application.

[0186] Further, the prefetching manager can obtain historical access records at a second preset period (for example, 100 milliseconds), and the historical access records comprise access information of each cache memory block within a preset time period.

[0187] For example, the prefetching manager can extract recent historical access records from the recent access record ring queue as analysis samples at a time interval of every 100 milliseconds, and the recent historical access records comprise access records of the last 1000 cache memory blocks.

[0188] Further, the prediction model can perform temporal locality analysis on the historical access records, detect and count high-frequency cache memory blocks in the historical access records, identify hot data, determine a first candidate set, and the first candidate set comprises at least one first candidate memory block. The first candidate memory block is a cache memory block in the historical access records whose access frequency is greater than a preset threshold.

[0189] Further, the prediction model can perform spatial locality analysis on the historical access records, detect the continuity of cache block identifiers (i.e., video memory block IDs of the cache video memory blocks), extract sequential access features in the historical access records, and determine a second candidate set based on the sequential access features, the second candidate set including at least one second candidate video memory block, the second candidate video memory block being a cache video memory block having the sequential access features.

[0190] Further, the prediction model can perform access pattern analysis on the historical access records, identify complex access rules such as cyclic access patterns, sequential access patterns, etc. in the historical access records through a sliding window algorithm, and determine a third candidate set, the third candidate set including at least one third candidate video memory block, the third candidate video memory block being a cache video memory block having the cyclic access patterns, the sequential access patterns, etc.

[0191] The cyclic access patterns refer to the periodicity of access to the cache video memory blocks, and the sequential access patterns refer to linear access patterns of the cache video memory blocks. The sequential access patterns can be determined by matching historical sequence templates, and can be used to predict subsequent access streams.

[0192] The multi-dimensional analysis of the prediction model can make the prefetching decision of the system more in line with the actual application characteristics.

[0193] Further, the system can combine the first candidate set, the second candidate set, and the third candidate set into a to-be-de-duplicated list, and perform de-duplication processing on all candidate video memory blocks in the to-be-de-duplicated list to obtain a to-be-prefetched video memory block list, the to-be-prefetched video memory block list including a plurality of to-be-prefetched video memory blocks.

[0194] Further, the prediction model can generate a corresponding prediction confidence for each to-be-prefetched video memory block, sort all to-be-prefetched video memory blocks according to their corresponding prediction confidences, select a plurality of target candidate video memory blocks with the highest prediction confidences from among them, and write data of each target candidate video memory block into a standby buffer in advance through an independent transmission stream.

[0195] The plurality of target candidate video memory blocks with the highest prediction confidences are data cache video memory blocks that are most likely to be accessed by the application in the future.

[0196] This intelligent prefetching mechanism can enable the computing stream and the transmission stream to be executed asynchronously, which can effectively avoid blocking of the main computing stream. If the target candidate video memory block is a video memory block of a remote node, an independent prefetching worker thread can be used to implement RDMA network transmission of remote data to carry the remote data to the local, and the main thread only needs to check a prefetching status flag to quickly obtain a local cache address of the target candidate video memory block.

[0197] In addition, the double buffering mechanism allows data transmission to be parallel to kernel computation, and when the application accesses the current buffer, the standby buffer has been filled with predicted data in advance, and when the application needs to access the data of the standby buffer, the system can achieve seamless switching of the two buffers. This pipeline mechanism completely hides the latency of remote access in the computation period, which can reduce the overhead of GPU idle waiting.

[0198] Optionally, the system can maintain a prefetch window configuration to ensure that data writing to the standby buffer is completed 2-3 computation periods in advance to reduce access delay.

[0199] Optionally, when the system detects that the cache pressure of the buffer increases or the hit rate decreases, the prefetch range of the standby buffer can be automatically expanded to maintain performance.

[0200] Optionally, to further reduce the delay of the system, the system combines the transmission optimization algorithm with the double buffering mechanism.

[0201] Specifically, if the system needs to simultaneously initiate multiple data transmission requests, each data transmission request is used to request data of a second memory block, the system can sort the data transmission requests according to the access addresses of the second memory blocks corresponding to the data transmission requests, check whether the access addresses of the second memory blocks corresponding to adjacent data transmission requests are continuous, and check whether the size of the multiple data transmission requests with continuous access addresses of the second memory blocks is within a data transmission threshold after merging.

[0202] If the access addresses of the second memory blocks corresponding to adjacent data transmission requests are continuous, and the size of the multiple data transmission requests with continuous access addresses of the second memory blocks is within the data transmission threshold after merging, the data transmission requests can be merged into one work request.

[0203] Further, based on the work request, the data of the second memory blocks required by all the data transmission requests is obtained at one time.

[0204] Optionally, the size of the work request obtained by merging is not more than 64 KB.

[0205] This transmission optimization algorithm's batch merging mechanism significantly reduces network round trips and protocol overhead for data transmission requests. The double buffering mechanism allows the system to establish two buffers, enabling overlap between the data transmission and computation streams through a prefetching worker thread. While the main thread uses the current buffer, the prefetching worker thread fills the backup buffer in the background. The synergistic work of the transmission optimization algorithm and the double buffering mechanism allows the system to batch merge data transmission requests to reduce the overhead of small data transmissions. Double buffering reduces data transmission latency, improving the overall system throughput, effectively optimizing network efficiency, and reducing end-to-end latency, thus providing high-performance support for high-concurrency memory access.

[0206] by Figure 15 For example, without the batch merging mechanism, if there are 16 data transmission requests, the system will generate 16 network round trips. After the batch merging mechanism is introduced, the 16 data transmission requests can be merged into two work requests. For example, data transmission requests 1 to 7 can be merged into work request 1, and data transmission requests 8 to 16 can be merged into work request 2. Then the system will only generate 2 network round trips, and the network round trip overhead of the system can be significantly reduced.

[0207] by Figure 16 For example, in addition to the batch merging mechanism, the system also introduces a double buffering mechanism: assuming the system uses the current buffer for computation between 0 and 100 milliseconds, then at 50 milliseconds, the system can prefetch the backup buffer; at 100 milliseconds, the system can directly switch from the current buffer to the backup buffer and continue computation using the prefetched data; at 150 milliseconds, the system can prefetch the current buffer; and at 200 milliseconds, the system can directly switch from the backup buffer to the current buffer and continue computation using the prefetched data. This mechanism achieves overlap between the data transmission stream and the computation stream, further reducing system latency.

[0208] In some embodiments, in response to a video memory release request from an application, a video memory block to be released is determined based on a hook function; the video memory block to be released is retained in a local buffer based on a delayed release strategy; if the current available cache capacity of the local buffer is insufficient, or the available video memory resources in the video memory resource pool are insufficient, the video memory block to be released is released based on the LRU algorithm.

[0209] Specifically, when an application initiates a video memory release request, it can directly call the memory release function in the standard API to request the system to release the video memory resources of the video memory block to be released. At this time, the transparent interception layer based on the transparent Hook mechanism responds to the application's video memory release request. It can intercept the memory release function of the standard API based on the hook function and query the video memory blocks to be released managed by the system.

[0210] Further, according to the delayed release strategy, the hook function can mark the to-be-released graphics memory block as a cached state, and move the to-be-released graphics memory block into the local buffer, instead of immediately releasing the graphics memory resource of the to-be-released graphics memory block, so as to realize fast reuse of the graphics memory block within a certain time.

[0211] Further, the system continuously monitors the resource usage of the local buffer and the graphics memory resource pool, and if the current available cache capacity of the local buffer is insufficient, or the available graphics memory resource of the graphics memory resource pool is insufficient, the to-be-released graphics memory block is released based on the LRU algorithm.

[0212] Optionally, if the to-be-released graphics memory block is a remote graphics memory block (i.e. the second graphics memory block), the system can also notify the coordination center to release the resource reference count.

[0213] The delayed release strategy can reduce the overhead of frequent allocation and release of graphics memory blocks, and is beneficial to improve the reuse rate of graphics memory resources.

[0214] As can be seen from the above embodiments, the graphics memory allocation process, the graphics memory access process and the graphics memory release process are all realized through the transparent interception layer based on the transparent Hook mechanism, and therefore the Hook mechanism is the core of the system to realize transparent graphics memory extension and management, and the mechanism is mainly realized based on the dynamic linker.

[0215] As shown in Figure 17 When the application program is started, the dynamic linker can forcibly pre-load the Hook library through the environment variable, and use the dynamic linker to analyze the priority, so as to redirect the call of the application program to the standard API such as the memory allocation function, the data transmission function and the memory release function to the wrapper function (i.e. the hook function) of the Hook library.

[0216] Further, the Hook library initialization is performed: when the Hook library is initialized, the constructor is used to complete the acquisition of the original function pointer and the initialization of each subsystem before the execution of the main function, each subsystem can start a background monitoring agent thread, establish a connection with the coordination center, and complete the Hook library initialization.

[0217] Each subsystem refers to a process module on each remote node.

[0218] Furthermore, during system runtime, for intercepted memory allocation functions, the Hook library's wrapper function first attempts to allocate local video memory resources. If allocation is successful, it records the local allocation information and returns the virtual address of the local video memory resource. If allocation fails due to insufficient local video memory resources, it switches to the extended allocation path, attempting to obtain video memory resources from the first video memory layer, the second video memory layer, or the system memory layer. After successfully obtaining video memory resources, it returns the virtual address of the video memory resource. This process is completely transparent to the application layer, and the error codes from the application layer are consistent with the native interface.

[0219] For intercepted memory allocation functions, the wrapper function of the Hook library checks the global mapping table to see if the video memory block to be accessed by the application belongs to a remote video memory block. If the source address carried by the video memory access request points to a remote video memory block, a prefetch mechanism is triggered to ensure that the data of the remote video memory block is cached in the local video memory swap space, and the video memory access request is redirected to the local cache address to perform a local data copy. If the target address carried by the video memory access request points to a remote video memory block, the video memory access request also needs to be redirected to the local cache address, and a dirty data flag is added to the remote video memory block, which is then asynchronously written back by a background thread. The address translation latency of this process can be controlled at the microsecond level, and its impact on system data transfer performance is negligible.

[0220] For intercepted memory release functions, the wrapper function of the Hook library will adopt a delayed release strategy. After querying the system-managed memory block to be released, it will only update the status of the memory block to be released to the cached state and move it into the local buffer, instead of releasing the memory resources immediately, so as to achieve fast reuse of memory blocks.

[0221] like Figure 18 As shown, the transparency of the transparent interception layer mainly relies on the address virtual mapping layer. The transparent interception layer needs to maintain a virtual address mapping table to implement the mapping between the virtual address of the video memory block and the video memory block identifier. From the application's perspective, the role of the virtual address mapping table is to map the contiguous virtual addresses it uses to the dispersed physical storage locations of the video memory resource pool. For the application, it can only perceive a unified virtual address space, and the complexity of the underlying multi-level video memory structure is completely hidden.

[0222] by Figure 18For example, assume that the application is assigned three memory blocks, and the virtual addresses corresponding to the three memory blocks are memory block address 1, memory block address 2, and memory block address 3, respectively. The address virtual mapping layer of the transparent interception layer can determine, according to the virtual address mapping table, that the memory block corresponding to the memory block address 1 is a memory block with a memory block identifier 1, the memory block corresponding to the memory block address 2 is a memory block with a memory block identifier 2, and the memory block corresponding to the memory block address 3 is a memory block with a memory block identifier 3. The memory block with the memory block identifier 1 is from the local memory layer, the memory block with the memory block identifier 2 is from the first memory layer, and the memory block with the memory block identifier 3 is from the second memory layer. Moreover, the data of the memory block with the memory block identifier 3 is cached in the local buffer after the pre-fetch process.

[0223] Compared with the prior art, the memory management method provided in the embodiment has at least the following advantages:

[0224] (1) Good transparency and ecological compatibility: The Hook technology is introduced, and an interception mechanism is constructed when the API is called. The application layer can completely reuse the standardized programming interface without code modification, and can be compatible with deep learning frameworks such as PyTorch and TensorFlow, and MPI (Message Passing Interface) parallel applications, with low development cost.

[0225] (2) A highly intelligent adaptive management system is constructed: The system is built-in and introduces a multi-level intelligent decision-making mechanism. The coordination center can dynamically evaluate the scores of each remote node according to the network health status, resource availability, load balancing, and other dimensions to achieve intelligent selection in fine granularity. The intelligent degradation mechanism can automatically switch to the system memory layer in extreme cases to ensure continuous application running. The intelligent prefetching mechanism can actively predict future data requirements and perform early scheduling through analysis of the access mode of the memory block, and can convert passive response into active service.

[0226] (3) System performance optimization: The multi-level memory pool architecture breaks through the limitation of single-machine memory resources and realizes cross-node memory pooling sharing. Through the GPU Direct RDMA technology, the CPU and the kernel are bypassed to build a direct channel between the GPUs, and the transmission efficiency is deeply optimized through the double buffering mechanism and the batch merging algorithm, which can effectively improve the system performance. The optimistic locking protocol is used to solve the concurrent conflict between different requests in a lock-free design, which can improve the utilization rate of memory resources. The above mechanisms work together to make the remote access performance approach the local memory experience.

[0227] (4) Significant economic and operational value: Through virtualization and pooling management of video memory resources, the hardware utilization rate of the system is greatly improved, which helps to reduce the dependence on high-end GPUs, thereby achieving considerable savings in hardware procurement and energy consumption costs; centralized monitoring and automated fault recovery strategies can reduce operational complexity and enhance the system's self-healing ability.

[0228] (5) Improved system reliability and resource scalability: The optimistic locking mechanism based on version number and the design of RDMA atomic operation are conducive to ensuring the consistency of cross-node data; the multi-level video memory resource pool architecture supports smooth expansion from a single machine to a large-scale remote node cluster. This linear architecture design is conducive to ensuring that the system performance can grow steadily with the node scale; the complete heartbeat detection and incremental state synchronization mechanism is conducive to providing the system with telecom-grade reliability assurance.

[0229] The present invention also provides a video memory management device. Please refer to [link / reference]. Figure 19 , Figure 19 This is a schematic diagram of the video memory management device provided by the present invention. In this embodiment, the video memory management device includes a resource selection module 1910 and a resource allocation module 1920.

[0230] The resource selection module 1910 is used to select target video memory resources from the video memory resource pool in response to the application's video memory allocation request, based on hook functions.

[0231] The resource allocation module 1920 is used to allocate target video memory resources to the application and return the virtual address of the target video memory resources to the application.

[0232] The video memory resource pool includes a local video memory layer, a first video memory layer, and a second video memory layer. The local video memory layer includes the local video memory resources of the local device, the first video memory layer includes the first video memory resources of the same device, and the second video memory layer includes the second video memory resources of the device used by the remote node. The local device is the device currently used by the server, and the same device is the device deployed on the same server as the local device. The target video memory resource is determined by the hook function from the local video memory resources, the first video memory resources, and the second video memory resources based on the hierarchical resource search strategy.

[0233] In some embodiments, the target GPU resource includes a target GPU block, the second GPU layer includes a plurality of remote nodes, and the second GPU resource of each remote node includes at least one second GPU block; based on the hook function, the target GPU resource is selected from the GPU resource pool, including: based on the hook function, detecting whether the local GPU resource is sufficient; if the local GPU resource is not sufficient, detecting whether the first GPU resource is sufficient; if the first GPU resource is not sufficient, detecting whether the second GPU resource is sufficient; if the second GPU resource is sufficient, selecting a target remote node from all remote nodes; and selecting a second GPU block that meets the GPU allocation request from the target remote node as the target GPU block.

[0234] In some embodiments, the target remote node is determined based on the following steps: performing node screening on all remote nodes to obtain a set of available nodes; the set of available nodes includes at least two candidate nodes that meet the GPU allocation request; based on node information of each candidate node, respectively determining a comprehensive score of each candidate node; the node information includes network delay, bandwidth, node available GPU proportion, and load; based on each comprehensive score, determining the target remote node; and the target remote node is the candidate node with the highest comprehensive score.

[0235] In some embodiments, one comprehensive score is determined based on a delay score and a first weight corresponding to the delay score, a bandwidth score and a second weight corresponding to the bandwidth score, an availability score and a third weight corresponding to the availability score, and a load score and a fourth weight corresponding to the load score of one candidate node; wherein the delay score is determined based on the network delay of the candidate node, the bandwidth score is determined based on the bandwidth of the candidate node, the availability score is determined based on the node available GPU proportion of the candidate node, and the load score is determined based on the load of the candidate node.

[0236] In some embodiments, the GPU resource pool further includes a system memory layer, the system memory layer includes a plurality of memory blocks of a local host, and the local host is a host currently used by the server; before detecting whether the first GPU resource is sufficient, further including: performing network health state detection according to a first preset period; if the heartbeat time of each remote node is greater than a preset time threshold, or the network connection of all remote nodes is interrupted, or the second GPU resource has been exhausted, determining that the network health state is a failure state, and selecting a memory block that meets the GPU allocation request from the local host as the target GPU block.

[0237] In some embodiments, the target graphics memory resource includes a target graphics memory block, and the local graphics memory layer includes at least one local graphics memory block; and selecting the target graphics memory resource from the graphics memory resource pool based on the hook function includes: detecting, based on the hook function, whether the local graphics memory resource is sufficient; and if the local graphics memory resource is sufficient, selecting a local graphics memory block that meets the graphics memory allocation request from the local graphics memory layer as the target graphics memory block.

[0238] In some embodiments, the target graphics memory resource includes a target graphics memory block, and the local graphics memory layer includes at least one local graphics memory block; and selecting the target graphics memory resource from the graphics memory resource pool based on the hook function includes: detecting, based on the hook function, whether the local graphics memory resource is sufficient; and if the local graphics memory resource is sufficient, selecting a local graphics memory block that meets the graphics memory allocation request from the local graphics memory layer as the target graphics memory block.

[0239] In some embodiments, the second graphics memory resource of the remote node includes a second graphics memory block; and after allocating the target graphics memory resource to the application program and returning a virtual address of the target graphics memory resource to the application program, the method further includes: in response to a graphics memory access request of the application program, detecting, based on the hook function, whether a to-be-accessed graphics memory block required to be accessed by the application program is the second graphics memory block; if the to-be-accessed graphics memory block required to be accessed by the application program is the second graphics memory block, detecting whether the to-be-accessed graphics memory block can be hit in a local buffer; the local buffer includes a plurality of cached graphics memory blocks; if the to-be-accessed graphics memory block cannot be hit in the local buffer and a current available cache capacity of the local buffer is insufficient, releasing at least one cached graphics memory block based on an LRU algorithm, writing data of the second graphics memory block of the remote node into the local buffer to generate a to-be-accessed cached graphics memory block, determining a local cache address of the to-be-accessed cached graphics memory block in the local buffer, and redirecting the graphics memory access request to the local cache address.

[0240] In some embodiments, after responding to the graphics memory access request of the application program, the method further includes: acquiring historical access records at a second preset period; the historical access records include access information of each cached graphics memory block within a preset time period; performing temporal locality analysis on the historical access records to determine a first candidate set; the first candidate set includes at least one first candidate graphics memory block; performing spatial locality analysis on the historical access records to determine a second candidate set; the second candidate set includes at least one second candidate graphics memory block; performing access pattern analysis on the historical access records to determine a third candidate set; the third candidate set includes at least one third candidate graphics memory block; selecting a plurality of target candidate graphics memory blocks from the first candidate set, the second candidate set and the third candidate set, and writing data of each target candidate graphics memory block into a standby buffer.

[0241] In some embodiments, in response to a graphics memory release request of an application program, a graphics memory block to be released is determined based on a hook function; the graphics memory block to be released is retained to a local buffer based on a delayed release strategy; and if a current available cache capacity of the local buffer is insufficient or available graphics memory resources of a graphics memory resource pool are insufficient, the graphics memory block to be released is released based on an LRU algorithm.

[0242] The application further provides an electronic device. Figure 20 is a structural schematic diagram of the electronic device provided by the application, as Figure 20 shown, the electronic device can include a processor 2010, a communication interface 2020, a memory 2030, and a communication bus 2040, wherein the processor 2010, the communication interface 2020, and the memory 2030 complete mutual communication through the communication bus 2040. The processor 2010 can invoke a logical instruction in the memory 2030 to execute a graphics memory management method.

[0243] In addition, the logical instruction in the memory 2030 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0244] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the graphics memory management method provided by each method described above.

[0245] The application further provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer readable storage medium, and the computer program being executed by a processor, the computer being capable of executing the graphics memory management method provided by each method described above.

[0246] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0247] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0248] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video memory management method, characterized in that, include: In response to the application's video memory allocation request, a target video memory resource is selected from the video memory resource pool based on hook functions; Allocate the target video memory resource to the application and return the virtual address of the target video memory resource to the application; The video memory resource pool includes a local video memory layer, a first video memory layer, and a second video memory layer. The local video memory layer includes local video memory resources of local devices. The first video memory layer includes first video memory resources of devices on the same machine. The second video memory layer includes second video memory resources of devices used by remote nodes. The local device is the device currently used by the server. The devices on the same machine are devices deployed on the same server as the local device. The target video memory resource is determined by the hook function from the local video memory resources, the first video memory resources, and the second video memory resources based on a hierarchical resource search strategy. Following the application's request for video memory access, the system also includes: According to the second preset period, historical access records are obtained; the historical access records include access information of each cached memory block within a preset time period, and the cached memory block is stored in a local buffer. Temporal locality analysis is performed on the historical access records to determine a first candidate set; the first candidate set includes at least one first candidate memory block; Spatial locality analysis is performed on the historical access records to determine a second candidate set; the second candidate set includes at least one second candidate memory block; Access pattern analysis is performed on the historical access records to determine a third candidate set; the third candidate set includes at least one third candidate memory block; Multiple target candidate memory blocks are selected from the first candidate set, the second candidate set, and the third candidate set, and the data of each target candidate memory block is written into the spare buffer.

2. The video memory management method according to claim 1, characterized in that, The target video memory resource includes a target video memory block, and the second video memory layer includes a plurality of remote nodes, wherein the second video memory resource of each remote node includes at least one second video memory block. The process of selecting target video memory resources from the video memory resource pool based on hook functions includes: Based on the hook function, it is determined whether the local video memory resources are sufficient; If the local video memory resources are insufficient, then check whether the first video memory resources are sufficient. If the first video memory resources are insufficient, then check whether the second video memory resources are sufficient. If the second video memory resources are sufficient, then the target remote node is selected from all the remote nodes; The second video memory block that satisfies the video memory allocation request is selected from the target remote node and is used as the target video memory block.

3. The video memory management method according to claim 2, characterized in that, The target remote node is determined based on the following steps: Node filtering is performed on all the remote nodes to obtain a set of available nodes; the set of available nodes includes at least two candidate nodes that satisfy the video memory allocation request; Based on the node information of each candidate node, a comprehensive score is determined for each candidate node; the node information includes network latency, bandwidth, node available video memory ratio, and load. Based on each of the comprehensive scores, the target remote node is determined; the target remote node is the candidate node with the highest comprehensive score.

4. The video memory management method according to claim 2, characterized in that, The video memory resource pool also includes a system memory layer, which includes multiple memory blocks of a local host, and the local host is the host currently used by the server. Before detecting whether the first video memory resource is sufficient, the method further includes: Perform network health status checks according to the first preset cycle; If the heartbeat time of each of the remote nodes is greater than a preset time threshold, or the network connection of all the remote nodes is interrupted, or the second video memory resources are exhausted, then the network health status is determined to be a fault state, and the memory block that satisfies the video memory allocation request is selected from the local host as the target video memory block.

5. The video memory management method according to claim 1, characterized in that, The target video memory resource includes a target video memory block, and the first video memory layer includes multiple co-machine devices. The first video memory resource of each co-machine device includes at least one first video memory block. The process of selecting target video memory resources from the video memory resource pool based on hook functions includes: Based on the hook function, it is determined whether the local video memory resources are sufficient; If the local video memory resources are insufficient, then check whether the first video memory resources are sufficient. If the first video memory resources are sufficient, then the first video memory block that satisfies the video memory allocation request is selected from the first video memory layer as the target video memory block.

6. The video memory management method according to claim 1, characterized in that, The target video memory resource includes a target video memory block, and the local video memory layer includes at least one local video memory block; The process of selecting target video memory resources from the video memory resource pool based on hook functions includes: Based on the hook function, it is determined whether the local video memory resources are sufficient; If the local video memory resources are sufficient, a local video memory block that satisfies the video memory allocation request is selected from the local video memory layer as the target video memory block.

7. The video memory management method according to claim 1, characterized in that, The second video memory resource of the remote node includes a second video memory block; After allocating the target video memory resource to the application and returning the virtual address of the target video memory resource to the application, the method further includes: In response to the application's video memory access request, based on the hook function, it is detected whether the video memory block to be accessed by the application is the second video memory block; If the video memory block to be accessed by the application is the second video memory block, then it is detected whether the video memory block to be accessed can be hit in the local buffer; the local buffer includes multiple cached video memory blocks; If the memory block to be accessed fails to be hit in the local buffer and the current available cache capacity of the local buffer is insufficient, then based on the LRU algorithm, at least one of the cached memory blocks is released, and the data of the second memory block of the remote node is written into the local buffer to generate a cached memory block to be accessed. Determine the local cache address of the cache memory block to be accessed in the local buffer; The video memory access request is redirected to the local cache address.

8. A video memory management device, characterized in that, include: The resource selection module is used to respond to the application's video memory allocation request and select the target video memory resource from the video memory resource pool based on hook functions; The resource allocation module is used to allocate the target video memory resources to the application and return the virtual address of the target video memory resources to the application. The video memory resource pool includes a local video memory layer, a first video memory layer, and a second video memory layer. The local video memory layer includes local video memory resources of local devices. The first video memory layer includes first video memory resources of devices on the same machine. The second video memory layer includes second video memory resources of devices used by remote nodes. The local device is the device currently used by the server. The devices on the same machine are devices deployed on the same server as the local device. The target video memory resource is determined by the hook function from the local video memory resources, the first video memory resources, and the second video memory resources based on a hierarchical resource search strategy. Following the application's request for video memory access, the system also includes: According to the second preset period, historical access records are obtained; the historical access records include access information of each cached memory block within a preset time period, and the cached memory block is stored in a local buffer. Temporal locality analysis is performed on the historical access records to determine a first candidate set; the first candidate set includes at least one first candidate memory block; Spatial locality analysis is performed on the historical access records to determine a second candidate set; the second candidate set includes at least one second candidate memory block; Access pattern analysis is performed on the historical access records to determine a third candidate set; the third candidate set includes at least one third candidate memory block; Multiple target candidate memory blocks are selected from the first candidate set, the second candidate set, and the third candidate set, and the data of each target candidate memory block is written into the spare buffer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the video memory management method as described in any one of claims 1 to 7.

Images