Memory management method and apparatus, and computing device

Abstract

A memory management method and apparatus, and a computing device. The method is executed by a GMEM system. When receiving a physical memory range, a memory management unit operation function, and an accelerator device address of an accelerator device, the GMEM system configures a memory management unit operation function interface for the accelerator device, and a device where the GMEM system is located is enabled to use a driver of the accelerator device to manage a memory management unit and a physical memory of the accelerator device by means of the memory management unit operation function interface for the accelerator device, so as to read and write data. In the present application, the GMEM system extends a native memory management interface of an operating system into a heterogeneous memory management interface, supporting programming based on a unified virtual address space, and memory management unit operation function interfaces are configured for different types of accelerator devices, so that the device where the GMEM system is located can support the use of different types of accelerator devices, without the need to consider the compatibility of memory interfaces of different types of accelerator devices.

Description

Memory management method, device and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 28, 2023, with application number 202311095589.3 and application name “A memory management method, device and computing device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of artificial intelligence technology, and in particular to a memory management method, device and computing equipment. Background Art

[0003] With the development of artificial intelligence (AI), a single processor cannot meet the computing needs of fields such as AI. In related technologies, chip manufacturers can bring together multiple different types of processors, accelerators, coprocessors and other computing devices to form heterogeneous chips. Heterogeneous chips can use multiple computing devices to jointly perform computing tasks to meet the computing needs of fields such as AI. In the heterogeneous programming architecture of heterogeneous chips, two sets of virtual address spaces need to be distinguished, and each piece of data needs to maintain two different caches (buffers). For general data, users need to manually manage data migration. For data with complex structures such as graphs and trees, users need to perform complex data format conversions and then manually manage data migration.

[0004] In order to solve the problem of data migration in heterogeneous chips, the relevant technology provides a programming solution based on a unified virtual address space (UVAS), specifically: adding a third-party driver to the heterogeneous chip. The third-party driver can manage the specified memory and work with the user-mode unified memory application interface. For example, the driver of Nvidia's unified computing architecture (CUDA) and unified virtual memory (UVM) is the cudaMallocManaged interface provided by CUDA. Both the central processing unit (CPU) and the graphics processing unit (GPU) can directly access the virtual address requested by this interface. When the CPU or GPU accesses the virtual address, the UVM driver establishes a mapping relationship between the virtual address and the CPU memory or GPU memory by linking the GPU memory management (MM) and the Linux MM to ensure normal access by the CPU or GPU.

[0005] However, third-party drivers configured for each heterogeneous chip cannot be copied to other heterogeneous chips, resulting in high development costs. Adding a third-party driver to a heterogeneous chip requires a complex memory management system to manage the accelerator's local memory, leading to high code duplication within the heterogeneous chip and degraded performance.

[0006] Summary of the Invention

[0007] To address the aforementioned issues, embodiments of the present application provide a memory management method that extends the native memory interface of an operating system to an interface that supports heterogeneous memory management. This interface can support the simultaneous use of multiple types of accelerator chips in user processes without having to consider compatibility issues between the memory interfaces of various accelerator chips. Furthermore, the present application also provides a memory management device and computing device corresponding to the memory management method.

[0008] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0009] In a first aspect, the present application provides a memory management method, which is executed by a general memory management GMEM system, and the method includes: receiving device information of at least one mounted accelerator device, the device information including the physical memory address of the accelerator device, a memory management unit operation function and the accelerator device address; configuring a memory management unit operation function interface for the at least one loading device, the memory management unit operation function interface using the memory management unit operation function to use the driver of the accelerator device corresponding to the accelerator device address; and using the driver of the accelerator device to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

[0010] In this embodiment, after receiving the physical memory range, memory management unit operation function, and accelerator device address of the accelerator device, the GMEM system configures a memory management unit operation function interface for the accelerator device, allowing the device where the GMEM system is located to use the accelerator device's driver to manage the accelerator device's memory management unit and physical memory through the accelerator device memory management unit operation function interface to read and write data. The GMEM system of the present application can expand the native memory management interface of the operating system into a heterogeneous memory management interface, support programming of a unified virtual address space, and configure memory management unit operation function interfaces for various types of accelerator devices, so that the device where the GMEM system is located can support the use of multiple types of accelerator devices without having to consider the compatibility issues of the memory interfaces of various types of accelerator devices.

[0011] In one embodiment, the method further includes: receiving a non-uniform memory access NUMA identifier ID; in response to an accelerator device corresponding to the NUMA ID being in peer mode, sending a request instruction to the accelerator device corresponding to the NUMA ID, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

[0012] In this embodiment, after receiving the NUMA ID, the GMEM system determines that the accelerator device corresponding to the NUMA ID is in peer mode. It can then request a virtual address and have the accelerator device corresponding to the NUMA ID request the same virtual address. Based on the requested virtual address, the GMEM system can copy its own data to the memory of the accelerator device corresponding to the NUMA ID.

[0013] In one embodiment, the method further includes: in response to a page fault being triggered when writing data, detecting whether its own logical page table has its own mapping relationship, the page fault means that the virtual address cannot be mapped to the physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; in response to the fact that its own logical page table does not have its own mapping relationship, establishing its own mapping relationship, and saving its own mapping relationship in its own logical page table.

[0014] In this embodiment, when the GMEM system writes data and triggers a page fault, it checks its own logical page table to see if a mapping exists. If the GMEM system detects no mapping in its own logical page table, meaning the current virtual address is not mapped to physical memory, the GMEM system updates the logical page table and establishes a mapping between the requested virtual address and its own memory, allowing the GMEM system to continue writing input data.

[0015] In one embodiment, the method further includes: receiving page fault information of a first accelerator device, the page fault information indicating that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; querying its own logical page table based on the page fault information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address exists in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establishing the mapping relationship of the first accelerator device.

[0016] In this embodiment, when the accelerator device triggers a page fault, it generates page fault information and sends it to the GMEM system. After receiving the page fault information, the GMEM system checks whether its own logical page table has a mapping relationship with the accelerator device. When the GMEM system determines that its own logical page table has a mapping relationship with the accelerator device, it can call the memory management unit operation function interface of the accelerator device to establish the mapping relationship with the accelerator device, thereby copying its own data to the accelerator device's memory.

[0017] In one embodiment, the method further includes: in response to a page fault being triggered when reading data output by the first accelerator device, detecting whether there is a mapping relationship of the current virtual address in its own logical page table; in response to the existence of a mapping relationship of the current virtual address in its own logical page table, copying the data of the memory of the first accelerator device pointed to by the current virtual address to its own memory, and modifying the mapping relationship of the current virtual address pointing to the memory of the first accelerator device to a mapping relationship of the current virtual address pointing to its own memory.

[0018] In this embodiment, when the GMEM system reads the calculation results of the accelerator device, a page fault is triggered. The GMEM system checks its own logical page table to see if there is a mapping relationship for the current virtual address. If the GMEM system determines that a mapping relationship exists, it can map the current virtual address to the accelerator device memory and copy the accelerator device memory data mapped to the current virtual address to the device memory where the GMEM system is located, enabling the GMEM system to read the calculation results of the accelerator device.

[0019] In one embodiment, the method further includes: responding to a system call system call for memory copy hmemcpy; determining, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; and copying the data of the source device to the target device.

[0020] In this implementation, the GMEM system can use a memory copy system call, eliminating the need to specify the physical locations of the source and destination addresses. Data can be automatically retrieved from the logical page table, improving usability. When copying data between the GMEM system and the accelerator device, page prefetching semantics can be incorporated, eliminating the need for master node transfer and efficiently utilizing the bandwidth of the high-speed bus between devices.

[0021] In one embodiment, the method further includes: after receiving the NUMA ID, copying the data of the source device to the accelerator device corresponding to the NUMA ID.

[0022] In this implementation, users can also specify the target device, such as by sending a heterogeneous NUMA ID to the GMEM system. The GMEM system can directly and implicitly copy data from the source device to the target device using a memory copy system call. Compared to prefetch-then-copy or copy-then-prefetch methods, this saves the overhead of one memory copy.

[0023] In one embodiment, the method further includes: detecting whether the used memory in the memory pool is greater than a set threshold; in response to the used memory in the memory pool being greater than the set threshold, releasing part or all of the memory that is no longer used.

[0024] In this implementation, the GMEM system can request memory from the memory pool to run the accelerator device. The GMEM system counts the used memory in the memory pool to see if it exceeds a set threshold. If so, the GMEM system uses page discard semantics on the cached memory to release physical memory until the threshold is met or all cached memory is released.

[0025] In one embodiment, the method further includes: after receiving the NUMA ID and the virtual address, querying whether there is a mapping relationship of the virtual address in its own logical page table; in response to the existence of the mapping relationship of the virtual address in its own logical page table, copying the data of the virtual address mapped memory to the accelerator device memory corresponding to the NUMA ID, and establishing a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; in response to the absence of the mapping relationship of the virtual address in its own logical page table, establishing a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

[0026] In this embodiment, the GMEM system receives the NUMA ID and virtual address, detects the presence of a mapping relationship for the virtual address in its own logical page table, and directly copies the data mapped to the virtual address to the accelerator device corresponding to the NUMA ID. By prefetching data in advance, the GMEM system can reduce execution time and improve the performance of the GMEM system.

[0027] In one embodiment, the method further includes: receiving failure information from the first accelerator device, the failure information being a message that the first accelerator device failed to apply for memory; sending a copy instruction to the first accelerator device, the copy instruction instructing the first accelerator device to copy the data in the memory being used to its own memory, and to release the memory being used.

[0028] In this embodiment, when the accelerator device memory is insufficient, part of the memory in use can be swapped to the host memory to release the accelerator device memory, so as to realize the memory over-allocation function of the GMEM system supporting the accelerator device.

[0029] In the second aspect, the present application provides a memory management device, comprising: a transceiver unit for receiving device information of at least one mounted accelerator device, the device information including the physical memory address of the accelerator device, a memory management unit operation function and an accelerator device address; a processing unit for configuring a memory management unit operation function interface for the at least one loading device respectively, the memory management unit operation function interface using the memory management unit operation function to use the driver of the accelerator device corresponding to the accelerator device address; and using the driver of the accelerator device to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

[0030] In one embodiment, the transceiver unit is further used to receive a non-uniform memory access NUMA identifier ID; the processing unit is further used to send a request instruction to the accelerator device corresponding to the NUMA ID in response to the accelerator device corresponding to the NUMA ID being in peer mode, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

[0031] In one embodiment, the processing unit is also used to detect whether its own logical page table has its own mapping relationship in response to a page fault triggered when writing data, where the page fault refers to the inability to map the virtual address to the physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; and in response to the absence of its own mapping relationship in its own logical page table, establish its own mapping relationship and save its own mapping relationship in its own logical page table.

[0032] In one embodiment, the transceiver unit is further used to receive page fault information of the first accelerator device, wherein the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; the processing unit is further used to query its own logical page table based on the page fault information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to establish a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address exists in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish a mapping relationship of the first accelerator device.

[0033] In one embodiment, the processing unit is further used to, in response to triggering a page fault when reading data output by the first accelerator device, detect whether there is a mapping relationship of the current virtual address in its own logical page table; in response to the existence of the mapping relationship of the current virtual address in its own logical page table, copy the data of the memory of the first accelerator device pointed to by the current virtual address to its own memory, and modify the mapping relationship of the current virtual address pointing to the memory of the first accelerator device to the mapping relationship of the current virtual address pointing to its own memory.

[0034] In one embodiment, the processing unit is also used to respond to a system call that calls memory copy; based on its own logical page table, determine the source device corresponding to the source address of the system call and the target device corresponding to the target address of the system call; and copy the data of the source device to the target device.

[0035] In one embodiment, the processing unit is further configured to, after receiving the NUMA ID, copy the data of the source device to the accelerator device corresponding to the NUMA ID.

[0036] In one embodiment, the processing unit is further configured to detect whether the used memory in the memory pool is greater than a set threshold; in response to the used memory in the memory pool being greater than the set threshold, release part or all of the memory that is no longer used.

[0037] In one embodiment, the transceiver unit is further used to query whether there is a mapping relationship of the virtual address in its own logical page table after receiving the NUMA ID and the virtual address; the processing unit is further used to copy the data of the virtual address mapped memory to the accelerator device memory corresponding to the NUMA ID in response to the existence of the mapping relationship of the virtual address in its own logical page table, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; in response to the absence of the mapping relationship of the virtual address in its own logical page table, establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

[0038] In one embodiment, the transceiver unit is further used to receive failure information from the first accelerator device, where the failure information is a message that the first accelerator device failed to apply for memory; and send a copy instruction to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and release the memory being used.

[0039] In a third aspect, an embodiment of the present application provides a computing device comprising: at least one memory, a plurality of processors, the plurality of processors including a main processor, the main processor being used to execute instructions stored in the at least one memory, so that the computing device can be implemented as any of the possible embodiments of the first aspect.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementation embodiments of the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements each possible implementation embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0043] FIG1 is a schematic diagram of the structure of a CUDA in the related art;

[0044] FIG2 is a schematic diagram of the structure of a heterogeneous device provided in an embodiment of the present application;

[0045] FIG3 is a schematic diagram of the structure of a general memory management system provided in an embodiment of the present application;

[0046] FIG4 is a schematic diagram of the structure of the kernel state provided in an embodiment of the present application;

[0047] FIG5 is a schematic diagram of a process of applying for a unified virtual address in a GMEM system according to an embodiment of the present application;

[0048] FIG6 is a schematic diagram of a process for data transmission between a heterogeneous device and a mounted device based on a memory access consistency protocol according to an embodiment of the present application;

[0049] FIG7 is a schematic diagram of a process for using heterogeneous memory semantics in a GMEM system according to an embodiment of the present application;

[0050] FIG8 is a schematic diagram of a structure of data transmission between a host side and a slave side according to an embodiment of the present application;

[0051] FIG9 is a schematic diagram of a process of performing memory replication on a server according to an embodiment of the present application;

[0052] FIG10 is a schematic diagram of a process for applying for memory in a user state based on an AI framework according to an embodiment of the present application;

[0053] FIG11 is a schematic diagram of a process of performing memory super-resolution by an NPU according to an embodiment of the present application;

[0054] FIG12 is a schematic structural diagram of a memory management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0056] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0057] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0058] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0060] To facilitate understanding of the solutions of the embodiments of the present application, the technical terms involved in this document are first explained below.

[0061] Peer-to-peer mode refers to a system consisting of multiple nodes (peers) with independent operating systems. The multiple nodes include a master node and at least one peer node, with the master node managing the peer nodes. Furthermore, to assist the master node in managing the peer nodes, processes are created on each node. The master node's process is the primary process, while the peer node's process is the co-process.

[0062] The master-slave mode refers to a system consisting of a node with an independent operating system and at least one slave device (without an independent operating system).

[0063] A logical page table is a hardware-independent page table. It maps virtual addresses to physical addresses to support paging. Typically, a logical page table divides a process's virtual address space into fixed-size pages (typically 4KB / 2MB) and maps each page to a corresponding page in physical memory. The logical page table records the mapping between each virtual page and its corresponding physical page.

[0064] Memory consistency means that in a multi-core or multi-processor system, after one core or processor writes data to a virtual address, when another core or processor accesses the same virtual address, it can ensure that the data written by the former core or processor can be read.

[0065] Page migration refers to the process of moving a page from one physical memory location to another in an operating system. In a multi-core or multi-processor system, when migrating a page from a source node to a target node, the source node first unmaps the local virtual address, invalidating the page table cache (TLB). The target node then requests memory and copies the data from the source node's memory to the target node's memory. After the source node completes the copy operation, it can release the memory.

[0066] Page discard semantics refers to a strategy used in virtual memory management to determine when a page can be marked as no longer needed. When a page is no longer in use by an active process or thread, page discard semantics determine when it should be released. Page discard semantics enable efficient memory management, freeing up physical memory for use by other active processes when needed.

[0067] Data prefetching is a technique in which a processor or storage system retrieves data in advance to reduce data access latency. Data prefetching, triggered by a user action, pre-loads data into the cache or storage system to reduce latency during subsequent access.

[0068] Reverse mapping refers to the process of finding the corresponding virtual address based on a physical address. In operating system virtual memory management, reverse mapping is often used to perform reverse memory mapping. Conversely, forward mapping is the process of mapping a virtual address to a physical address. In other words, given a virtual address, it finds the corresponding physical memory address.

[0069] Figure 1 is a schematic diagram of the CUDA architecture in related art. As shown in Figure 1, CUDA includes an application programming interface (API), a UVM driver, a Linux MM, a GPU MM, a CPU address (ADDR), a CPU memory management unit (MMU), a GPU ADDR, and a GPU MMU.

[0070] CUDA calls a customized API (such as cudaMallocManaged) to request memory. The UVM driver first calls the Linux MM interface to request a virtual address, and then maintains the requested virtual address in the GPU MM. Writing the input data of the operator will trigger a page fault in the CPU. Among them, a page fault means that when the CPU writes data, it will access the virtual address, and the CPU's MMU will trigger an error in mapping the virtual address to the physical address. The CPU sends a page fault to the operator, allowing the operator to run and trigger a page fault in the GPU. At this time, the UVM driver needs to read the CPU's page table to obtain the CPU memory corresponding to the virtual address that triggered the page fault. The UVM driver copies the data in the CPU memory to the GPU memory to complete the page fault processing. After the operator finishes running, the memory is released.

[0071] In related technologies, UVM drivers can address the difficulty of GPU programming with the UVAS programming framework for CUDA. However, due to software stack limitations, UVM drivers cannot be reused for other types of accelerator chips. Furthermore, the entire solution involves complex interactions between the UVM driver, Linux MM, and GPU MM, resulting in redundant metadata and processing flows, and poor performance.

[0072] In order to address the defects existing in the related art, an embodiment of the present application provides a memory management method. After the operating system receives the physical memory range, memory management unit operation function and accelerator device address of the accelerator device, it configures a memory management unit operation function interface for the accelerator device, allowing the device where the GMEM system is located to use the accelerator device driver to manage the memory management unit and physical memory of the accelerator device through the accelerator device memory management unit operation function interface to read and write data. The operating system can expand the native memory management interface of the operating system into a heterogeneous memory management interface, support the programming of a unified virtual address space, and configure memory management unit operation function interfaces for various types of accelerator devices. This allows the device where the GMEM system is located to support the use of multiple types of accelerator devices without considering the compatibility issues of the memory interfaces of various types of accelerator devices.

[0073] FIG2 is a schematic diagram of a heterogeneous device according to an embodiment of the present invention. As shown in FIG2 , a heterogeneous device 200 includes an application 210 , an operating system 220 , a basic input and output system (BIOS) 230 , and hardware components 240 .

[0074] Application 210 is a computer program that performs one or more specific tasks. Installed on operating system 220, application 210 interacts with the user, creating and executing tasks upon receiving user commands. Each application 210 runs in an independent process and has its own address space.

[0075] The operating system 220 is a set of interrelated system software programs that manage and control computer operations, run hardware and software resources, and provide public services to organize user interactions. The operating system 220 schedules various resource modules of the heterogeneous device 200, including hardware and software devices, data, and information. Using a computer operating system can reduce the workload of manual resource allocation, reduce user intervention in computer operations, and significantly improve the computer's intelligent operating efficiency. The operating system 320 can be a Linux system, a real-time operating system (RTOS), or other systems.

[0076] BIOS 230 is an industry-standard firmware interface. It is a set of programs hardwired into a read-only memory (ROM) chip on the computer's motherboard. It stores the computer's most important basic input and output routines, the post-boot self-test program, and the system startup program. The primary function of BIOS 230 is to provide the lowest-level, most direct hardware configuration and control for the computer. BIOS 230 does not directly control hardware components 240, but rather provides an abstraction layer and directly controls them.

[0077] The hardware component 240 includes various hardware components of the heterogeneous device 200, such as an accelerator, a GPU, a CPU, a neural processing unit (NPU), etc. The hardware component 240 is used to support the normal operation of the heterogeneous device 200.

[0078] The operating system 220 kernel is the core component of the operating system 220, responsible for managing system resources and providing various services. The operating system 220 divides its operating memory into two parts: kernel space and user space. User space refers to the space where user program code runs. Kernel space refers to the space where kernel code runs. When a process runs in user space, the process is in "user mode." A process in user mode is also called a user mode process. When a process runs in kernel space, the process is in "kernel mode." A process in kernel mode is also called a kernel mode process.

[0079] In the design of operating system 220, user mode refers to a non-privileged execution state. The operating system 220 kernel prohibits code in this state from performing potentially dangerous operations, such as writing system configuration files, killing other users' processes, or restarting the system. Kernel mode refers to a privileged execution state. The operating system 220 kernel can perform any operation on code in this state.

[0080] FIG3 is a schematic diagram of the structure of a generalized memory management system provided in an embodiment of the present application. As shown in FIG3 , the generalized memory management (GMEM) system 300 includes a user state 310 and a kernel state 320. The user state 310 includes the native API of the AI ​​framework, applications, and the operating system 220.

[0081] The AI ​​Framework is a software toolset for developing, training, and deploying AI models. It provides developers with a range of tools and interfaces that make it easy to build and operate AI models. Common AI Frameworks include TensorFlow, PyTorch, Keras, MXNet, and MindSpore. In the embodiments of this application, both the AI ​​Framework and the application can support programming of accelerator devices.

[0082] An API is a set of functions, classes, or methods provided by the operating system 220 to applications for interacting and operating with the operating system 220. The API defines the communication protocols and interface specifications between applications and the operating system, enabling applications to leverage the functionality of the operating system 220 to complete various tasks. In the embodiments of the present application, the AI ​​Framework or applications can access and utilize various functions provided by the operating system 220, such as file system access, process management, and network communication, by calling the native API of the operating system 220.

[0083] The native API of the operating system 220 may be an interface such as mmap or brk. The AI ​​Framework or an application may use the interface such as mmap or brk to request memory for the accelerator.

[0084] Kernel state 320 includes a heterogeneous memory management unit (HMU) and an accelerator driver. The HMU is responsible for virtual address management and physical memory management for devices such as the CPU, GPU, NPU, and accelerators. The accelerator driver is responsible for interacting with accelerators such as the GPU and NPU. The accelerator driver adds accelerator-local memory management functions, such as memory management unit operations (MMU ops), for accessing the kernel state 320 memory management system.

[0085] As shown in Figure 4, the native API of the operating system 220 can expand the native memory management interface of the operating system 220 to a GMEM interface, such as mmap, brk and other interfaces. When the operating system 220 is in peer mode, it first applies for a virtual address in the master process, and then applies for the same virtual address in the co-process of the peer node. When the operating system 220 is in master-slave mode, it supports heterogeneous devices 200 to directly use the virtual addresses applied for by the host's mmap, brk and other interfaces. The newly added GMEM interface (such as the hmadvise interface) can apply a variety of heterogeneous memory semantics to support page discard semantics, data prefetching and other functions, which can add new functions to the GMEM system 300 and improve the performance of the GMEM system 300.

[0086] The accelerator driver may be internally provided with an MMU ops program to provide a registration interface for MMU operations (including page tables, TLBs, etc.), allowing various types of accelerator drivers to implement MMU ops. The MMU ops program allows various types of accelerator drivers to register with the operating system 220 and be managed uniformly by the operating system 220.

[0087] A logical page table is added to the GMEM system 300, and the logical page table includes multiple mapping relationships, each of which represents a mapping between a virtual address and a physical address. In an embodiment of the present application, the heterogeneous memory management unit can maintain a mapping relationship between the same virtual address and any physical address (located in different heterogeneous devices) based on the logical page table. When the same virtual address is accessed by different heterogeneous devices 200, the heterogeneous memory management unit can use the registered MMU ops interface with the help of the logical page table to transparently coordinate multiple MMU page tables and complete data migration or data replication to ensure memory access consistency. The operating system 220 can dynamically switch the underlying memory access consistency protocol and flexibly apply it in different application scenarios to achieve optimal performance.

[0088] During initialization, the GMEM system 300 inputs device information, including the physical memory, MMU ops, device policy (e.g., peer-to-peer mode, master-slave mode), and device address, of the external device into the heterogeneous device 200 and registers it with the heterogeneous memory management unit in kernel state 320. After receiving the device information for each heterogeneous device, the heterogeneous memory management unit assigns a non-uniform memory access (NUMA) node identifier (ID) to each external device and downloads the driver for each external device. This allows each external device to access the unified heterogeneous memory management architecture of the operating system of the heterogeneous device 200, enabling the heterogeneous device 200 to utilize the corresponding device driver through MMU ops and manage physical memory using the device driver to read and write data.

[0089] FIG5 is a schematic diagram of a process for a GMEM system to apply for a unified virtual address in an embodiment of the present application. As shown in FIG5 , the process for the GMEM system 300 to apply for a unified virtual address is as follows:

[0090] S501: The user state 310 mounts a device involved in computing and obtains the NUMA ID of the mounted device. The mounted device refers to a device that sends its own device information to the heterogeneous device 200.

[0091] S502: When the kernel state 320 determines that the device corresponding to the NUMA ID is in peer mode, a co-process is created and bound.

[0092] S503, the user state 310 calls the mmap interface to apply for memory.

[0093] S504 , the kernel state 320 applies for a virtual address in response to the request of the mmap interface for memory application.

[0094] S505, kernel state 320 traverses mounted devices.

[0095] S506: The kernel state 320 determines whether the mounted device is in peer-to-peer mode. In one case, if the kernel state 320 determines that the mounted device is in peer-to-peer mode, the process proceeds to S507. In another case, if the kernel state 320 determines that the mounted device is not in peer-to-peer mode, i.e., in master-slave mode, the process ends directly.

[0096] S507 , when the kernel state 320 determines that the mounted device is in peer mode, the same virtual address is applied for in the co-process of the device in peer mode.

[0097] S508: The kernel state 320 detects whether all devices in peer mode have successfully applied for the same virtual address. In one case, if the kernel state 320 determines that all devices in peer mode have successfully applied for the same virtual address, the process ends directly. In another case, if the kernel state 320 determines that all devices in peer mode have failed to apply for the same virtual address, the process proceeds to S504.

[0098] In the embodiment of the present application, when a user process starts, user state 310 will call the mmap interface to apply for a section of virtual memory and traverse the mounted devices of the current process. For mounted devices in master-slave mode, user state 310 does not need additional operations. For mounted devices in peer-to-peer mode, user state 310 can first create a co-process on the peer device and bind the co-process to the master process. Then, kernel state 320 applies for the same virtual address on the co-process. If kernel state 320 fails to apply for a device, it will continue to apply for other virtual addresses until all devices have applied for the same virtual address.

[0099] FIG6 is a flow chart of data transmission between a heterogeneous device and a mounted device based on a memory access consistency protocol according to an embodiment of the present application. As shown in FIG6 , the process of data transmission between a heterogeneous device 200 and a mounted device based on a memory access consistency protocol is as follows:

[0100] S601: A page fault is triggered when a device accesses a virtual address. At this point, the heterogeneous device determines that the virtual address is not mapped to a physical address, resulting in the device being unable to access the virtual address. This is called a "page fault." To distinguish heterogeneous devices in this state from normal heterogeneous devices, a heterogeneous device in this state can be defined as a "page fault device."

[0101] S602 , the page fault device sends page fault information to the GMEM system 300 .

[0102] S603 , the GMEM system 300 queries the logical page table based on the page fault information.

[0103] At step S604, the GMEM system 300 detects whether the logical page table contains a mapping relationship for a page-missing device. In one case, if the GMEM system 300 detects that the logical page table contains a mapping relationship for a page-missing device, the process proceeds to step S605. In another case, if the GMEM system 300 detects that the logical page table does not contain a mapping relationship for a page-missing device, the process proceeds to step S611.

[0104] At step S605, the GMEM system 300 detects whether the page fault is caused by a permissions issue. In one case, if the GMEM system 300 detects that the page fault is caused by a permissions issue, the process proceeds to step S606. In another case, if the GMEM system 300 detects that the page fault is not caused by a permissions issue, the process proceeds to step S609. The permissions issue refers to the permissions of the MMU page table, i.e., the MMU page table indicates a read operation, or the MMU page table indicates a write operation.

[0105] In step S606, the GMEM system 300 detects whether the permission can be modified. In one case, the GMEM system 300 detects that the permission can be modified, and the process S607 is executed. In another case, the heterogeneous device 200 detects that the permission cannot be modified, and the process S608 is executed.

[0106] S607: The GMEM system 300 modifies the permissions. That is, the page fault is caused by a permissions issue, and the GMEM system 300 modifies the mapped permissions to meet the requirements of the page fault device.

[0107] S608: The GMEM system 300 returns an error.

[0108] S609 , the GMEM system 300 calls the MMU ops registered by the current mapping device to release the mapping relationship in the logical page table.

[0109] S610 , the GMEM system 300 copies the data of the current mapped device memory to the memory of the page-missing device.

[0110] S611 , the GMEM system 300 calls the MMU ops registered by the page fault device to establish or modify the mapping relationship in the logical page table.

[0111] S612 , the GMEM system 300 updates the logical page table.

[0112] In an embodiment of the present application, when a page fault device accesses a virtual address, a page fault is triggered, and the page fault information is forwarded to the GMEM system 300. After receiving the page fault information, the GMEM system 300 can query the logical page table based on the page fault information. If the GMEM system 300 finds that its own logical page table does not have a mapping relationship for the page fault device, it can call the MMU ops registered by the page fault device to establish a mapping relationship. After the GMEM system 300 completes the establishment of the mapping relationship, it can update the logical page table and establish a mapping relationship for the page fault device in the logical page table. If the GMEM system 300 finds that its own logical page table has a mapping relationship for the page fault device, it can detect whether the page fault is caused by a permission issue.

[0113] If the GMEM system 300 detects that the page fault is caused by a permission issue, it can detect whether the permission can be modified. If the GMEM system 300 detects that the permission can be modified, it modifies the permission and then calls the MMU ops registered by the page fault device to modify the mapping relationship. If the GMEM system 300 detects that the permission cannot be modified, it directly returns an error. If the GMEM system 300 detects that the page fault is not caused by a permission issue, it can obtain the current mapping device through the logical page table and then call the MMU ops registered by the current mapping device to release the mapping relationship between the GMEM system 300 and the current mapping device. The GMEM system 300 can copy the data to the memory of the page fault device and then call the MMU ops registered by the page fault device to establish or modify the mapping relationship. The heterogeneous device 200 updates the logical page table and establishes a mapping relationship pointing to the page fault device in the logical page table.

[0114] FIG7 is a schematic diagram of a process for a GMEM system to use heterogeneous memory semantics in an embodiment of the present application. As shown in FIG7 , the process for a GMEM system 300 to use heterogeneous memory semantics is as follows:

[0115] S701 , the GMEM system 300 initializes the user process in the user state 310 .

[0116] S702 , the user state 310 obtains the NUMA ID of the device corresponding to the virtual address.

[0117] S703, the user state 310 calls the GMEM interface to pass the device information of each mounted device to the kernel state 320. The device information includes NUMA ID, requested virtual address, used semantic parameters (such as page discard semantics, data prefetching) and other information.

[0118] S704: Kernel state 320 detects whether a NUMA ID has been received. In one case, kernel state 320 receives a NUMA ID and executes process S705. In another case, kernel state 320 does not receive a NUMA ID and executes process S706.

[0119] S705 , the kernel state 320 calls the MMU ops registered by the device corresponding to the NUMA ID to deliver heterogeneous memory semantics.

[0120] S706 , the kernel state 320 traverses the mounted devices and calls the MMU ops registered by the mounted devices to deliver heterogeneous memory semantics.

[0121] In the embodiment of the present application, the GMEM system 300, in the user process of user state 310, can obtain the NUMA ID of the corresponding device based on the virtual address, and then pass device information such as the NUMA ID, virtual address, and used semantic parameters to kernel state 320 through the GMEM interface. If kernel state 320 receives the NUMA ID, it can call the MMU ops registered for the specified device to deliver heterogeneous memory semantics. If kernel state 320 does not receive the NUMA ID, it can traverse the mounted devices and call the MMU ops registered for the mounted devices to deliver heterogeneous memory semantics.

[0122] To facilitate understanding, at least one application scenario of an embodiment of the present application is first introduced below.

[0123] FIG8 is a schematic diagram of a structure for data transmission between a host side and a co-processor side, provided in an embodiment of the present application. As shown in FIG8 , the host side can be a server, and the co-processor side can be an NPU. The server's operating system can extend the original memory management to heterogeneous memory management. Heterogeneous memory management can implement a memory access consistency protocol through a logical page table to support the UVAS programming framework in peer mode and improve the performance of NPU operators.

[0124] In the server, user state includes the main process and the mmap / brk interface, while kernel state includes the heterogeneous memory management unit (HMU). The HMU includes a logical page table, a memory coherence protocol, and an MMU ops interface. The memory coherence protocol enables interaction between the CPU MMU ops program and the NPU MMU ops program through the MMU ops interface and the NPU driver.

[0125] The NPU includes the NPU driver, which in turn includes the NPU MMU manager. A message channel and a direct memory access (DMA) channel connect the server and the NPU. The message channel connects the NPU MMU ops program and the NPU MMU manager, used to transmit messages at the software layer. The DMA channel connects the host memory and the NPU memory, used to migrate data at the hardware layer.

[0126] During server initialization, the server can start the main process and establish a communication connection between the server and the NPU to enable the NPU to be mounted on the main process. The main process can notify the NPU to create a co-process and bind the co-process to the main process. When the server prepares the memory required for the NPU operator to run, the server's user state can call the mmap interface to request memory. After the mmap interface falls into kernel state, the kernel state can first request a virtual address and then notify the NPU co-process to request the same virtual address. If the kernel state fails to request a virtual address, the main process requests another virtual address again, and requests the same virtual address through the co-process until the request is successful. NPU operators are symbols, instructions, or functions that run specific operations in the NPU.

[0127] When the server's main process prepares input data for the NPU operator, after the main process applies for a virtual address, it can write the NPU operator's input data to the requested virtual address. When the NPU operator writes the input data, it triggers a page fault on the server. When handling a page fault, the server's main process checks whether there is a mapping relationship in its own logical page table. If the main process detects that there is no mapping relationship in its own logical page table, that is, the current virtual address is not mapped to physical memory. At this point, the main process can directly execute the server's page fault processing flow, update the logical page table, and establish a mapping relationship in the logical page table from the requested virtual address to the server's memory. After completing the page fault processing, the main process allows the NPU operator to continue writing input data.

[0128] The server's main process ensures that during the operation of the NPU operator based on the memory access consistency protocol, the NPU operator will trigger a page fault in the NPU when it accesses input data for the first time when running on the NPU. The NPU generates page fault information and transmits it to the server through the NPU driver, allowing the server to detect whether its own logical page table has a mapping relationship with the current virtual address. The main process detects whether its own logical page table has a mapping relationship with the current virtual address and determines the host memory mapped by the current virtual address. The main process copies the host memory page mapped by the current virtual address to the NPU memory. The main process updates the logical page table and modifies the mapping relationship in the logical page table from the current virtual address pointing to the server's host memory to the current virtual address pointing to the NPU memory of the NPU.

[0129] When an NPU operator runs on the NPU and outputs computational results, it triggers a page fault in the NPU. The NPU generates page fault information and transmits it to the server through the NPU driver. The server then checks whether its own logical page table contains the mapping for the current virtual address. Typically, if the server detects that its own logical page table does not contain the mapping for the current virtual address, it can directly call the NPU MMU ops program registered with the NPU to establish the mapping. The server then updates the logical page table and establishes a mapping to the NPU memory in the logical page table.

[0130] When the server reads the calculation results generated by the NPU operator in the NPU, the main process will trigger a page fault on the server when reading the calculation results. The main process will detect whether there is a mapping relationship of the current virtual address in its own logical page table. The main process detects that there is a mapping relationship of the current virtual address in its own logical page table, that is, the current virtual address is mapped to the NPU memory. At this time, the main process determines that the current virtual address is mapped to the NPU memory, and can map the NPU memory mapped by the current virtual address to the host memory corresponding to the server, and copy the page of the NPU memory mapped by the current virtual address to the corresponding host memory. The main process updates the logical page table and modifies the mapping relationship of the current virtual address pointing to the NPU memory in the logical page table to the mapping relationship of the current virtual address pointing to the host memory of the server.

[0131] Before the NPU operator accesses the input data for the first time, the main process can call the data prefetch (prefetch) provided by the hmadvise interface to prefetch the input data. When the main process calls prefetch, it needs to pass in the NUMA ID of the NPU. Prefetch will evenly divide the input data so that the divided input data can be processed in parallel by multiple threads. The main process queries the logical page table, and after detecting that there is a mapping relationship between the virtual address of the input data and the server memory, it asks the server to copy the page to the NPU. The NPU updates the logical page table and changes the mapping relationship from the current virtual address in the logical page table to the server's host memory to the current virtual address to the NPU memory. When the NPU operator runs on the NPU, it can directly access the input data without triggering a page fault on the NPU, reducing the execution time of the operator. Under a unified heterogeneous memory management framework, the server and NPU improve the performance of the NPU operator by prefetching data in advance.

[0132] FIG9 is a schematic diagram of a process flow of performing memory replication on a server according to an embodiment of the present application. As shown in FIG9 , the process flow of performing memory replication on a server is as follows:

[0133] S901: The kernel state of the server invokes a system call for memory copy (hmemcpy).

[0134] A system call is a request from a program running in user space to the operating system for a service that requires higher privileges. A system call provides an interface between user programs and the operating system to implement memory copy functionality.

[0135] S902: The server executes multi-threaded operations in parallel according to page granularity. Page granularity refers to the division of the virtual address space into fixed-size pages, which are generally 4KB or 2MB in size.

[0136] S903: The server queries the logical page table to obtain the source device mapped by the source address.

[0137] When a server performs a memory copy, the system call typically indicates a source address and a destination address, copying data from the device corresponding to the source address to the device corresponding to the destination address. Both the source and destination addresses are virtual addresses. The logical page table includes a mapping relationship that can map virtual addresses to physical addresses. Physical addresses can be physical addresses of other devices. The server can query the logical page table to find the device corresponding to the physical address mapped to the source address and determine it as the source device.

[0138] S904: The server detects whether the NUMA ID is received. In one case, the server detects the NUMA ID and the process is executed in S905. In another case, the server does not detect the NUMA ID and the process is executed in S908.

[0139] S905: The server queries the logical page table to obtain the device currently mapped by the NUMA ID.

[0140] At step S906, the server determines whether the NUMA ID of the currently mapped device is the same as the NUMA ID of the target device. In one case, the server determines that the NUMA ID of the currently mapped device is different from the NUMA ID of the target device, and then proceeds to step S907. In another case, the server determines that the NUMA ID of the currently mapped device is the same as the NUMA ID of the target device, and then proceeds to step S909.

[0141] S907: The server calls the MMU ops registered by the currently mapped device to cancel the mapping relationship, invalidate the TLB, and release the physical page.

[0142] S908 , the server queries the logical page table to obtain the device mapped by the target address, and sets the device mapped by the target address as the target device.

[0143] S909: The server copies the page of the source device to the target device.

[0144] S910: The server calls the MMU ops registered by the target device to establish a mapping relationship.

[0145] S911: The server updates the logical page table and points the target address to the target device.

[0146] In an embodiment of the present application, the server and the NPU implement parallel memory replication in any direction under a unified heterogeneous memory management framework. The server's kernel state can use the system call for memory replication, without specifying the physical location of the source and target addresses, and automatically obtain data according to the logical page table to improve usability. When copying data between the server and the NPU, page prefetch semantics can be combined, without the need for transfer through the master node, and the bandwidth of the high-speed bus between devices can be efficiently utilized.

[0147] When the server uses the memory copy system call to copy data across nodes, it does not need to explicitly specify the physical location of the source address and the target address. The server's kernel state can obtain the specific physical location based on the logical page table and directly complete the data copy. In addition, the user can also specify the target device, such as sending it to the server through a heterogeneous NUMA ID. The server can directly and implicitly complete the data copy from the source device to the target device when using the memory copy system call. Compared with the method of prefetching and then copying (without specifying the NUMA ID) or copying and then prefetching (without specifying the NUMA ID), the overhead of one memory copy can be saved. When the server uses the memory copy system call, the data is parallelized according to the memory page granularity to maximize the use of the bus bandwidth. In addition, there is no need to transfer through the master node, and the high-speed bus between devices can be fully utilized.

[0148] Figure 10 is a schematic diagram of a user state process for applying for memory based on an AI framework in an embodiment of the present application. As shown in Figure 10, the user state uses MindSpore as an example of the AI ​​framework, and the specific process for applying for memory is as follows:

[0149] S1001: MindSpore starts an asynchronous cache release thread to release the cache periodically.

[0150] S1002: MindSpore instructs the NPU operator to request memory from the memory pool.

[0151] In step S1003, MindSpore detects whether the used memory in the memory pool is greater than a set threshold. In one case, if MindSpore detects that the used memory in the memory pool is greater than the set threshold, step S1004 is executed. In another case, if MindSpore detects that the used memory in the memory pool is less than or equal to the set threshold, step S1007 is executed.

[0152] S1004: MindSpore uses page discard semantics for some used memory to release physical memory.

[0153] In step S1005, MindSpore detects whether the used memory after releasing part of the cache is greater than a set threshold. In one case, if MindSpore detects that the used memory after releasing part of the cache is greater than the set threshold, step S1007 is executed. In another case, if MindSpore detects that the used memory after releasing part of the cache is less than or equal to the set threshold, step S1006 is executed.

[0154] S1006: MindSpore detects whether all used memory has been released. In one case, if MindSpore detects that all used memory has been released, the process proceeds to step S1007. In another case, if MindSpore detects that all used memory has not been released, the process proceeds to step S1004.

[0155] S1007: MindSpore executes the NPU operator.

[0156] S1008: After the NPU operator finishes running, MindSpore releases the used memory.

[0157] S1009: MindSpore places the released memory in the memory pool.

[0158] In an embodiment of the present application, the server and NPU can implement memory management with high memory utilization based on the MindSpore framework, UVAS programming framework, and page discard semantics provided by the GMEM interface. Due to the static mapping between virtual addresses and physical memory, when MindSpore applies for virtual addresses through the NPU software stack, MindSpore can only use a virtual address space of equal size to the NPU physical memory to achieve secondary management. When MindSpore generates memory fragmentation, the lowest physical memory utilization may be only 30%. Therefore, the GMEM system removes the restrictions of the static mapping between virtual addresses and physical memory, eliminates the memory fragmentation generated during the MindSpore memory management process, and improves memory utilization to 99%+.

[0159] Before MindSpore dispatches NPU operators, it must request memory from the memory pool for NPU operator execution. Because the memory pool is implemented using UVAS and is not limited by the NPU's physical memory size, virtual address requests are always successful. MindSpore then counts whether the used memory exceeds a set threshold. If so, MindSpore uses page discard semantics on cached freed memory to release physical memory until the threshold is met or all cached memory is freed. If the used memory is less than or equal to the set threshold, the NPU operator is executed. When the NPU operator completes execution, the requested memory is released and cached in the memory pool, where it can be reused or released. Cached memory is periodically and asynchronously released using page discard semantics.

[0160] In an embodiment of the present application, the NPU driver of the NPU can set up an NPU memory recovery unit. The NPU memory recovery unit can borrow the server's memory when the memory is insufficient to realize the super-resolution function of the NPU. When the NPU memory recovery unit needs to reclaim the NPU physical memory, it can extract the NPU physical page from the linked list formed by connecting the currently used NPU physical pages in series according to the elimination strategy. Then, the NPU memory recovery unit first applies for the physical page on the server according to the page granularity, and then completes the page migration to the server for the selected NPU physical page, and updates the logical page table to point to the server memory. The NPU memory recovery unit can temporarily store part of the NPU physical memory data in the server memory, and release the NPU physical memory for the NPU operator to run and use. When the swapped out data is accessed again by the NPU operator, it will trigger a page fault of the NPU.

[0161] FIG11 is a schematic diagram of a process for NPU to perform memory super-resolution according to an embodiment of the present application. As shown in FIG11 , the process for NPU to perform memory super-resolution is as follows:

[0162] S1101, when the NPU operator runs on the NPU, a page fault is triggered.

[0163] S1102: The NPU operator applies for NPU physical memory.

[0164] S1103: The NPU operator detects whether the memory application is successful. In one case, if the NPU operator detects that the memory application is unsuccessful, the process S1104 is executed. In another case, if the NPU operator detects that the memory application is successful, the process S1105 is executed.

[0165] S1104: The NPU operator synchronously reclaims the NPU memory.

[0166] S1105: The NPU operator detects whether the remaining NPU memory is greater than a set threshold. In one case, if the NPU operator detects that the remaining NPU memory is less than or equal to the set threshold, process S1106 is executed. In another case, if the NPU operator detects that the remaining NPU memory is greater than the set threshold, process S1107 is executed.

[0167] S1106: The NPU operator wakes up the asynchronous thread and reclaims the NPU memory.

[0168] S1107: The NPU operator records the requested physical page in a linked list.

[0169] S1108: The NPU operator processes a page fault.

[0170] In this embodiment of the present application, when an NPU operator triggers a page fault, it is necessary to apply for NPU physical memory. If the NPU operator's application fails, the NPU memory is reclaimed. If the NPU operator's application succeeds, it is determined whether the current NPU remaining memory is greater than a set threshold. If it is less than the threshold, it is necessary to wake up the asynchronous thread and reclaim the NPU operator. Finally, the NPU operator records the requested physical page in a linked list and continues to handle the page fault.

[0171] FIG12 is a schematic diagram of the structure of a memory management device provided in an embodiment of the present application. As shown in FIG12 , the memory management device 1200 can be divided into a transceiver unit 1210 and a processing unit 1220 according to the execution function. The specific implementation process of the memory management device 1200 is as follows:

[0172] The transceiver unit 1210 is configured to receive device information from at least one mounted accelerator device. This device information includes the accelerator device's physical memory address, memory management unit operation function, and the accelerator device address. The processing unit 1220 is configured to configure a memory management unit operation function interface for each of the at least one mounted devices. The memory management unit operation function interface utilizes the memory management unit operation function to access the driver of the accelerator device corresponding to the accelerator device address. The processing unit 1220 is further configured to utilize the accelerator device driver to manage the physical memory corresponding to the accelerator device's physical memory address to read and write data.

[0173] In one embodiment, the transceiver unit 1210 is further configured to receive a non-uniform memory access NUMA identifier ID. The processing unit 1220 is further configured to, in response to the accelerator device corresponding to the NUMA ID being in peer mode, send a request instruction to the accelerator device corresponding to the NUMA ID. The request instruction instructs the accelerator device corresponding to the NUMA ID to apply for the first virtual address. The first virtual address is the virtual address applied for by the accelerator device itself.

[0174] In one embodiment, the processing unit 1220 is further configured to detect whether its own mapping relationship exists in its own logical page table in response to a page fault being triggered when writing data. A page fault occurs when a virtual address cannot be mapped to a physical address. A mapping relationship refers to the relationship between a virtual address and a physical address. The processing unit 1220 is further configured to establish its own mapping relationship in response to the absence of its own mapping relationship in its own logical page table, and store its own mapping relationship in its own logical page table.

[0175] In one embodiment, the transceiver unit 1210 is further used to receive page fault information of the first accelerator device. The page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address. At least one accelerator device includes the first accelerator device. The processing unit 1220 is further used to query its own logical page table based on the page fault information whether there is a mapping relationship for the virtual address accessed by the first accelerator device. The processing unit 1220 is further used to call the memory management unit operation function interface of the first accelerator device to establish a mapping relationship for the virtual address accessed by the first accelerator device in response to the fact that there is a mapping relationship for the virtual address accessed by the first accelerator device in its own logical page table. The processing unit 1220 is further used to call the memory management unit operation function interface of the first accelerator device to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device in response to the fact that there is a mapping relationship for the virtual address accessed by the first accelerator device in its own logical page table, and establish a mapping relationship for the first accelerator device.

[0176] In one embodiment, the processing unit 1220 is further configured to, in response to a page fault being triggered when reading data output by the first accelerator device, detect whether a mapping relationship for the current virtual address exists in its own logical page table. The processing unit 1220 is further configured to, in response to a mapping relationship for the current virtual address existing in its own logical page table, copy data pointed to by the current virtual address in the memory of the first accelerator device to its own memory, and modify the mapping relationship from the current virtual address pointing to the memory of the first accelerator device to a mapping relationship from the current virtual address pointing to the memory of the own memory.

[0177] In one embodiment, processing unit 1220 is further configured to respond to a system call invoking a memory copy. Processing unit 1220 is further configured to determine, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call. Processing unit 1220 is further configured to copy data from the source device to the target device.

[0178] In one embodiment, the processing unit 1220 is further configured to copy the data of the source device to the accelerator device corresponding to the NUMA ID after receiving the NUMA ID.

[0179] In one embodiment, the processing unit 1220 is further configured to detect whether the used memory in the memory pool is greater than a set threshold. In response to the used memory in the memory pool being greater than the set threshold, the processing unit 1220 is further configured to release part or all of the unused memory.

[0180] In one embodiment, the transceiver unit 1210 is further configured to, after receiving the NUMA ID and the virtual address, query whether a mapping relationship for the virtual address exists in its own logical page table. The processing unit 1220 is further configured to, in response to the existence of a mapping relationship for the virtual address in its own logical page table, copy the data of the virtual address-mapped memory to the accelerator device memory corresponding to the NUMA ID, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID. The processing unit 1220 is further configured to, in response to the absence of a mapping relationship for the virtual address in its own logical page table, establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

[0181] In one embodiment, the transceiver unit 1210 is further configured to receive failure information from the first accelerator device. The failure information is a message indicating that the first accelerator device failed to apply for memory. The transceiver unit 1210 is further configured to send a copy instruction to the first accelerator device. The copy instruction instructs the first accelerator device to copy the data in the currently used memory to its own memory and to release the currently used memory.

[0182] A computing device is also provided in an embodiment of the present application. The computing device includes at least one memory and at least one processor. The at least one processor can execute the technical solutions shown in Figures 2-11 and the corresponding protections above, so that the computing device has the technical effects of the technical solutions protected above.

[0183] In an embodiment of the present application, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the memory management method described in any one of Figures 2 to 11 and the corresponding description content.

[0184] A computer program product is also provided in an embodiment of the present application. The computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the memory management method described in any one of Figures 2 to 11 and the corresponding description content.

[0185] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0186] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0187] In the above embodiment, the memory management device 1200 in Figure 12 can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0188] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0189] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0192] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A memory management method, characterized in that: The method is performed by a general memory management GMEM system, and the method comprises: Receive device information of at least one mounted accelerator device, the device information including a physical memory address of the accelerator device, a memory management unit operation function, and an accelerator device address; A memory management unit operation function interface is configured for each of the at least one loading devices, wherein the memory management unit operation function interface uses the driver of the accelerator device corresponding to the accelerator device address by using the memory management unit operation function; The driver of the accelerator device is used to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

2. The method according to claim 1, characterized in that The method further comprises: Receive the non-uniform memory access NUMA identifier ID; In response to the accelerator device corresponding to the NUMA ID being in peer mode, a request instruction is sent to the accelerator device corresponding to the NUMA ID, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to a page fault being triggered when writing data, detecting whether its own logical page table has its own mapping relationship, wherein the page fault refers to the inability to map a virtual address to a physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; In response to the fact that the own logical page table does not have its own mapping relationship, the own mapping relationship is established, and the own mapping relationship is stored in the own logical page table.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: receiving page fault information of a first accelerator device, wherein the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; querying its own logical page table based on the missing page information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; In response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish the mapping relationship of the virtual address accessed by the first accelerator device; In response to the existence of the mapping relationship of the virtual address in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to triggering a page fault when reading data output by the first accelerator device, detecting whether its own logical page table has a mapping relationship of the current virtual address; In response to the existence of a mapping relationship of the current virtual address in its own logical page table, the data of the memory of the first accelerator device pointed to by the current virtual address is copied to its own memory, and the mapping relationship of the current virtual address pointing to the memory of the first accelerator device is modified to a mapping relationship of the current virtual address pointing to its own memory.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to a system call for hmemcpy; Determine, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; The data of the source device is copied to the target device.

7. The method according to claim 6, characterized in that The method further comprises: After receiving the NUMA ID, the data of the source device is copied to the accelerator device corresponding to the NUMA ID.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Check whether the used memory in the memory pool is greater than the set threshold; In response to the used memory in the memory pool being greater than a set threshold, part or all of the memory that is no longer used is released.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: After receiving the NUMA ID and the virtual address, query whether there is a mapping relationship of the virtual address in its own logical page table; In response to the existence of the mapping relationship of the virtual address in its own logical page table, copy the data of the virtual address mapping memory to the accelerator device memory corresponding to the NUMA ID, and establish the mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; In response to the fact that the mapping relationship of the virtual address does not exist in its own logical page table, a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID is established through a memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: receiving failure information of the first accelerator device, where the failure information is a message that the first accelerator device fails to apply for memory; A copy instruction is sent to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and to release the memory being used.

11. A memory management device, characterized in that: include: A transceiver unit, configured to receive device information of at least one mounted accelerator device, wherein the device information includes a physical memory address of the accelerator device, a memory management unit operation function, and an accelerator device address; A processing unit, configured to respectively configure a memory management unit operation function interface for the at least one loading device, wherein the memory management unit operation function interface uses a driver of the accelerator device corresponding to the accelerator device address by using the memory management unit operation function; The driver of the accelerator device is used to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

12. The device according to claim 11, characterized in that The transceiver unit is further used to receive a non-uniform memory access NUMA identifier ID; The processing unit is further used to send a request instruction to the accelerator device corresponding to the NUMA ID in response to the accelerator device corresponding to the NUMA ID being in peer mode, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

13. The device according to claim 11 or 12, characterized in that The processing unit is further configured to detect whether its own logical page table has its own mapping relationship in response to triggering a page fault when writing data, wherein the page fault refers to the inability to map a virtual address to a physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; as well as In response to the fact that the own logical page table does not have its own mapping relationship, the own mapping relationship is established, and the own mapping relationship is stored in the own logical page table.

14. The device according to any one of claims 11 to 13, characterized in that: The transceiver unit is further used to receive page fault information of a first accelerator device, wherein the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; The processing unit is further configured to query its own logical page table based on the missing page information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; In response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish the mapping relationship of the virtual address accessed by the first accelerator device; In response to the existence of the mapping relationship of the virtual address in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

15. The device according to any one of claims 11 to 14, characterized in that: The processing unit is further configured to trigger a page fault in response to reading data output by the first accelerator device, and detect whether its own logical page table has a mapping relationship of the current virtual address; In response to the existence of a mapping relationship of the current virtual address in its own logical page table, the data of the memory of the first accelerator device pointed to by the current virtual address is copied to its own memory, and the mapping relationship of the current virtual address pointing to the memory of the first accelerator device is modified to a mapping relationship of the current virtual address pointing to its own memory.

16. The device according to any one of claims 11 to 15, characterized in that: The processing unit is further configured to respond to a system call invoking memory replication; Determine, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; The data of the source device is copied to the target device.

17. The device according to claim 16, characterized in that The processing unit is further configured to copy the data of the source device to the accelerator device corresponding to the NUMA ID after receiving the NUMA ID.

18. The device according to any one of claims 11 to 17, characterized in that: The processing unit is further used to detect whether the used memory in the memory pool is greater than a set threshold; In response to the used memory in the memory pool being greater than a set threshold, part or all of the memory that is no longer used is released.

19. The device according to any one of claims 11 to 18, characterized in that: The transceiver unit is further used to query whether there is a mapping relationship of the virtual address in its own logical page table after receiving the NUMA ID and the virtual address; The processing unit is further configured to, in response to the existence of a mapping relationship of the virtual address in its own logical page table, copy the data of the virtual address mapping memory to the accelerator device memory corresponding to the NUMA ID, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; In response to the fact that the mapping relationship of the virtual address does not exist in its own logical page table, a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID is established through a memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

20. The device according to any one of claims 11 to 19, characterized in that: The transceiver unit is further used to receive failure information of the first accelerator device, where the failure information is a message that the first accelerator device fails to apply for memory; A copy instruction is sent to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and to release the memory being used.

21. A computing device, characterized in that include: at least one memory, At least one processor, the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device performs the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product stores instructions, and when the instructions are executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 10.

Images