Memory management method and device and computing equipment

Abstract

The invention discloses a memory management method and device and computing equipment. The method is executed by a GMEM system, after the GMEM system receives a physical memory range, a memory management unit operation function and an accelerator device address of the accelerator device, a memory management unit operation function interface is configured for the accelerator device, and the device where the GMEM system is located is enabled to operate the function interface through the accelerator device memory management unit. And managing a memory management unit and a physical memory of the accelerator equipment by utilizing a driver of the accelerator equipment so as to read and write data. According to the GMEM system, a native memory management interface of an operating system is expanded into a heterogeneous memory management interface, programming of a unified virtual address space is supported, and memory management unit operation function interfaces are configured for various types of accelerator devices, so that the device where the GMEM system is located can support the use of the various types of accelerator devices, and the operation efficiency of the accelerator devices is improved. And the compatibility problem of memory interfaces of various types of accelerator equipment does not need to be considered.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a memory management method, device and computing equipment. Background Art

[0002] With the development of artificial intelligence (AI), a single processor cannot meet the computing needs of fields such as AI. In related technologies, chip manufacturers can bring together multiple different types of processors, accelerators, coprocessors and other computing devices to form heterogeneous chips. Heterogeneous chips can use multiple computing devices to jointly perform computing tasks to meet the computing needs of fields such as AI. In the heterogeneous programming architecture of heterogeneous chips, two sets of virtual address spaces need to be distinguished, and each piece of data needs to maintain two different caches (buffers). For general data, users need to manually manage data migration. For data with complex graphs, trees and other structures, users need to perform complex data format conversions and then manually manage data migration.

[0003] In order to solve the problem of data migration of heterogeneous chips, the relevant technology provides a programming solution based on unified virtual address space (UVAS), specifically: adding a third-party driver to the heterogeneous chip. The third-party driver can manage the specified memory and match the user-mode unified memory application interface. For example, the driver of Nvidia's unified computing architecture (CUDA) and unified virtual memory (UVM) is provided by CUDA with the cudaMallocManaged interface. Both the central processing unit (CPU) and the graphics processing unit (GPU) can directly access the virtual address applied for by this interface. When the CPU or GPU accesses the virtual address, the UVM driver establishes a mapping relationship from the virtual address to the CPU memory or GPU memory by linking the GPU memory management (MM) and Linux MM to ensure normal access of the CPU or GPU.

[0004] However, the third-party drivers configured for each heterogeneous chip cannot be copied to other types of heterogeneous chips, resulting in high development costs. After adding a third-party driver to a heterogeneous chip, the heterogeneous chip needs to be configured with a complex memory management system to manage the local memory of the accelerator, resulting in high code duplication within the heterogeneous chip and performance degradation of the heterogeneous chip. Summary of the invention

[0005] In order to solve the above problems, a memory management method is provided in an embodiment of the present application, wherein the native memory interface of the operating system is extended to an interface supporting heterogeneous memory management, and the interface can support the simultaneous use of multiple types of accelerator chips in the user process without having to consider the compatibility issues of the memory interfaces of various types of accelerator chips. In addition, the present application also provides a memory management device and a computing device corresponding to the memory management method.

[0006] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0007] In a first aspect, the present application provides a memory management method, which is executed by a general memory management GMEM system, and the method includes: receiving device information of at least one mounted accelerator device, the device information including the physical memory address of the accelerator device, a memory management unit operation function and the accelerator device address; configuring a memory management unit operation function interface for the at least one loading device, the memory management unit operation function interface using the memory management unit operation function to use the driver of the accelerator device corresponding to the accelerator device address; using the driver of the accelerator device to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

[0008] In this embodiment, after receiving the physical memory range, memory management unit operation function and accelerator device address of the accelerator device, the GMEM system configures the memory management unit operation function interface for the accelerator device, so that the device where the GMEM system is located can use the accelerator device driver to manage the memory management unit and physical memory of the accelerator device through the accelerator device memory management unit operation function interface to read and write data. The GMEM system of the present application can expand the native memory management interface of the operating system into a heterogeneous memory management interface, support the programming of a unified virtual address space, and configure the memory management unit operation function interface for various types of accelerator devices, so that the device where the GMEM system is located can support the use of multiple types of accelerator devices without considering the compatibility issues of the memory interfaces of various types of accelerator devices.

[0009] In one embodiment, the method also includes: receiving a non-uniform memory access NUMA identifier ID; in response to an accelerator device corresponding to the NUMA ID being in a peer mode, sending a request instruction to the accelerator device corresponding to the NUMA ID, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, wherein the first virtual address is a virtual address applied for by itself.

[0010] In this implementation, after receiving the NUMA ID, the GMEM system can apply for a virtual address when it determines that the accelerator device corresponding to the NUMA ID is in peer mode, and allow the accelerator device corresponding to the NUMA ID to apply for the same virtual address. The GMEM system can copy its own data to the memory of the accelerator device corresponding to the NUMA ID based on the applied virtual address.

[0011] In one embodiment, the method further includes: in response to a page fault being triggered when writing data, detecting whether its own logical page table has its own mapping relationship, wherein the page fault refers to the inability to map the virtual address to the physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; in response to the fact that its own logical page table does not have its own mapping relationship, establishing its own mapping relationship, and saving its own mapping relationship in its own logical page table.

[0012] In this implementation, when the GMEM system writes data, after triggering a page fault, it detects whether there is a mapping relationship in its own logical page table. If the GMEM system detects that there is no mapping relationship in its own logical page table, that is, the current virtual address is not mapped to the physical memory, the GMEM system updates the logical page table and establishes a mapping relationship in the logical page table from the requested virtual address to its own memory, so that the GMEM system continues to write input data.

[0013] In one embodiment, the method also includes: receiving page fault information of a first accelerator device, the page fault information indicating that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; based on the page fault information, querying whether there is a mapping relationship of the virtual address accessed by the first accelerator device in its own logical page table; in response to the fact that there is no mapping relationship of the virtual address accessed by the first accelerator device in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that there is a mapping relationship of the virtual address in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

[0014] In this implementation, when the accelerator device triggers a page fault, page fault information is generated and sent to the GMEM system. After receiving the page fault information, the GMEM system detects whether its own logical page table has a mapping relationship with the accelerator device. When the GMEM system determines that its own logical page table has a mapping relationship with the accelerator device, it can call the memory management unit operation function interface of the accelerator device to establish the mapping relationship of the accelerator device, so as to copy its own data to the memory of the accelerator device.

[0015] In one embodiment, the method also includes: in response to a page fault being triggered when reading data output by the first accelerator device, detecting whether there is a mapping relationship of the current virtual address in its own logical page table; in response to the existence of a mapping relationship of the current virtual address in its own logical page table, copying the data of the memory of the first accelerator device pointed to by the current virtual address to its own memory, and modifying the mapping relationship of the current virtual address pointing to the memory of the first accelerator device to a mapping relationship of the current virtual address pointing to its own memory.

[0016] In this implementation, when the GMEM system reads the calculation result of the accelerator device, a page fault is triggered. The GMEM system detects whether there is a mapping relationship of the current virtual address in its own logical page table. When the GMEM system determines that there is a mapping relationship, it can map the accelerator device memory based on the current virtual address, and copy the accelerator device memory data mapped by the current virtual address to the device memory where the GMEM system is located, so that the GMEM system can read the calculation result of the accelerator device.

[0017] In one embodiment, the method also includes: responding to a system call system call that calls memory copy hmemcpy; based on its own logical page table, determining a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; and copying data of the source device to the target device.

[0018] In this implementation, the GMEM system can use the system call of memory copy, without specifying the physical location of the source address and the target address, and automatically obtain data according to the logical page table to improve ease of use. When copying data between the GMEM system and the accelerator device, the page prefetch semantics can be combined, without the need for transfer through the master node, and the bandwidth of the high-speed bus between devices can be efficiently utilized.

[0019] In one embodiment, the method further includes: after receiving the NUMA ID, copying the data of the source device to the accelerator device corresponding to the NUMA ID.

[0020] In this implementation, the user can also specify the target device, such as sending it to the GMEM system through a heterogeneous NUMA ID. The GMEM system can directly and implicitly complete the data replication from the source device to the target device by using the system call of memory replication. Compared with the method of prefetching first and then copying or copying first and then prefetching, the cost of one memory replication can be saved.

[0021] In one embodiment, the method further includes: detecting whether the used memory in the memory pool is greater than a set threshold; in response to the used memory in the memory pool being greater than the set threshold, releasing part or all of the memory that is no longer used.

[0022] In this implementation, the GMEM system can apply for memory from the memory pool to run the accelerator device. The GMEM system needs to count whether the memory used by the memory pool is greater than the set threshold. If the used memory is greater than the set threshold, the GMEM system needs to use page discard semantics for the cached released memory to release physical memory until the threshold is met or all caches are released.

[0023] In one embodiment, the method also includes: after receiving the NUMA ID and the virtual address, querying whether there is a mapping relationship of the virtual address in its own logical page table; in response to the existence of the mapping relationship of the virtual address in its own logical page table, copying the data of the virtual address mapped memory to the accelerator device memory corresponding to the NUMA ID, and establishing a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; in response to the absence of the mapping relationship of the virtual address in its own logical page table, establishing a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

[0024] In this implementation, the GMEM system receives the NUMA ID and the virtual address, and upon detecting that the mapping relationship of the virtual address exists in its own logical page table, the data mapped by the virtual address can be directly copied to the accelerator device corresponding to the NUMA ID. The GMEM system can reduce the execution time and improve the performance of the operation of the GMEM system by pre-fetching data in advance.

[0025] In one embodiment, the method further includes: receiving failure information of the first accelerator device, wherein the failure information is a message that the first accelerator device fails to apply for memory; sending a copy instruction to the first accelerator device, wherein the copy instruction instructs the first accelerator device to copy the data of the memory being used to its own memory, and release the memory being used.

[0026] In this implementation, when the accelerator device memory is insufficient, part of the memory in use can be swapped to the host memory to release the accelerator device memory, so as to implement the GMEM system to support the memory over-allocation function of the accelerator device.

[0027] In a second aspect, the present application provides a memory management device, comprising: a transceiver unit, used to receive device information of at least one mounted accelerator device, the device information including the physical memory address of the accelerator device, a memory management unit operation function and the accelerator device address; a processing unit, used to configure a memory management unit operation function interface for each of the at least one loading devices, the memory management unit operation function interface using the memory management unit operation function to use the driver of the accelerator device corresponding to the accelerator device address; and using the driver of the accelerator device to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

[0028] In one embodiment, the transceiver unit is further used to receive a non-uniform memory access NUMA identifier ID; the processing unit is further used to send a request instruction to the accelerator device corresponding to the NUMA ID in response to the accelerator device corresponding to the NUMA ID being in a peer mode, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

[0029] In one embodiment, the processing unit is also used to detect whether its own logical page table has its own mapping relationship in response to a page fault triggered when writing data, the page fault means that the virtual address cannot be mapped to the physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; and in response to the fact that its own logical page table does not have its own mapping relationship, establish its own mapping relationship, and save its own mapping relationship in its own logical page table.

[0030] In one embodiment, the transceiver unit is also used to receive page fault information of the first accelerator device, and the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; the processing unit is also used to query its own logical page table based on the page fault information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, call the memory management unit operation function interface of the first accelerator device to establish a mapping relationship of the virtual address accessed by the first accelerator device; in response to the fact that the mapping relationship of the virtual address exists in its own logical page table, call the memory management unit operation function interface of the first accelerator device to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

[0031] In one embodiment, the processing unit is also used to detect whether there is a mapping relationship of the current virtual address in its own logical page table in response to triggering a page fault when reading data output by the first accelerator device; in response to the existence of the mapping relationship of the current virtual address in its own logical page table, copy the data of the memory of the first accelerator device pointed to by the current virtual address to its own memory, and modify the mapping relationship of the current virtual address pointing to the memory of the first accelerator device to the mapping relationship of the current virtual address pointing to its own memory.

[0032] In one embodiment, the processing unit is also used to respond to a system call that calls memory copy; based on its own logical page table, determine the source device corresponding to the source address of the system call and the target device corresponding to the target address of the system call; and copy the data of the source device to the target device.

[0033] In one implementation, the processing unit is further configured to, after receiving the NUMA ID, copy the data of the source device to the accelerator device corresponding to the NUMA ID.

[0034] In one embodiment, the processing unit is further used to detect whether the used memory in the memory pool is greater than a set threshold; in response to the used memory in the memory pool being greater than the set threshold, release part or all of the memory that is no longer used.

[0035] In one embodiment, the transceiver unit is further used to query whether there is a mapping relationship of the virtual address in its own logical page table after receiving the NUMA ID and the virtual address; the processing unit is further used to copy the data of the virtual address mapped memory to the accelerator device memory corresponding to the NUMA ID in response to the existence of the mapping relationship of the virtual address in its own logical page table, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; in response to the absence of the mapping relationship of the virtual address in its own logical page table, establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

[0036] In one embodiment, the transceiver unit is also used to receive failure information of the first accelerator device, where the failure information is a message that the first accelerator device failed to apply for memory; send a copy instruction to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and release the memory being used.

[0037] In a third aspect, an embodiment of the present application provides a computing device, comprising: at least one memory, multiple processors, the multiple processors including a main processor, the main processor being used to execute instructions stored in the at least one memory, so that the computing device can be implemented as each of the possible embodiments of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementation embodiments of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements each possible implementation embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0041] Figure 1 It is a structural diagram of a CUDA in the related art;

[0042] Figure 2 A schematic diagram of the structure of a heterogeneous device provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a general memory management system provided in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of the kernel state provided in the embodiments of the present application;

[0045] Figure 5 A schematic diagram of a process of applying for a unified virtual address in a GMEM system is provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a process for data transmission between a heterogeneous device and a mounted device based on a memory access consistency protocol is provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a process of using heterogeneous memory semantics in a GMEM system is provided in an embodiment of the present application;

[0048] Figure 8 A structural schematic diagram of data transmission between a host side and a slave side is provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of a process of memory replication by a server is provided in an embodiment of the present application;

[0050] Fig.10 A schematic diagram of a process of applying for memory in a user state based on an AI framework is provided in an embodiment of the present application;

[0051] Fig.11 A schematic diagram of a process of performing memory super-resolution by an NPU is provided in an embodiment of the present application;

[0052] Fig.12 A schematic diagram of the structure of a memory management device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0054] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0055] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0058] To facilitate understanding of the solutions of the embodiments of the present application, the technical terms involved in this document are first explained below.

[0059] Peer-to-peer mode refers to a system consisting of multiple nodes (peers) with independent operating systems, where the multiple nodes include a master node and at least one peer node, and the master node manages the peer nodes. In addition, in order to assist the master node in managing the peer nodes, it is necessary to create processes on each node, with the master node's process being the main process and the peer node's process being the co-process.

[0060] The master-slave mode refers to a system consisting of a node with an independent operating system and at least one slave device (without an independent operating system).

[0061] A logical page table is a page table that is independent of hardware. The logical page table is used to implement the mapping relationship between virtual addresses and physical addresses to support the paging mechanism. Usually, the logical page table divides the virtual address space of the process into fixed-size pages (usually 4KB / 2MB in size) and maps each page to the corresponding page in physical memory. The logical page table records the mapping relationship between each virtual page and its corresponding physical page.

[0062] Memory consistency means that in a multi-core or multi-processor system, after one core or processor writes data to a virtual address, when another core or processor accesses the same virtual address, it can ensure that the data written by the former core or processor can be read.

[0063] Page migration refers to the process of migrating a page from one physical memory location to another in an operating system. In a multi-core or multi-processor system, when migrating a page from a source node to a target node, the source node first unmaps the local virtual address and invalidates the page table cache (translation lookaside buffer, TLB). Then, the target node needs to apply for memory and copy the data in the source node memory to the target node memory. After the source node completes the copy operation, the source node's memory can be released.

[0064] Page discard semantics refers to a strategy for virtual memory management that determines when a page can be marked as no longer needed. When a page is no longer used by an active process or thread, page discard semantics can determine when the page is released. Page discard semantics can manage memory efficiently and release physical memory for use by other active processes when needed.

[0065] Data prefetching is a technology that allows a processor or storage system to obtain data in advance to reduce data access latency. After receiving a trigger action from the user, data prefetching loads data into the cache or storage system in advance to reduce the waiting time for subsequent access.

[0066] Reverse mapping refers to the process of finding the corresponding virtual address based on the physical address. In the virtual memory management of the operating system, reverse mapping is usually used to achieve the reverse query of memory mapping. Conversely, forward mapping is the process of mapping the virtual address to the physical address. In other words, the corresponding physical memory address is found according to the given virtual address.

[0067] Figure 1 FIG. 1 is a schematic diagram of a CUDA structure in the related art. Figure 1 As shown, CUDA includes an application programming interface (API), a UVM driver, a Linux MM, a GPU MM, a CPU address (ADDR), a CPU memory management unit (MMU), a GPU ADDR, and a GPU MMU.

[0068] CUDA calls a customized API (such as cudaMallocManaged) to apply for memory. The UVM driver first calls the Linux MM interface to apply for a virtual address, and then maintains the applied virtual address in the GPU MM. The input data written to the operator will trigger a page fault in the CPU. Among them, a page fault means that when the CPU writes data, it will access the virtual address, and the CPU's MMU will trigger an error in mapping the virtual address to the physical address. The CPU sends a page fault to the operator, allowing the operator to run and trigger a page fault in the GPU. At this time, the UVM driver needs to read the CPU's page table to obtain the CPU memory corresponding to the virtual address that triggers the page fault. The UVM driver copies the data in the CPU memory to the GPU memory to complete the page fault processing. After the operator runs, the memory is released.

[0069] In the related art, UVM driver can solve the GPU programming difficulty problem of UVAS programming framework of GPU and CUDA. However, due to the limitation of software stack, UVM driver cannot be reused to other types of accelerator chips. Moreover, the whole solution involves complex interaction between UVM driver, Linux MM and GPU MM, with redundant metadata and processing flow, and poor performance.

[0070] In order to solve the defects existing in the related art, the embodiment of the present application provides a memory management method. After the operating system receives the physical memory range, memory management unit operation function and accelerator device address of the accelerator device, the memory management unit operation function interface is configured for the accelerator device, so that the device where the GMEM system is located can use the driver of the accelerator device to manage the memory management unit and physical memory of the accelerator device through the accelerator device memory management unit operation function interface to read and write data. The operating system can expand the native memory management interface of the operating system into a heterogeneous memory management interface, support the programming of a unified virtual address space, and configure the memory management unit operation function interface for various types of accelerator devices, so that the device where the GMEM system is located can support the use of multiple types of accelerator devices without considering the compatibility issues of the memory interfaces of various types of accelerator devices.

[0071] Figure 2 Schematic diagram of a heterogeneous device provided in an embodiment of the present application. Figure 2 As shown, the heterogeneous device 200 includes an application 210 , an operating system 220 , a basic input output system (BIOS) 230 , and hardware components 240 .

[0072] Application 210 refers to a computer program that completes one or more specific tasks. Application 210 is installed in operating system 220, can interact with the user, and create one or more execution tasks after receiving the user's operation instructions. Each application 210 runs in an independent process and has an independent address space.

[0073] The operating system 220 is a group of interrelated system software programs that manage and control computer operations, run and operate hardware and software resources, and provide public services to organize user interactions. The operating system 220 can schedule various resource modules of the heterogeneous device 200, including hardware and software devices, data information, etc. The use of a computer operating system can reduce the workload of manual resource allocation, reduce the degree of user intervention in computer operations, and greatly improve the intelligent work efficiency of the computer. The operating system 320 can be a Linux system, a real-time operating system (RTOS), etc.

[0074] BIOS 230 is an industry-standard firmware interface. BIOS 230 is a set of programs that are fixed to a read-only memory (ROM) chip on the motherboard of the computer. It stores the most important basic input and output programs of the computer, the self-test program after powering on, and the system self-starting program. The main function of BIOS 230 is to provide the lowest-level and most direct hardware settings and controls for the computer. BIOS 230 does not directly control the hardware component 240, but provides an abstract layer and directly controls the hardware component 240.

[0075] The hardware component 240 includes various hardware of the heterogeneous device 200, such as an accelerator, a GPU, a CPU, a neural processing unit (NPU), etc. The hardware component 240 is used to support the normal operation of the heterogeneous device 200.

[0076] The kernel of the operating system 220 is the core part of the operating system 220, which is responsible for managing the resources of the system and providing various services. The memory space in which the operating system 220 can run itself is divided into two parts, namely the kernel space and the user space. The user space refers to the space in which the user program code runs. The kernel space refers to the space in which the kernel code runs. When a process runs in the user space, the process is in the "user mode". A process in the user mode is also called a user mode process. When a process runs in the kernel space, the process is in the "kernel mode". A process in the kernel mode is also called a kernel mode process.

[0077] In the design of operating system 220, user state refers to a non-privileged execution state. The kernel of operating system 220 prohibits code in this state from performing potentially dangerous operations, such as writing system configuration files, killing other users' processes, restarting the system, etc. Kernel state refers to a privileged execution state. The kernel of operating system 220 can perform any operation on code in this state.

[0078] Figure 3 Schematic diagram of the structure of a general memory management system provided in an embodiment of the present application. Figure 3 As shown, a generalized memory management (GMEM) system 300 includes a user state 310 and a kernel state 320. The user state 310 includes an AI framework, an application program, and a native API of the operating system 220.

[0079] AI Framework is a software toolset for developing, training, and deploying AI models. AI Framework provides developers with a series of tools and interfaces that can easily build and operate AI models. Common AI Frameworks can be TensorFlow, PyTorch, Keras, MXNet, MindSpore, etc. In the embodiments of the present application, both AI Framework and the application can support programming of accelerator devices.

[0080] API is a set of functions, classes or methods provided by the operating system 220 to the application program to implement the functions of interacting and operating with the operating system 220. The API defines the communication protocol and interface specifications between the application program and the operating system, enabling the application program to use the functions of the operating system 220 to complete various tasks. In the embodiment of the present application, the AI ​​Framework or the application program can access and utilize various functions provided by the operating system 220, such as file system access, process management, network communication, etc., by calling the native API of the operating system 220.

[0081] The native API of the operating system 220 may be an interface such as mmap, brk, etc. The AI ​​Framework or application may use the interface such as mmap, brk, etc. to apply for the memory of the accelerator.

[0082] The kernel state 320 includes a heterogeneous memory management unit and an accelerator driver. The heterogeneous memory management unit is responsible for virtual address management and physical memory management of devices such as CPU, GPU, NPU, and accelerator. The accelerator driver is responsible for interacting with accelerators such as GPU and NPU. The accelerator driver adds accelerator local memory management functions, such as memory management unit operations (MMU ops), which are used to access the memory management system of the kernel state 320.

[0083] like Figure 4 As shown, the native API of the operating system 220 can expand the native memory management interface of the operating system 220 to a GMEM interface, such as mmap, brk and other interfaces. When the operating system 220 is in peer mode, first apply for a virtual address in the main process, and then apply for the same virtual address in the co-process of the peer node. When the operating system 220 is in master-slave mode, it supports heterogeneous devices 200 to directly use the virtual addresses applied for by the host's mmap, brk and other interfaces. The newly added GMEM interface (such as the hmadvise interface) can apply a variety of heterogeneous memory semantics to support page abandonment semantics, data prefetching and other functions, which can increase new functions of the GMEM system 300 and improve the performance of the GMEM system 300.

[0084] The accelerator driver may be provided with an MMU ops program to provide a registration interface for MMU operations (including page tables, TLBs, etc.) so that various types of accelerator drivers can implement MMU ops. The MMU ops program allows various types of accelerator drivers to register with the operating system 220 and be managed uniformly by the operating system 220.

[0085] A logical page table is added to the GMEM system 300, and the logical page table includes multiple mapping relationships, each of which represents the mapping between a virtual address and a physical address. In the embodiment of the present application, the heterogeneous memory management unit can maintain the mapping relationship of the same virtual address to any physical address (located in different heterogeneous devices) based on the logical page table. When the same virtual address is accessed by different heterogeneous devices 200, the heterogeneous memory management unit can use the registered MMU ops interface with the help of the logical page table to transparently coordinate multiple MMU page tables and complete data migration or data replication to ensure memory access consistency. The operating system 220 can dynamically switch the underlying memory access consistency protocol and flexibly apply it in different application scenarios to achieve optimal performance.

[0086] When the GMEM system 300 is initialized, the physical memory, MMU ops, device policy (such as in peer mode, master-slave mode), device address and other device information of the external device are input to the heterogeneous device 200, and registered to the heterogeneous memory management unit of the kernel state 320. After receiving the device information of each heterogeneous device, the heterogeneous memory management unit allocates the identifier (identifier, ID) of the non-uniform memory access (NUMA) node to each external device, and downloads the driver of each external device, so that each external device can access the unified heterogeneous memory management architecture of the operating system of the heterogeneous device 200, and realize the heterogeneous device 200 using the corresponding device driver using MMU ops; using the device driver to manage the physical memory to read and write data. .

[0087] Figure 5 A schematic diagram of a process for applying for a unified virtual address in a GMEM system is provided in an embodiment of the present application. Figure 5 As shown, the specific process of the GMEM system 300 applying for a unified virtual address is as follows:

[0088] S501, the user state 310 mounts the device involved in the calculation and obtains the NUMA ID of the mounted device. The mounted device refers to a device that sends its own device information to the heterogeneous device 200.

[0089] S502: When the kernel state 320 determines that the device corresponding to the NUMA ID is in peer mode, a co-process is created and bound.

[0090] S503, the user state 310 calls the mmap interface to apply for memory.

[0091] S504, the kernel state 320 applies for a virtual address in response to the request of the mmap interface to apply for memory.

[0092] S505, kernel state 320 traverses mounted devices.

[0093] S506, the kernel state 320 determines whether the mounted device is in peer mode. In one case, when the kernel state 320 determines that the mounted device is in peer mode, the process S507 is executed. In another case, when the kernel state 320 determines that the mounted device is not in peer mode, that is, in master-slave mode, the process ends directly.

[0094] S507, when the kernel state 320 determines that the mounted device is in peer mode, apply for the same virtual address in the co-process of the device in peer mode.

[0095] S508, kernel state 320 detects whether all devices in peer mode successfully apply for the same virtual address. In one case, when kernel state 320 determines that all devices in peer mode successfully apply for the same virtual address, the process ends directly. In one case, when kernel state 320 determines that all devices in peer mode fail to apply for the same virtual address, process S504 is executed.

[0096] In the embodiment of the present application, when the user process starts, the user state 310 will call the mmap interface to apply for a section of virtual memory, and traverse the mount devices of the current process. For the mount devices in the master-slave mode, the user state 310 does not need additional operations. For the mount devices in the peer-to-peer mode, the user state 310 can first create a co-process on the peer device, and bind the co-process to the main process. Then, the kernel state 320 applies for the same virtual address on the co-process. If the kernel state 320 fails to apply for a device, it will continue to apply for other virtual addresses until all devices apply for the same virtual address.

[0097] Figure 6 The present invention provides a schematic diagram of a process for transmitting data between a heterogeneous device and a mounted device based on a memory access consistency protocol. Figure 6 As shown, the process of data transmission between the heterogeneous device 200 and the mounted device based on the memory access consistency protocol is specifically as follows:

[0098] S601, a page fault is triggered when the page fault device accesses a virtual address. At this time, the heterogeneous device determines that the virtual address is not mapped to a physical address, resulting in the heterogeneous device being unable to access the virtual address, which is called a "page fault". In order to distinguish the heterogeneous device in this state from the heterogeneous device in the normal state, the heterogeneous device in this state can be defined as a "page fault device".

[0099] S602 , the page fault device sends page fault information to the GMEM system 300 .

[0100] S603, the GMEM system 300 queries the logical page table based on the missing page information.

[0101] S604, the GMEM system 300 detects whether the logical page table has a mapping relationship of a missing page device. In one case, the GMEM system 300 detects that the logical page table has a mapping relationship of a missing page device, and executes process S605. In another case, the GMEM system 300 detects that the logical page table does not have a mapping relationship of a missing page device, and executes process S611.

[0102] S605, the GMEM system 300 detects whether the page fault is caused by a permission problem. In one case, the GMEM system 300 detects that the page fault is caused by a permission problem, and executes process S606. In another case, the GMEM system 300 detects that the page fault is not caused by a permission problem, and executes process S609. The permission problem refers to the permission of the MMU page table, that is, the MMU page table indicates a read operation, or the MMU page table indicates a write operation.

[0103] S606, the GMEM system 300 detects whether the permission can be modified. In one case, the GMEM system 300 detects that the permission can be modified, and executes process S607. In another case, the heterogeneous device 200 detects that the permission cannot be modified, and executes process S608.

[0104] S607, the GMEM system 300 modifies the permissions. That is, the page fault is caused by a permission problem, and the GMEM system 300 modifies the mapped permissions to meet the requirements of the page fault device.

[0105] S608, the GMEM system 300 returns an error.

[0106] S609, the GMEM system 300 calls the MMU ops registered by the current mapping device to release the mapping relationship in the logical page table.

[0107] S610 , the GMEM system 300 copies the data of the current mapped device memory to the memory of the page-missing device.

[0108] S611, the GMEM system 300 calls the MMU ops registered by the page fault device to establish or modify the mapping relationship in the logical page table.

[0109] S612 , the GMEM system 300 updates the logical page table.

[0110] In the embodiment of the present application, when the page missing device accesses the virtual address, a page missing fault is triggered, and the page missing information is forwarded to the GMEM system 300. After receiving the page missing information, the GMEM system 300 can query the logical page table based on the page missing information. If the GMEM system 300 finds that its own logical page table does not have a mapping relationship for the page missing device, it can call the MMU ops registered by the page missing device to establish a mapping relationship. After the GMEM system 300 completes the establishment of the mapping relationship, it can update the logical page table and establish a mapping relationship for the page missing device in the logical page table. If the GMEM system 300 finds that its own logical page table has a mapping relationship for the page missing device, it can detect whether the page missing fault is caused by a permission problem.

[0111] If the GMEM system 300 detects that the page fault is caused by a permission problem, it can detect whether the permission can be modified. If the GMEM system 300 detects that the permission can be modified, it modifies the permission and then calls the MMU ops registered by the page fault device to modify the mapping relationship. If the GMEM system 300 detects that the permission cannot be modified, it directly returns an error. If the GMEM system 300 detects that the page fault is not caused by a permission problem, it can obtain the current mapping device through the logical page table and then call the MMU ops registered by the current mapping device to release the mapping relationship between the GMEM system 300 and the current mapping device. The GMEM system 300 can copy the data to the memory of the page fault device and then call the MMU ops registered by the page fault device to establish or modify the mapping relationship. The heterogeneous device 200 updates the logical page table and establishes a mapping relationship pointing to the page fault device in the logical page table.

[0112] Figure 7 A schematic diagram of a process of using heterogeneous memory semantics in a GMEM system is provided in an embodiment of the present application. Figure 7 As shown, the process of using heterogeneous memory semantics in the GMEM system 300 is as follows:

[0113] S701 , the GMEM system 300 initializes the user process in the user state 310 .

[0114] S702, the user state 310 obtains the NUMAID of the device corresponding to the virtual address.

[0115] S703, the user state 310 calls the GMEM interface to transfer the device information of each mounted device to the kernel state 320. The device information includes NUMAID, applied virtual address, used semantic parameters (such as page discard semantics, data prefetching) and other information.

[0116] S704, the kernel state 320 detects whether the NUMAID is received. In one case, the kernel state 320 receives the NUMAID, and the process S705 is executed. In another case, the kernel state 320 does not receive the NUMAID, and the process S706 is executed.

[0117] S705 , the kernel state 320 calls the MMU ops registered by the device corresponding to the NUMAID to deliver heterogeneous memory semantics.

[0118] S706, the kernel state 320 traverses the mounted devices and calls the MMU ops registered by the mounted devices to issue heterogeneous memory semantics.

[0119] In the embodiment of the present application, the GMEM system 300 can obtain the NUMAID of the corresponding device according to the virtual address in the user process of the user state 310, and then pass the device information such as the NUMAID, the virtual address and the semantic parameters used to the kernel state 320 through the GMEM interface. If the kernel state 320 receives the NUMAID, the MMUops registered by the specified device can be called to issue heterogeneous memory semantics. If the kernel state 320 does not receive the NUMAID, the mounted devices can be traversed and the MMU ops registered by the mounted device can be called to issue heterogeneous memory semantics.

[0120] To facilitate understanding, at least one application scenario of an embodiment of the present application is first introduced below.

[0121] Figure 8 A schematic diagram of a structure for data transmission between a host side and a slave side is provided in an embodiment of the present application. Figure 8 As shown in the figure, the host side can be a server, and the co-host side can be an NPU. The server's operating system can expand the original memory management to heterogeneous memory management. Heterogeneous memory management can implement a memory access consistency protocol through a logical page table to support the UVAS programming framework in peer mode and improve the performance of NPU operator operation.

[0122] In the server, the user state includes the main process and the mmap / brk interface, and the kernel state includes the heterogeneous memory management unit. The heterogeneous memory management unit includes the logical page table, the memory access consistency protocol, and the MMU ops interface. Among them, the memory access consistency protocol can realize the interaction between the CPU MMU ops program and the NPU MMU ops program through the MMU ops interface and the NPU driver.

[0123] In the NPU, it includes the NPU driver, and the NPU driver includes the NPU MMU management program. A message channel and a direct memory access (DMA) channel are connected between the server and the NPU. The message channel is connected between the NPU MMUops program and the NPU MMU management, and is used to transmit messages at the software layer. The DMA channel is connected between the host memory and the NPU memory, and is used to migrate data at the hardware layer.

[0124] During the initialization process of the server, the main process can be started to establish a communication connection between the server and the NPU to enable the NPU to be mounted on the main process. The main process can notify the NPU to create a co-process and bind the co-process to the main process. When the server is preparing the memory required for the NPU operator to run, the user state of the server can call the mmap interface to apply for memory. After the mmap interface falls into the kernel state, the kernel state can first apply for a virtual address, and then notify the NPU co-process to apply for the same virtual address. If the kernel state fails to apply for a virtual address, the main process applies for other virtual addresses again, and applies for the same virtual address through the co-process until the application is successful. NPU operators refer to symbols, instructions, or functions that run specific operations in the NPU.

[0125] When the main process of the server prepares input data for the NPU operator, after the main process applies for the virtual address, the input data of the NPU operator can be written to the applied virtual address. When the NPU operator writes the input data, it triggers a page fault on the server. When the main process of the server handles the page fault, it detects whether there is a mapping relationship in its own logical page table. If the main process detects that there is no mapping relationship in its own logical page table, that is, the current virtual address is not mapped to the physical memory. At this time, the main process can directly execute the server's page fault processing flow, update the logical page table, and establish a mapping relationship in the logical page table from the applied virtual address to the server memory. After completing the page fault processing, the main process allows the NPU operator to continue writing input data.

[0126] The main process of the server ensures that the NPU operator is running based on the memory access consistency protocol. When the NPU operator accesses the input data for the first time when running on the NPU, it will trigger a page fault of the NPU. The NPU generates page fault information and transmits it to the server through the NPU driver, allowing the server to detect whether its own logical page table has a mapping relationship with the current virtual address. The main process detects whether its own logical page table has a mapping relationship with the current virtual address and determines the host memory mapped by the current virtual address. The main process copies the host memory page mapped by the current virtual address to the NPU memory. The main process updates the logical page table and modifies the mapping relationship of the current virtual address in the logical page table pointing to the server's host memory to the mapping relationship of the current virtual address pointing to the NPU memory of the NPU.

[0127] When the NPU operator runs on the NPU and outputs the calculation results, it triggers a page fault of the NPU. The NPU generates page fault information and transmits it to the server through the NPU driver, allowing the server to detect whether its own logical page table has a mapping relationship with the current virtual address. Usually, the server detects that its own logical page table does not have a mapping relationship with the current virtual address. The server can directly call the NPU MMU ops program registered by the NPU to establish a mapping relationship. The server updates the logical page table and establishes a mapping relationship pointing to the NPU memory in the logical page table.

[0128] When the server reads the calculation results generated by the NPU operator in the NPU, the server's page fault will be triggered when the main process reads the calculation results. The main process will detect whether there is a mapping relationship of the current virtual address in its own logical page table. The main process detects that there is a mapping relationship of the current virtual address in its own logical page table, that is, the current virtual address is mapped to the NPU memory. At this time, the main process determines that the current virtual address is mapped to the NPU memory, and the NPU memory mapped by the current virtual address can be mapped to the host memory corresponding to the server, and the page of the NPU memory mapped by the current virtual address can be copied to the corresponding host memory. The main process updates the logical page table and modifies the mapping relationship of the current virtual address pointing to the NPU memory of the NPU in the logical page table to the mapping relationship of the current virtual address pointing to the host memory of the server.

[0129] Before the NPU operator accesses the input data for the first time, the main process can call the data prefetch provided by the hmadvise interface to prefetch the input data. When the main process calls prefetch, it needs to pass in the NUMA ID of the NPU. Prefetch will evenly divide the input data so that the divided input data can be processed in parallel by multiple threads. The main process queries the logical page table, and after detecting the mapping relationship between the virtual address of the input data and the server memory, it asks the server to copy the page to the NPU. The NPU updates the logical page table and modifies the mapping relationship from the current virtual address in the logical page table to the server's host memory to the current virtual address to the NPU memory. When the NPU operator runs on the NPU, it can directly access the input data without triggering the NPU page fault, thereby reducing the execution time of the operator. Under the unified heterogeneous memory management framework, the server and NPU can improve the performance of the NPU operator by prefetching data in advance.

[0130] Fig. 9 A schematic diagram of a process of performing memory replication on a server is provided in an embodiment of the present application. Fig. 9 As shown in the figure, the specific process of memory replication by the server is as follows:

[0131] S901, the kernel state of the server calls a system call (system call) of memory copy (hmemcpy).

[0132] A system call is a request from a program running in user space to the operating system for a service that requires higher permissions. A system call can provide an interface between user programs and the operating system to implement memory copy functions.

[0133] S902, the server executes in parallel with multiple threads according to page granularity. Page granularity refers to the virtual address space being divided into pages of fixed size, generally 4KB or 2MB in size.

[0134] S903, the server queries the logical page table to obtain the source device mapped by the source address.

[0135] When the server performs memory copy, the system call generally indicates the source address and the target address, and copies the data of the device corresponding to the source address to the device corresponding to the destination address. Among them, the source address and the target address are both virtual addresses. The logical page table includes a mapping relationship, and the mapping relationship can map the virtual address to the physical address. The physical address can be the physical address of other devices. The server can query the device corresponding to the physical address mapped by the source address in the logical page table and determine it as the source device.

[0136] S904, the server detects whether the NUMA ID is received. In one case, the server detects the NUMA ID, and the process S905 is executed. In another case, the server does not detect the NUMA ID, and the process S908 is executed.

[0137] S905: The server queries the logical page table to obtain the device currently mapped by the NUMA ID.

[0138] S906, the server determines whether the NUMA ID of the currently mapped device is the same as the NUMA ID of the target device. In one case, the server determines that the NUMA ID of the currently mapped device is not the same as the NUMA ID of the target device, and executes process S907. In another case, the server determines that the NUMA ID of the currently mapped device is the same as the NUMA ID of the target device, and executes process S909.

[0139] S907, the server calls the MMU ops registered by the currently mapped device to release the mapping relationship, invalidate the TLB, and release the physical page.

[0140] S908: The server queries the logical page table to obtain the device mapped by the target address, and sets the device mapped by the target address as the target device.

[0141] S909, the server copies the page of the source device to the target device.

[0142] S910, the server calls the MMU ops registered by the target device to establish a mapping relationship.

[0143] S911, the server updates the logical page table and points the target address to the target device.

[0144] In the embodiment of the present application, the server and the NPU implement parallel memory replication in any direction under a unified heterogeneous memory management framework. The kernel state of the server can use the system call of memory replication, without specifying the physical location of the source address and the target address, and automatically obtain data according to the logical page table to improve ease of use. When copying data between the server and the NPU, the page prefetch semantics can be combined, and there is no need to transfer through the master node, which can efficiently utilize the bandwidth of the high-speed bus between devices.

[0145] When the server uses the system call of memory replication to perform cross-node data replication, it does not need to explicitly specify the physical location of the source address and the target address. The kernel state of the server can obtain the specific physical location based on the logical page table and directly complete the data replication. In addition, the user can also specify the target device, such as sending it to the server through a heterogeneous NUMA ID. The server can directly and implicitly complete the data replication from the source device to the target device when using the system call of memory replication. Compared with the method of prefetching first and then copying (without specifying NUMA ID) or copying first and then prefetching (without specifying NUMA ID), the cost of one memory replication can be saved. When the server uses the system call of memory replication, the data is parallelized according to the granularity of the memory page to maximize the use of the bus bandwidth. In addition, there is no need to transfer through the master node, and the high-speed bus between devices can be fully utilized.

[0146] Fig.10 A schematic diagram of a user-mode memory application process based on an AI framework is provided in the embodiment of the present application. Fig.10 As shown in the figure, the user state takes MindSpore as an example, and the specific process of applying for memory is as follows:

[0147] S1001, MindSpore starts the asynchronous cache release thread to release the cache regularly.

[0148] S1002, MindSpore lets the NPU operator apply for memory from the memory pool.

[0149] S1003, MindSpore detects whether the memory used by the memory pool is greater than the set threshold. In one case, MindSpore detects that the memory used by the memory pool is greater than the set threshold, and executes step S1004. In another case, MindSpore detects that the memory used by the memory pool is less than or equal to the set threshold, and executes step S1007.

[0150] S1004: MindSpore uses page discard semantics for some of the used memory to release physical memory.

[0151] S1005, MindSpore detects whether the used memory after releasing part of the cache is greater than the set threshold. In one case, MindSpore detects that the used memory after releasing part of the cache is greater than the set threshold, and executes step S1007. In another case, MindSpore detects that the used memory after releasing part of the cache is less than or equal to the set threshold, and executes step S1006.

[0152] S1006, MindSpore detects whether all used memory has been released. In one case, if MindSpore detects that all used memory has been released, step S1007 is executed. In another case, if MindSpore detects that all used memory has not been released, step S1004 is executed.

[0153] S1007: MindSpore executes the NPU operator.

[0154] S1008: After the NPU operator runs, MindSpore releases the used memory.

[0155] S1009, MindSpore places the released memory in the memory pool.

[0156] In an embodiment of the present application, the server and NPU can implement memory management with high memory utilization based on the MindSpore framework, UVAS programming framework, and page discard semantics provided by the GMEM interface. Due to the static mapping between virtual addresses and physical memory, when MindSpore applies for virtual addresses through the NPU software stack, MindSpore can only use a virtual address space of equal size to the NPU physical memory to achieve secondary management. When MindSpore generates memory fragmentation, the minimum physical memory utilization may be only 30%. Therefore, the GMEM system removes the restrictions on static mapping between virtual addresses and physical memory, eliminates memory fragmentation generated during MindSpore memory management, and improves memory utilization to 99%+.

[0157] Before MindSpore sends NPU operators, it needs to apply for memory from the memory pool for running NPU operators. Since the memory pool is implemented based on UVAS and is not limited by the physical memory size of the NPU, the application for virtual address can always succeed. MindSpore needs to count whether the used memory is greater than the set threshold. If the used memory is greater than the set threshold, MindSpore needs to use page abandonment semantics for the cached released memory to release physical memory until the threshold is met or all caches are released. If the used memory is less than or equal to the set threshold, execute the NPU operator. When the NPU operator ends, the requested memory is released and cached in the memory pool, which can be reused or actually released. The cached memory will periodically call page abandonment semantics for release asynchronously.

[0158] In an embodiment of the present application, the NPU driver of the NPU may set an NPU memory recovery unit. The NPU memory recovery unit may borrow the memory of the server when the memory is insufficient to realize the super-resolution function of the NPU. When the NPU memory recovery unit needs to reclaim the NPU physical memory, it may extract the physical pages of the NPU from the linked list formed by connecting the currently used NPU physical pages in series according to the elimination strategy. Then, the NPU memory recovery unit first applies for physical pages on the server according to the page granularity, and then completes the page migration to the server for the selected NPU physical pages, and updates the logical page table to point to the server memory. The NPU memory recovery unit may temporarily store part of the data of the NPU physical memory in the server memory, and release the NPU physical memory for the NPU operator to run and use. When the swapped out data is accessed by the NPU operator again, a page fault of the NPU will be triggered.

[0159] Fig.11 A schematic diagram of a process of performing memory super-resolution on an NPU is provided in an embodiment of the present application. Fig.11 As shown in the figure, the specific process of NPU memory super-resolution is as follows:

[0160] S1101, when the NPU operator runs on the NPU, a page fault is triggered.

[0161] S1102: The NPU operator applies for NPU physical memory.

[0162] S1103, the NPU operator detects whether the memory application is successful. In one case, when the NPU operator detects that the memory application is unsuccessful, the process S1104 is executed. In another case, when the NPU operator detects that the memory application is successful, the process S1105 is executed.

[0163] S1104, the NPU operator synchronously reclaims the memory of the NPU.

[0164] S1105, the NPU operator detects whether the NPU remaining memory is greater than a set threshold. In one case, when the NPU operator detects that the NPU remaining memory is less than or equal to the set threshold, the process S1106 is executed. In another case, when the NPU operator detects that the NPU remaining memory is greater than the set threshold, the process S1107 is executed.

[0165] S1106, the NPU operator wakes up the asynchronous thread and reclaims the NPU memory.

[0166] S1107, the NPU operator records the requested physical page in a linked list.

[0167] S1108, the NPU operator processes a page fault.

[0168] In the embodiment of the present application, when the NPU operator triggers a page fault, it is necessary to apply for NPU physical memory. If the NPU operator application fails, the NPU memory is reclaimed. If the NPU operator application succeeds, it is determined whether the current NPU remaining memory is greater than the set threshold. If it is less than the threshold, it is necessary to wake up the asynchronous thread, and the NPU operator reclaims the NPU memory. Finally, the NPU operator records the applied physical page in the linked list and continues to process the page fault.

[0169] Fig.12 A schematic diagram of the structure of a memory management device is provided in an embodiment of the present application. Fig.12 As shown, the memory management device 1200 can be divided into a transceiver unit 1210 and a processing unit 1220 according to the execution function. The specific implementation process of the memory management device 1200 is as follows:

[0170] The transceiver unit 1210 is used to receive device information of at least one mounted accelerator device. The device information includes the physical memory address of the accelerator device, the memory management unit operation function and the accelerator device address. The processing unit 1220 is used to configure the memory management unit operation function interface for at least one loading device respectively. The memory management unit operation function interface uses the driver of the accelerator device corresponding to the accelerator device address using the memory management unit operation function. The processing unit 1220 is also used to manage the physical memory corresponding to the physical memory address of the accelerator device using the driver of the accelerator device to read and write data. .

[0171] In one embodiment, the transceiver unit 1210 is further configured to receive a non-uniform memory access NUMA identifier ID. The processing unit 1220 is further configured to send a request instruction to the accelerator device corresponding to the NUMA ID in response to the accelerator device corresponding to the NUMA ID being in peer mode. The request instruction instructs the accelerator device corresponding to the NUMA ID to apply for the first virtual address. The first virtual address is a virtual address applied for by itself.

[0172] In one embodiment, the processing unit 1220 is also used to detect whether its own logical page table has its own mapping relationship in response to triggering a page fault when writing data. A page fault refers to the inability to map a virtual address to a physical address. A mapping relationship refers to the relationship between a virtual address and a physical address. The processing unit 1220 is also used to establish its own mapping relationship in response to the absence of its own mapping relationship in its own logical page table, and save its own mapping relationship in its own logical page table.

[0173] In one embodiment, the transceiver unit 1210 is also used to receive page fault information of the first accelerator device. The page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address. At least one accelerator device includes the first accelerator device. The processing unit 1220 is also used to query its own logical page table based on the page fault information whether there is a mapping relationship of the virtual address accessed by the first accelerator device. The processing unit 1220 is also used to call the memory management unit operation function interface of the first accelerator device to establish a mapping relationship of the virtual address accessed by the first accelerator device in response to the fact that there is a mapping relationship of the virtual address accessed by the first accelerator device in its own logical page table. The processing unit 1220 is also used to call the memory management unit operation function interface of the first accelerator device to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device in response to the fact that there is a mapping relationship of the virtual address in its own logical page table, and establish a mapping relationship of the first accelerator device.

[0174] In one embodiment, the processing unit 1220 is further configured to, in response to triggering a page fault when reading data output by the first accelerator device, detect whether there is a mapping relationship of the current virtual address in its own logical page table. The processing unit 1220 is further configured to, in response to the existence of a mapping relationship of the current virtual address in its own logical page table, copy the data of the memory of the first accelerator device pointed to by the current virtual address to its own memory, and modify the mapping relationship of the current virtual address pointing to the memory of the first accelerator device to the mapping relationship of the current virtual address pointing to its own memory.

[0175] In one embodiment, the processing unit 1220 is further configured to respond to a system call for invoking memory copy. The processing unit 1220 is further configured to determine, based on its own logical page table, a source device corresponding to a source address of the system call and a target device corresponding to a target address of the system call. The processing unit 1220 is further configured to copy data of the source device to the target device.

[0176] In one implementation, the processing unit 1220 is further configured to copy the data of the source device to the accelerator device corresponding to the NUMA ID after receiving the NUMA ID.

[0177] In one embodiment, the processing unit 1220 is further configured to detect whether the used memory in the memory pool is greater than a set threshold. The processing unit 1220 is further configured to release part or all of the memory that is no longer used in response to the used memory in the memory pool being greater than the set threshold.

[0178] In one embodiment, the transceiver unit 1210 is also used to query whether there is a mapping relationship of the virtual address in its own logical page table after receiving the NUMA ID and the virtual address. The processing unit 1220 is also used to copy the data of the virtual address mapping memory to the accelerator device memory corresponding to the NUMA ID in response to the existence of the mapping relationship of the virtual address in its own logical page table, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID. The processing unit 1220 is also used to establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID in response to the absence of the mapping relationship of the virtual address in its own logical page table. .

[0179] In one embodiment, the transceiver unit 1210 is further used to receive failure information of the first accelerator device. The failure information is a message that the first accelerator device fails to apply for memory; the transceiver unit 1210 is further used to send a copy instruction to the first accelerator device. The copy instruction instructs the first accelerator device to copy the data of the memory being used to its own memory and release the memory being used.

[0180] The present application also provides a computing device in an embodiment. The computing device includes at least one memory and at least one processor. The at least one processor can execute the following steps: Figure 2-Figure 11 And the above corresponding protected technical solutions, so that the computing device has the technical effect of the above protected technical solutions.

[0181] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer executes the above Figure 2-Figure 11 And the memory management method described in any one of the corresponding description contents.

[0182] In an embodiment of the present application, a computer program product is also provided, wherein the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the above Figure 2-Figure 11 And the memory management method described in any one of the corresponding description contents.

[0183] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0184] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0185] In the above embodiment, Fig.12The memory management device 1200 described in the embodiment can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0186] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or an access network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0191] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A memory management method, characterized in that: The method is performed by a general memory management GMEM system, and the method comprises: Receive device information of at least one mounted accelerator device, the device information including a physical memory address of the accelerator device, a memory management unit operation function, and an accelerator device address; A memory management unit operation function interface is configured for each of the at least one loading devices, wherein the memory management unit operation function interface uses the driver of the accelerator device corresponding to the accelerator device address by using the memory management unit operation function; The driver of the accelerator device is used to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

2. The method according to claim 1, characterized in that The method further comprises: Receive the non-uniform memory access NUMA identifier ID; In response to the accelerator device corresponding to the NUMA ID being in peer mode, a request instruction is sent to the accelerator device corresponding to the NUMA ID, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: In response to a page fault being triggered when writing data, detecting whether its own logical page table has its own mapping relationship, wherein the page fault refers to the inability to map a virtual address to a physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; In response to the fact that the own logical page table does not have its own mapping relationship, the own mapping relationship is established, and the own mapping relationship is stored in the own logical page table.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: receiving page fault information of a first accelerator device, wherein the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; querying its own logical page table based on the missing page information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; In response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish the mapping relationship of the virtual address accessed by the first accelerator device; In response to the existence of the mapping relationship of the virtual address in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to triggering a page fault when reading data output by the first accelerator device, detecting whether its own logical page table has a mapping relationship of the current virtual address; In response to the existence of a mapping relationship of the current virtual address in its own logical page table, the data of the memory of the first accelerator device pointed to by the current virtual address is copied to its own memory, and the mapping relationship of the current virtual address pointing to the memory of the first accelerator device is modified to a mapping relationship of the current virtual address pointing to its own memory.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to a system call for hmemcpy; Determine, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; The data of the source device is copied to the target device.

7. The method according to claim 6, characterized in that The method further comprises: After receiving the NUMA ID, the data of the source device is copied to the accelerator device corresponding to the NUMA ID.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Check whether the used memory in the memory pool is greater than the set threshold; In response to the used memory in the memory pool being greater than a set threshold, part or all of the memory that is no longer used is released.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: After receiving the NUMA ID and the virtual address, query whether there is a mapping relationship of the virtual address in its own logical page table; In response to the existence of the mapping relationship of the virtual address in its own logical page table, copy the data of the virtual address mapping memory to the accelerator device memory corresponding to the NUMA ID, and establish the mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; In response to the fact that the mapping relationship of the virtual address does not exist in its own logical page table, a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID is established through a memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: receiving failure information of the first accelerator device, where the failure information is a message that the first accelerator device fails to apply for memory; A copy instruction is sent to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and to release the memory being used.

11. A memory management device, characterized in that: include: A transceiver unit, configured to receive device information of at least one mounted accelerator device, wherein the device information includes a physical memory address of the accelerator device, a memory management unit operation function, and an accelerator device address; A processing unit, configured to respectively configure a memory management unit operation function interface for the at least one loading device, wherein the memory management unit operation function interface uses a driver of the accelerator device corresponding to the accelerator device address by using the memory management unit operation function; The driver of the accelerator device is used to manage the physical memory corresponding to the physical memory address of the accelerator device to read and write data.

12. The device according to claim 11, characterized in that The transceiver unit is further used to receive a non-uniform memory access NUMA identifier ID; The processing unit is further used to send a request instruction to the accelerator device corresponding to the NUMA ID in response to the accelerator device corresponding to the NUMA ID being in peer mode, wherein the request instruction instructs the accelerator device corresponding to the NUMA ID to apply for a first virtual address, where the first virtual address is a virtual address applied for by itself.

13. The device according to claim 11 or 12, characterized in that The processing unit is further configured to detect whether its own logical page table has its own mapping relationship in response to triggering a page fault when writing data, wherein the page fault refers to the inability to map a virtual address to a physical address, and the mapping relationship refers to the relationship between the virtual address and the physical address; as well as In response to the fact that the own logical page table does not have its own mapping relationship, the own mapping relationship is established, and the own mapping relationship is stored in the own logical page table.

14. The device according to any one of claims 11 to 13, characterized in that: The transceiver unit is further used to receive page fault information of a first accelerator device, wherein the page fault information indicates that a page fault is triggered when the first accelerator device accesses a virtual address; the at least one accelerator device includes the first accelerator device; The processing unit is further configured to query its own logical page table based on the missing page information whether there is a mapping relationship of the virtual address accessed by the first accelerator device; In response to the fact that the mapping relationship of the virtual address accessed by the first accelerator device does not exist in its own logical page table, calling the memory management unit operation function interface of the first accelerator device to establish the mapping relationship of the virtual address accessed by the first accelerator device; In response to the existence of the mapping relationship of the virtual address in its own logical page table, the memory management unit operation function interface of the first accelerator device is called to copy the data of the memory mapped by the virtual address accessed by the first accelerator device to the memory of the first accelerator device, and establish the mapping relationship of the first accelerator device.

15. The device according to any one of claims 11 to 14, characterized in that: The processing unit is further configured to trigger a page fault in response to reading data output by the first accelerator device, and detect whether its own logical page table has a mapping relationship of the current virtual address; In response to the existence of a mapping relationship of the current virtual address in its own logical page table, the data of the memory of the first accelerator device pointed to by the current virtual address is copied to its own memory, and the mapping relationship of the current virtual address pointing to the memory of the first accelerator device is modified to a mapping relationship of the current virtual address pointing to its own memory.

16. The device according to any one of claims 11 to 15, characterized in that: The processing unit is further configured to respond to a system call invoking memory replication; Determine, based on its own logical page table, a source device corresponding to the source address of the system call and a target device corresponding to the target address of the system call; The data of the source device is copied to the target device.

17. The device according to claim 16, characterized in that The processing unit is further configured to copy the data of the source device to the accelerator device corresponding to the NUMA ID after receiving the NUMA ID.

18. The device according to any one of claims 11 to 17, characterized in that: The processing unit is further used to detect whether the used memory in the memory pool is greater than a set threshold; In response to the used memory in the memory pool being greater than a set threshold, part or all of the memory that is no longer used is released.

19. The device according to any one of claims 11 to 18, characterized in that: The transceiver unit is further used to query whether there is a mapping relationship of the virtual address in its own logical page table after receiving the NUMA ID and the virtual address; The processing unit is further configured to, in response to the existence of a mapping relationship of the virtual address in its own logical page table, copy the data of the virtual address mapping memory to the accelerator device memory corresponding to the NUMA ID, and establish a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID through the memory management unit operation function interface of the accelerator device corresponding to the NUMA ID; In response to the fact that the mapping relationship of the virtual address does not exist in its own logical page table, a mapping relationship from the virtual address to the accelerator device memory corresponding to the NUMA ID is established through a memory management unit operation function interface of the accelerator device corresponding to the NUMA ID.

20. The device according to any one of claims 11 to 19, characterized in that: The transceiver unit is further used to receive failure information of the first accelerator device, where the failure information is a message that the first accelerator device fails to apply for memory; A copy instruction is sent to the first accelerator device, where the copy instruction instructs the first accelerator device to copy the data in the memory being used to its own memory, and to release the memory being used.

21. A computing device, characterized in that include: at least one memory, At least one processor, the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device performs the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product stores instructions, and when the instructions are executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 10.

Images