NUMA memory allocation method used in AI server, medium and product

Abstract

The invention provides an NUMA memory allocation method used in an AI server, a medium and a product. The method comprises the following steps: receiving a memory allocation request for allocating an NUMA memory to an AI accelerator in an AI server; obtaining an ID of the AI accelerator according to context information of a current thread which is running in the AI server and is associated with the memory allocation request; determining an optimal affinity NUMA node corresponding to the AI accelerator based on the ID of the AI accelerator and a mapping table of the ID of the AI accelerator and the ID of the optimal affinity NUMA node, wherein the mapping table is constructed in an AI server; and allocating memory for the AI accelerator in the NUMA memory corresponding to the best affinity NUMA node.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a NUMA memory allocation method, a computer readable storage medium and a computer program product for an artificial intelligence (AI) server. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technologies, in order to meet the huge demand for computing power of model training and inference, the server hardware architecture is becoming increasingly complex. Modern AI servers usually adopt a Non-Uniform Memory Access (NUMA) architecture, which contains multiple NUMA nodes and multiple AI accelerators (such as neural network processing units (NPUs), graphics processing units (GPUs), etc.). In this architecture, the memory access time of the AI accelerator depends on the physical location of the memory relative to the AI accelerator, so there is a significant difference in the delay of the AI accelerator accessing the memory of different nodes. SUMMARY

[0003] In one aspect, the present disclosure discloses a NUMA memory allocation method for an AI server, comprising: receiving a memory allocation request for allocating a NUMA memory for an AI accelerator in the AI server; obtaining an ID of the AI accelerator according to context information of a current thread associated with the memory allocation request that is running in the AI server; determining a best-affinity NUMA node corresponding to the AI accelerator based on the ID of the AI accelerator and a mapping table of AI accelerator IDs and best-affinity NUMA node IDs constructed in the AI server; and allocating memory for the AI accelerator in the NUMA memory corresponding to the best-affinity NUMA node.

[0004] In other aspects, the present disclosure discloses a computer readable storage medium storing instructions and a computer program product comprising instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0005] Aspects of the disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is to be noted, however, that the various features illustrated in the drawings are not necessarily drawn to scale. Indeed, for clarity, various features can be shown in somewhat schematic form and certain sizes can be exaggerated to help improve understanding of the present disclosure.

[0006] Figure 1 A simplified schematic diagram of a NUMA architecture commonly used in AI servers is shown.

[0007] Figure 2 A schematic flowchart of a NUMA memory allocation method for an AI server according to an embodiment of the present disclosure is shown.

[0008] Figure 3 is a schematic diagram of a computing device that can implement embodiments according to the present application. DETAILED DESCRIPTION

[0009] The following disclosure provides many different embodiments, or examples, for implementing different characteristics of the provided subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to limit the scope of what is possible.

[0010] Furthermore, various operations will be described as multiple discrete operations, in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation.

[0011] With the rapid development of artificial intelligence and deep learning technologies, in order to meet the huge demand for computing power of model training and inference, the server hardware architecture is becoming increasingly complex. Modern AI server systems usually use NUMA architecture. With the surge in demand for memory bandwidth and capacity of AI workloads, NUMA architecture effectively alleviates the memory access bottleneck in traditional symmetric multiprocessing (SMP) architecture by providing scalable memory subsystems.

[0012] Figure 1 A simplified schematic diagram of the NUMA architecture commonly used in AI servers is shown. As shown in Figure 1 This NUMA architecture usually adopts a multi-socket design, and each central processing unit (CPU) socket constitutes an independent NUMA node with exclusive local memory and input / output (I / O) resources. The NUMA node includes a NUMA node core corresponding to one CPU socket, which can contain one or more CPU cores. This architecture allows multiple processors to work in parallel, while enabling inter-node communication through high-speed interconnection channels (such as Intel's Quick Path Interconnect (QPI) / Ultra Path Interconnect (UPI) or AMD's Infinity Fabric). Under this architecture, each NUMA node is usually directly connected to one or more AI accelerators (such as NPUs, GPUs, etc.), and data transmission is achieved through a PCIe bus or a higher-speed dedicated interconnect (such as NVLink).

[0013] In the AI computing process, the data transmission efficiency between the host and the AI acceleration device is often one of the key links that affect the overall performance. In order to maximize the throughput and reduce the delay of this process, the page-locked memory technology is generally used. This technology locks the host's memory page in the physical memory, preventing the operating system from swapping it out to the disk (i.e., avoiding paging exchange), thereby ensuring the long-term stability and continuity of the memory address. The use of page-locked memory allows the AI acceleration device to directly access the host memory through the direct memory access (DMA) engine without the involvement of the CPU and the participation of data copying.

[0014] However, in the AI server using the NUMA architecture, there is a significant delay difference in CPU access to the memory of different nodes, and the AI acceleration device is usually physically connected to a specific CPU NUMA node through the PCIe bus. For example, when the NUMA node where the page-locked memory applied by the host does not match the NUMA node to which the AI acceleration device belongs, the data transmission must pass through the CPU interconnection channel, resulting in a sharp increase in transmission delay and a decrease in bandwidth utilization, thereby severely restricting the overall performance of the AI training and inference tasks.

[0015] To solve this problem, existing solutions usually require developers to explicitly query the hardware topology when applying for page-locked memory and manually specify the NUMA node ID that is affinity to the target AI acceleration device. However, this approach has obvious defects in practical applications. First, the development complexity is high, and developers need to deeply understand the topology details of the server, increasing the learning cost and coding burden. Second, the code coupling degree is high, and a large amount of hardware-related configuration code is mixed in the business logic code, which damages the portability and maintainability of the software. In addition, it is easy to make mistakes in complex scenarios, such as in the complex scenario of multi-card parallel or dynamic switching of devices, manual management of the correspondence between memory and AI acceleration devices is prone to configuration errors or performance fluctuations.

[0016] Considering the above problems, according to the embodiments of the present application, a method is proposed that can shield the underlying NUMA hardware topology details and automatically implement the binding and corresponding memory allocation of the AI accelerator to its best affinity NUMA node.

[0017] Specifically, the present application proposes a NUMA memory automatic allocation scheme based on AI accelerator context awareness. According to this scheme, the information about the current AI acceleration device contained in the thread context variable of the AI application runtime can be used to automatically identify and match the best affinity NUMA node associated with the current AI acceleration device, thereby simplifying the user application interface call while ensuring the shortest physical link and the lowest delay of data transmission.

[0018] According to embodiments of the present application, it is first required to establish the mapping relationship between each AI accelerator in the AI server and the corresponding best-affinity NUMA node. For example, in the system initialization phase of the AI server, the hardware topology of the server system can be automatically scanned, and by reading the operating system interface (such as the system file system (sysfs) provided by the Linux kernel) or the hardware driver structure, a mapping table showing the mapping relationship between the identity (ID) of the AI accelerator in the AI server and the ID of the best-affinity NUMA node of the AI accelerator can be constructed. The mapping table can record the ID of each AI accelerator and the ID of its corresponding best-affinity NUMA node, which can include the best NUMA node core in the AI server determined based on the physical connection with the AI accelerator and the communication path condition, and the memory directly connected to the NUMA node core. For example, in the embodiment shown in FIG. 1, the mapping table can establish a mapping relationship between the ID of the AI accelerator 1 and the ID of the NUMA node 1 to indicate that the best-affinity NUMA node of the AI accelerator 1 is the NUMA node 1; and establish a mapping relationship between the ID of the AI accelerator 2 and the ID of the NUMA node 2 to indicate that the best-affinity NUMA node of the AI accelerator 2 and the AI accelerator 3 is the NUMA node 2. Figure 1

[0019] In addition, according to embodiments of the present application, the context information of the currently running thread can be maintained in the runtime software stack of the AI server, for example, which can include a thread-local context variable for indicating the ID of the AI accelerator associated with the current thread. The AI accelerator associated with the current thread is, for example, the AI accelerator that the current thread is operating, i.e., in a multi-threaded programming environment, the acceleration device that the current thread is bound to or explicitly specified for performing AI computing tasks. In some embodiments, the initial value of the context variable in the initial state is 0 by default, and the user (such as the developer of the AI server system) can switch the AI accelerator associated with the current thread by calling a standard device setting interface (such as setDevice(int device_id)). Accordingly, the context variable of the current thread can be updated based on the ID of the AI accelerator set by the user through the device setting interface.

[0020] According to embodiments of the present application, based on the aforementioned mapping table about the AI accelerator and its best-affinity NUMA node and the local context variable of the current thread maintained in the runtime software stack of the AI server, an AI accelerator context-aware NUMA memory automatic allocation scheme can be implemented. This memory allocation scheme will be described in detail below with reference to Figure 2

[0021] Figure 2 ​​A schematic flowchart of a method 200 for NUMA memory allocation in an AI server is shown, according to an embodiment of the present application. The method 200 can be performed by a system or device for NUMA memory allocation in an AI server. In Figure 2 In the embodiment shown, the method 200 can include operations 210-240. In other embodiments, a method for NUMA memory allocation in an AI server can include more or fewer or different operations. The present disclosure is not limited in this regard.

[0022] At 210, a memory allocation request for allocating a NUMA memory for an AI accelerator in an AI server is received. According to some embodiments, the memory allocation request can include a page-locked memory allocation request. For example, when a user invokes a memory allocation interface to apply for a page-locked memory (e.g., invoking mallocHost(size_t size)), the user only needs to specify the size of the required memory, without the need to pass in any NUMA architecture related parameters.

[0023] At 220, an identity ID of the AI accelerator is obtained according to context information of a current thread running in the AI server and associated with the memory allocation request from the AI accelerator. As previously described, context information of the current thread running in the runtime software stack of the AI server is maintained, which includes a thread-local context variable indicating the ID of the AI accelerator associated with the current thread. Thus, upon receiving the memory allocation request from the AI accelerator, the ID of the AI accelerator can be obtained according to the context information of the current thread associated with the memory allocation request.

[0024] At 230, a best-affinity NUMA node corresponding to the AI accelerator is determined based on the ID of the AI accelerator and a mapping table of AI accelerator IDs and best-affinity NUMA node IDs constructed in the AI server.

[0025] As previously described, during the system initialization phase of the AI server, a mapping table showing the mapping relationship between the IDs of the AI accelerators in the AI server and the IDs of the best-affinity NUMA nodes of the AI accelerators can be constructed by automatically scanning the hardware topology of the server system, reading the operating system interface or hardware driver structure. The mapping table can record the ID of each AI accelerator and the ID of its corresponding best-affinity NUMA node. Thus, the system can automatically obtain the ID of the corresponding best-affinity NUMA node by looking up the mapping table according to the ID of the AI accelerator issuing the memory allocation request obtained through context perception, so as to determine the best-affinity NUMA node to be bound for the AI accelerator and perform memory allocation for the AI accelerator in the best-affinity NUMA node.

[0026] At 240, memory is allocated for the AI accelerator in the NUMA memory corresponding to the best affinity NUMA node. According to some embodiments, allocating memory for the AI accelerator in the NUMA memory corresponding to the best affinity NUMA node includes allocating page-locked memory for the AI accelerator in the NUMA memory corresponding to the best affinity NUMA node.

[0027] According to embodiments of the present application, the best affinity NUMA node can include a best NUMA node core in the AI server determined based on physical connection and communication path conditions with the AI accelerator, and memory directly connected to the NUMA node core. Thus, according to embodiments of the present application, the system dynamically identifies the affinity of the AI accelerator and the NUMA node at runtime, by default ensuring that the host memory and the on-board memory of the AI accelerator are in the same NUMA domain, thereby by default achieving the shortest path DMA transmission of data of the AI accelerator, avoiding the problem of delay multiplication and bandwidth attenuation caused by cross- NUMA node access.

[0028] In summary, embodiments according to the present application provide an AI accelerator context-aware NUMA memory automatic allocation scheme. According to the scheme, a deep fusion of hardware topology awareness and memory management is achieved, and significant results are achieved while maintaining an extremely simple application program interface design. First, the scheme builds a complete transparent abstraction layer, completely hiding the complexity of the NUMA asymmetric architecture and the heterogeneous hardware topology details from the application layer, allowing developers to not need to pay attention to the differences in the underlying physical connection, and achieving true software and hardware decoupling. Secondly, in terms of performance, the system dynamically identifies the affinity of the AI acceleration device and the NUMA memory at runtime, avoiding the problem of delay multiplication and bandwidth attenuation caused by cross- NUMA node access. In addition, according to the scheme, developers can continue to use the current general setDevice device setting interface without modifying the existing memory allocation logic or embedding any hardware-related code, and can automatically obtain the optimal memory access performance, thereby greatly improving development efficiency.

[0029] Embodiments of the present application can include a computer-readable storage medium. The medium stores instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0030] Embodiments of the present application can also include a computer program product. The computer program product includes instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0031] Figure 3 is a schematic diagram of a computing device that can implement embodiments according to the present application, which can be used to implement the NUMA memory allocation method for an AI server described above. As shown in Figure 3As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more hardware processors 304 coupled to bus 302 for processing information. The one or more hardware processors 304 may include, for example, one or more general-purpose microprocessors.

[0032] like Figure 3 As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302. Main memory 306 is used to store information and instructions executed by one or more processors 304, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in storage media accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0033] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0034] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0035] Each of the processes, methods, and algorithms described in the preceding sections can be embodied in, and fully or partially automated by, code modules executed by one or more computer systems or computer processors comprising computer hardware. The processes and algorithms can be implemented partially or wholly in application-specific circuitry.

[0036] When the functions disclosed herein are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium that can be executed by a processor. The specific technical solutions disclosed herein (all or part) or aspects that contribute to the current technology can be embodied in the form of a software product. The software product can be stored in a storage medium, and include a number of instructions that cause a computing device (which can be a personal computer, a server, a network device, etc.) to perform all or some of the steps of the methods of the embodiments of the present application. The storage medium can include a flash drive, a portable hard drive, a ROM, a RAM, a disk, an optical disk, other media that can operate to store program codes, or any combination thereof.

[0037] Particular embodiments further provide systems that include a processor and a computer readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to the steps of any of the methods disclosed above. Particular embodiments further provide computer readable storage mediums having stored thereon instructions that are executable by one or more processors to cause the one or more processors to perform operations corresponding to the steps of any of the methods disclosed above.

[0038] Embodiments disclosed herein can be implemented by a cloud platform, a server, or a group of servers (hereinafter collectively referred to as “service system”) that interact with clients. The clients can be end devices or clients registered by users at the platform, where the end devices can be mobile terminals, personal computers (PCs), and any devices that can install a platform application.

[0039] The various features and processes described above can be used independently of one another or in various combinations. All possible combinations and subcombinations are intended to fall within the scope of the disclosure. In addition, certain method or process blocks can be omitted in some implementations. The methods and processes described herein are also not limited to any particular order or sequence, and the blocks or states relating thereto can be performed in other orders or sequences that are appropriate. For example, blocks or states described as occurring sequentially in some implementations can be performed at the same time, or in other sequences. Additionally, some blocks or states can be performed multiple times, or not at all. The example blocks or states can be performed continuously, or in a way that is not continuous. Blocks or states can be added to or removed from the disclosed example embodiments. The example systems and components described herein can be configured differently than described. For example, components can be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0040] Various operations described herein can be performed, at least partially, by an algorithm. The algorithm can be included in program code or instructions stored in memory (e.g., the computer-readable storage medium described above). The algorithm can include a machine learning algorithm. In some embodiments, a machine learning algorithm can not explicitly cause a computer to perform a function but can learn from training data to produce a predictive model that performs the function.

[0041] Various operations described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to operate in a manner that will be consistent with the operations described herein. Whether temporarily or permanently configured, the processors can constitute processor-implemented engines that operate to perform one or more operations or functions described herein.

[0042] Similarly, the methods described herein can be at least partially processor- implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented engines. Also, one or more processors can operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface (API)).

[0043] The performance of certain of the operations can be distributed among the processors, not only residing within a single machine. In some example embodiments, the processors or processor-implemented engines can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented engines can be distributed across multiple geographic locations.

[0044] Throughout this specification, plural instances can implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations can be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations can be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component can be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0045] As used in this document, "or" refers to a nonexclusive or, unless specified otherwise or indicated by context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C." Additionally, "and" is both conjunctive and disjunctive unless otherwise indicated or indicated by context. Therefore, in this document, "A and B" means "A and B, conjunctively or A and B, disjunctively." Additionally, plural instances can implement a single instance to satisfy a quantity requirement. Also, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and particular operations are presented within the context of specific illustrative configurations. Other allocations of functionality are envisioned and can fall within the scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations can be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource can be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of the present disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. This summary is provided to introduce some aspects of the subject matter described in this document, but is not intended to define the scope of the subject matter. The scope of the subject matter is defined by the claims.

[0046] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0047] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0048] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for non-uniform memory access (NUMA) memory allocation in an artificial intelligence server, comprising: The memory allocation method comprises: receiving a memory allocation request for allocating NUMA memory for an artificial intelligence accelerator in the artificial intelligence server; obtaining an identity ID of the artificial intelligence accelerator according to context information of a current thread associated with the memory allocation request and running in the artificial intelligence server; determining an optimal affinity NUMA node corresponding to the artificial intelligence accelerator based on the ID of the artificial intelligence accelerator and a mapping table of artificial intelligence accelerator IDs and optimal affinity NUMA node IDs constructed in the artificial intelligence server; and allocating memory for the artificial intelligence accelerator in the NUMA memory corresponding to the optimal affinity NUMA node.

2. The memory allocation method of claim 1, wherein, The optimal affinity NUMA node comprises an optimal NUMA node core determined based on a physical connection with the artificial intelligence accelerator and a communication path condition in the artificial intelligence server, and memory directly connected to the NUMA node core.

3. The memory allocation method of claim 1, wherein, The mapping table is created by reading an operating system interface or a hardware driver interface of the artificial intelligence server in a system initialization stage of the artificial intelligence server.

4. The memory allocation method of claim 1, wherein, The context information of the current thread comprises a thread-local context variable used to indicate the ID of the artificial intelligence accelerator associated with the current thread.

5. The memory allocation method of claim 4, wherein, An initial value of the context variable is 0.

6. The memory allocation method of claim 4, wherein, The memory allocation method further comprises updating the context variable based on the ID of the artificial intelligence accelerator set by a user through a device setting interface.

7. The memory allocation method according to any one of claims 1 to 6, wherein, The memory allocation request comprises a page-locked memory allocation request, and allocating memory for the artificial intelligence accelerator in the NUMA memory corresponding to the optimal affinity NUMA node comprises allocating page-locked memory for the artificial intelligence accelerator in the NUMA memory corresponding to the optimal affinity NUMA node.

8. The memory allocation method of claim 7, wherein, The artificial intelligence accelerator accesses the page-locked memory through a direct memory access engine.

9. A computer-readable storage medium storing instructions, the instructions comprising: The instructions, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-8.

10. A computer program product comprising instructions, characterized in that, The instructions, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-8.

Images