COMPUTING DEVICE, SERVER, DATA PROCESSING METHOD, AND STORAGE MEDIUM

Abstract

The present application provides a computing device, a server, a data processing method, and a storage medium, which belong to the computer technology field, and the computing device includes a central processing unit (CPU), an accelerator, and a first high-speed serial computer expansion bus standard PCIe exchanger, the accelerator is connected to a first downlink port of the first PCIe exchanger, an uplink port of the first PCIe exchanger is connected to the CPU, each port of the first PCIe exchanger supports a computing high-speed link (CXL) protocol, and the accelerator is configured to access a host memory according to the CXL protocol.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to a Chinese patent application filed with the China Patent Office on December 29, 2023, bearing application number 202311850460.9 and entitled "Computing device, server, data processing method and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to a computing device, a server, a data processing method, and a storage medium. [Background technology]

[0003] In recent years, the widespread application of artificial intelligence (AI) has brought many technical challenges, especially in meeting the demands of machine learning and deep learning. Traditional central processing units (CPUs) and graphics processing units (GPUs) can face performance bottlenecks when processing complex AI tasks, and the bottleneck is not caused by the computing power but by the memory of accelerators such as GPUs. Therefore, how to provide an architecture or method for expanding accelerator memory becomes a technical problem that needs to be solved immediately. Summary of the Invention

[0004] In a first aspect, according to an embodiment of the present application, there is provided a computing device, comprising: a central processing unit (CPU), an accelerator, and a first high-speed serial computer expansion bus standard (PCIe) switch; the accelerator is connected to a first downlink port of the first PCIe switch; The uplink port of the first PCIe switch is connected to the CPU; Each port on the first PCIe exchanger supports the Compute High-Speed ​​Link (CXL) protocol. The accelerator is configured to perform access operations on the host memory based on the CXL protocol.

[0005] In a second aspect, according to an embodiment of the present application, there is further provided a server, comprising a computing device as in any of the first aspects above.

[0006] In a third aspect, according to an embodiment of the present application, there is further provided a data processing method based on a computing device such as any of the first aspect, comprising: sending a first data request message to the host memory according to the CXL protocol by the accelerator; The host memory responds to the first data request message by transmitting the first task data to the accelerator via the first PCIe switch.

[0007] In a fourth aspect, according to an embodiment of the present application, there is further provided a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions, when executed by a processor, realizing a data processing method such as any of the third aspect above.

[0008] The details of one or more embodiments of the application are set forth in the drawings and description below. Other features and advantages of the application will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0009] In order to more clearly describe the technical solutions in the present application or related art, the following briefly introduces drawings that need to be used in the description of the embodiments or related art. It is obvious that the drawings in the following description are some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without any creative work. [Figure 1] 1 is a structural schematic diagram of a computing device according to an embodiment of the present application; [Figure 2]1 is a schematic diagram of a basic architecture in which a CPU and a GPU according to an embodiment of the present application realize use of host memory by the GPU via a PCIe / CXL interface. [Figure 3] FIG. 1 is a basic architecture diagram for implementing unified memory between a CPU and an FPGA according to an embodiment of the present application. [Figure 4] FIG. 1 is a schematic diagram illustrating a GPU accessing a unified memory using a Switch with CXL functionality according to an embodiment of the present application. [Figure 5] FIG. 1 is a schematic diagram of a direct memory access architecture according to an embodiment of the present application. [Figure 6] FIG. 1 is an expanded schematic diagram of a memory direct access architecture according to an embodiment of the present application. [Figure 7] 1 is a schematic diagram 1 of a memory pool extension architecture according to an embodiment of the present application. [Figure 8] 2 is a schematic diagram 2 of a memory pool extension architecture according to an embodiment of the present application. [Figure 9] 1 is a schematic diagram illustrating connections between eight GPUs, GPU0 to GPU7, according to an embodiment of the present application. [Figure 10] FIG. 2 is a structural schematic diagram of a server according to an embodiment of the present application; [Figure 11] 1 is a schematic flowchart of a data processing method according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0010] In order to clarify the objectives, technical solutions and advantages of the present application, the following will clearly and completely describe the technical solutions of the present application with reference to the drawings in the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, but not all of the embodiments, and all other embodiments obtained by those skilled in the art based on the embodiments of the present application without any creative work fall within the scope of protection of the present application.

[0011] The terms "first," "second," etc., used herein are used to distinguish between similar objects and are not intended to describe a particular order or chronological order. It should be understood that terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application may be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first" and "second" generally refer to one type and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "or" in this application refers to at least one of the connected objects. For example, "A or B" covers three types of solutions, i.e., Solution 1: including A and not including B; Solution 2: including B and not including A; and Solution 3: including both A and B. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0012] The term "instruction" in this application may be a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). Here, a direct instruction can be understood as an instruction sent by a sender that clearly notifies a receiver of specific information, an operation that needs to be performed, or a requested result, and an indirect instruction can be understood as a receiver determining corresponding information based on the instruction sent from the sender, or making a judgment and then determining an operation that needs to be performed or a requested result based on the judgment result.

[0013] In recent years, the widespread application of artificial intelligence (AI) has brought many technical challenges, especially in meeting the demands of machine learning and deep learning. Traditional central processing units (CPUs) and graphics processing units (GPUs) may face performance bottlenecks when processing complex AI tasks, which generally require large amounts of computing resources and memory capacity.

[0014] For AI applications, existing hardware can generally handle some tasks, but when processing large-scale, complex AI workloads, it may face performance and memory limitations. AI models are becoming increasingly large and complex, resulting in greater computational and storage demands. This places greater demands on hardware. In addition to the challenge of computing efficiency, AI training also faces storage and bandwidth issues. Deep learning models generally have a large number of parameters and require large amounts of memory to store and process data. At the same time, high memory bandwidth is also crucial due to the massively parallel computing demands of deep learning models.

[0015] To address these challenges, many hardware manufacturers are offering new solutions, including GPUs with higher memory capacities and dedicated AI accelerators. High-bandwidth memory technologies, such as High Bandwidth Memory (HBM) and Graphics Double Data Rate (GDDR), are also widely adopted.

[0016] The amount of computation required for AI training is increasing significantly every year, and the future bottleneck of AI training will not be limited by computing power, but by GPU memory. Therefore, how to provide an architecture or method for expanding GPU memory has become an urgent technical problem to be solved. The computing device, server, data processing method, and storage medium of the present application will be described below with reference to FIGS.

[0017] The computing device of the present application realizes a GPU memory expansion architecture and solves the problem of GPUs encountering a memory wall. This application provides a memory scale-out architecture that utilizes CXL (Compute Express Link) high-speed interconnect technology. Memory scale-out is a method for expanding the available memory capacity in a computing system. When a computing system needs to process large amounts of data or execute memory-intensive tasks, the memory capacity of a single node may become insufficient. Memory scale-out aims to connect multiple computing nodes (generally multiple computing devices or servers) to form a large cluster, share their memory resources, and thereby expand the available memory capacity.

[0018] Below we introduce CXL technology. CXL (Compute Express Link) is a high-speed interconnect technology that aims to solve the challenge of connecting memory and accelerators in data centers and computing systems. CXL is an open standard, jointly promoted by an alliance of computer hardware manufacturers. CXL technology was initially established to allow CPUs and AFUs (accelerated functional units) to share memory, and further realizes memory interconnect technology between processors (e.g., CPUs, ASICs, FPGAs, etc.).

[0019] CXL3.0 features include: Point-to-point information transmission (Peer-to-Peer messaging) between peripheral devices, i.e., providing a Direct Memory Access (DMA) transmission architecture between peripheral devices without going through the CPU. It can be combined with an enhanced hardware consistency mechanism, allowing the memory area of ​​a peripheral device to be shared and used by multiple host CPUs simultaneously. CXL3.0 is the latest version of CXL technology, and it is an improvement and extension based on the previous version. CXL3.0 has the following main characteristics:

[0020] High Bandwidth and Low Latency: CXL 3.0 provides high bandwidth and low latency data transmission capabilities, allowing data to be transmitted more quickly from storage devices to accelerators or other processing units, thereby improving system performance. Memory Expansion: CXL 3.0 supports memory expansion, allowing multiple devices to share physical memory, thereby increasing available memory capacity, which is extremely useful for processing large datasets and memory-intensive tasks.

[0021] Computational Acceleration: CXL 3.0 supports efficient connection with computational accelerators (e.g., GPUs, FPGAs, etc.), allowing them to better cooperate with the main processor and storage devices to accelerate computational tasks. Compatibility: CXL 3.0 is compatible with PCI Express (PCIe) and Memory CXL interconnect standards, allowing for relatively smooth upgrades and migrations of existing hardware and software.

[0022] CXL defines three types of peripheral applications: Type 1: This type operates using the CXL.io and CXL.cache protocols and is suitable for specialized accelerators that do not have their own independent memory. For example, some smart network cards or video accelerator cards can currently access the host CPU's memory using the CXL protocol, allowing these peripherals to share the memory and use it as a cache. The basic concept of a CXL Type 1 application device is similar to the architecture of a CPU with a built-in display device that can share the system's host memory, allowing peripherals to use the host memory in the same way. These peripherals do not need to have their own independent memory. CXL allows any PCIe peripheral that supports the CXL.io and CXL.cache protocols to use the system's host memory.

[0023] Type 2: Operates according to the CXL.io, CXL.cache, and CXL.mem protocols and is used for general-purpose accelerators with high-efficiency memory (GDDR or HBM memory), such as GPU cards, FPGAs, and ASIC-based accelerator cards. The CXL protocol provides two-way memory sharing between these peripherals and the host CPU, allowing peripherals to access the host CPU's memory, as well as the host CPU to access the peripheral's memory. CXL Type 2 applications share memory between the host CPU and peripherals, dynamically allocating memory resources between them and improving the overall system memory resource utilization efficiency.

[0024] Type 3: Operated by the CXL.io and CXL.mem protocols, such devices are memory expansion board cards based on Dynamic Random Access Memory (DRAM) or Storage Class Memory (SCM), and the host CPU can access the DRAM or non-volatile SCM memory installed on such memory expansion board cards via the CXL protocol.

[0025] 1 is a structural schematic diagram of a computing device according to an embodiment of the present application. As shown in FIG. 1, the computing device includes a central processing unit (CPU) 110, an accelerator 120, and a first high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) switch 130; The accelerator 120 is connected to a first downlink port of the first PCIe switch 130; The uplink port of the first PCIe exchange 130 is connected to the CPU 110; Each port of the first PCIe exchange 130 operates in a computing high-speed link CXL mode or a PCIe mode; The accelerator is configured to perform access operations on the host memory based on the CXL protocol.

[0026] Here, the CXL protocol specifically refers to the CXL.io and CXL.cache protocols. Preferably, the accelerator may be a GPU or other heterogeneous accelerator, such as a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), etc.

[0027] As will be understood, the accelerator is connected to the CPU via the first PCIe exchange, which allows the accelerator to directly access the host memory of the system based on the CXL protocol, where the access operation is to write data or read data.

[0028] Each port of the first PCIe exchange supports a computing high-speed link CXL protocol, and each port of the first PCIe exchange operates in a computing high-speed link CXL mode or a PCIe mode, that is, each port of the first PCIe exchange is a PCIe / CXL interface.

[0029] The following description will be given taking the accelerator as a GPU. Figure 2 is a basic architecture diagram illustrating an embodiment of the present application in which a CPU and a GPU use host memory via a PCIe / CXL interface. As shown in Figure 2, this is a CXL Type 1 application, and the GPU can access host memory through the CXL Type 1 application. This design provides several advantages.

[0030] Resource sharing: The GPU and the host CPU share the same memory space and can access data directly from the host memory, avoiding an extra copy of data from the host memory to the GPU-dedicated memory. Cost saving: GPU can directly expand and configure memory capacity through CXL.

[0031] Cache acceleration: By using host memory as cache memory, GPU data access can be accelerated and performance can be improved. The introduction of CXL technology further promotes cooperation between the GPU and the host CPU, providing more efficient memory access and sharing, further optimizing high performance computing applications.

[0032] In an embodiment of the present application, the accelerator is connected to the CPU via a first PCIe exchanger, which allows the accelerator to perform access operations to the host memory based on the CXL protocol, which allows the accelerator to directly operate the host memory, expand the accelerator's available memory, provide more efficient memory access and sharing, accelerate data access, and improve performance.

[0033] In some embodiments, the computing device further comprises a memory expansion unit; the memory expansion unit is connected to the second downlink port of the first PCIe switch; The local memory of the memory expansion unit and the host memory constitute a unified memory. The accelerator is configured to perform access operations on the unified memory based on the CXL protocol.

[0034] As can be understood, in this embodiment, the memory expansion unit is connected to the second downlink port of the first PCIe exchanger, the first downlink port of the first PCIe exchanger is connected to the accelerator, and the uplink port of the first PCIe exchanger is connected to the CPU, so that point-to-point communication can be realized between the accelerator and the memory expansion unit, and the memory expansion unit can perform access operations to the host memory, and accordingly, the CPU can perform access operations to the local memory of the memory expansion unit.

[0035] The local memory of the memory expansion unit and the host memory constitute a unified memory, and the accelerator can perform access operations to the unified memory based on the CXL protocol.

[0036] Below, we will introduce Unified Memory (Converged Memory). Converged Memory (Unified Memory) is a computing concept in which different types of memory technologies are integrated or combined into one single memory pool or architecture. This approach aims to address the limitations and challenges of traditional memory hierarchies, where different memory types, such as DRAM, SRAM (Static Random Access Memory), and NAND (Not AND) flash memory, are used for specific purposes, such as host memory, cache memory, and storage.

[0037] The unified memory concept creates one unified memory system that can provide better performance, energy efficiency, and simplified memory management. By combining multiple memory technologies, data can be more effectively shared and moved between different levels of the memory hierarchy, reducing data transmission demands between different memory types and potentially reducing latency.

[0038] Preferably, the memory expansion unit includes at least one first processing unit having an independent memory. Preferably, the first processing unit comprises any one or combination of a Field Programmable Gate Array (FPGA), a Complex Programmable Logic Device (CPLD), a Programmable Logic Device (PLD), an Application Specific Integrated Circuit (ASIC), a Generic Array Logic (GAL), a System on Chip (SOC), a Software Defined Architecture (SDI) device, and an Artificial Intelligence (AI) device.

[0039] The following description will be given taking the first processing unit as an FPGA. Figure 3 is a basic architecture diagram illustrating a unified memory between a CPU and an FPGA according to an embodiment of the present application. As shown in Figure 3, a typical CXL Type 2 application is bidirectional memory sharing between an FPGA and a host CPU. This allows devices such as FPGAs to have high-performance independent memory (e.g., GDDR or HBM memory) and share the memory with the CPU, achieving more efficient data transmission and processing. The CXL communication architecture allows the CPU and FPGA to achieve closer integration and cooperation, forming the CPU's host memory and the FPGA's optional memory into a unified memory (converged memory), improving the overall performance and flexibility of the system. This technology integrates and shares memory resources between the FPGA and the CPU. Conventionally, FPGAs and CPUs have independent memory spaces, and data must be explicitly copied or transferred between them. Converged memory technology realizes memory resource sharing at the hardware and software layers, allowing the FPGA and CPU to access the same physical memory, thereby achieving more efficient data transmission and sharing.

[0040] Converged Memory technology offers the following benefits: Data sharing: FPGA and CPU can directly share the same memory space, eliminating the need to duplicate data. In this way, data can be transmitted between FPGA and CPU more efficiently, reducing data transmission delays and copying overhead.

[0041] Flexibility: FPGA can access the memory space of the CPU, allowing it to directly process data stored in the CPU, which provides greater flexibility and possibilities for cooperation between FPGA and CPU. Memory Management: Converged Memory enables more unified memory management, simplifying data transmission and management between FPGA and CPU, and improving overall system performance.

[0042] Resource saving: Since they share the same memory resources, the physical memory capacity required for the system can be reduced, saving on hardware costs. This application utilizes a CXL-enabled switch (also referred to as a switch or exchange) to simultaneously achieve unified memory functionality between the CPU and FPGA and direct access by the GPU to the same memory space shared by the CPU and FPGA. Figure 4 is a schematic diagram of a CXL-enabled switch used in an embodiment of this application to enable a GPU to access unified memory. As shown in Figure 4, the switch's uplink port is connected to the CPU root complex, and its downlink port is connected to the GPU and FPGA. The GPU can use the switch to access FPGA memory to address the GPU's lack of local memory. The CXL 3.0 protocol is added, and its physical layer uses the PCIe 6.0 interface. PCIe 6.0 is the latest version of the PCIe (Peripheral Component Interconnect Express) bus, with a data rate of 16 GT / s (gigabits per second). That is, PCIe 6.0 has a speed of 16 gigabits per second, which is faster than previous versions (such as PCIe 5.0 and PCIe 4.0). The CXL protocol allows for fast memory access and sharing between the host CPU and peripheral devices.

[0043] The characteristics of the architecture in Figure 4 can be used extensively to realize the expansion of the memory pool. The computing device according to an embodiment of the present application further includes a memory expansion unit, which is connected to the second downlink port of the first PCIe switch. The local memory of the memory expansion unit and the host memory form a unified memory, thereby allowing the accelerator to access the memory space shared by the CPU and the memory expansion unit, reducing data transmission delays and copying overhead. Since the CPU and the memory expansion unit share the same memory resources, the physical memory capacity required for the system can be reduced, hardware costs can be saved, memory management can be more unified, and the overall performance of the system can be improved.

[0044] In some embodiments, the memory expansion unit includes a memory expansion board card. Specifically, the memory expansion board card is connected to the second downlink port of the first PCIe switch; The local memory and host memory of the memory expansion board card form a unified memory. The accelerator is configured to perform an access operation on the unified memory based on the CXL protocol, where the CXL protocol includes the CXL.io and CXL.mem protocols, and when operating according to the CXL.io and CXL.mem protocols, the local memory of the memory expansion board card and the host memory constitute a unified memory, thereby allowing the accelerator to access the local memory mounted on the memory expansion board card.

[0045] Preferably, the memory expansion board card is equipped with a dynamic random access memory DRAM or a storage class memory SCM. In a computing device according to an embodiment of the present application, the memory expansion unit may be a memory expansion board card, and the accelerator can access a unified memory consisting of the local memory of the memory expansion board card and the host memory, thereby expanding the memory available to the accelerator, providing more efficient memory access and sharing, accelerating data access, and improving performance.

[0046] In some embodiments, the computing device further includes a Non-Volatile Memory Express (NVMe) solid state drive (SSD), the NVMe solid state drive being connected to a third downlink port of the first PCIe switch; The accelerator is configured to perform access operations on NVMe solid-state drives based on the CXL protocol.

[0047] Specifically, PCIe switches support point-to-point (P2P) communication between NVMe SSDs and GPUs, allowing data to be transferred directly between the NVMe SSD and GPU without going through the host CPU. This direct communication path significantly reduces data transfer latency and CPU overhead, thereby improving overall system performance for certain workloads. In conventional PCIe configurations, data transfer between an NVMe SSD and a GPU requires data to first be sent from the NVMe SSD to host memory and then from the host memory to the GPU's memory. This process involves multiple jumps, increasing latency. However, PCIe switches' P2P communication allows the NVMe SSD and GPU to exchange data directly through the PCIe switch without going through the host memory or CPU. This P2P communication is particularly applicable to tasks that require frequent data exchange between the NVMe SSD and GPU, such as data-intensive workloads such as artificial intelligence, machine learning, and high-performance computing. By enabling direct communication, the P2P functionality of PCIe Switch can enhance the overall efficiency and performance of data-intensive applications and reduce data movement bottlenecks.

[0048] In some embodiments, the computing device further includes a network interface controller (NIC), the NIC connected to a fourth downlink port of the first PCIe switch, and the accelerator configured to exchange data with the NIC based on the CXL protocol.

[0049] Specifically, the PCIe switch hardware itself has a point-to-point (Peer to Peer) communication function and supports point-to-point shortest path transmission. The accelerator can transmit data directly to the NIC via the Ethernet using the Switch P2P function. The shortest path transmission only passes through the PCIe switch and does not cause any CPU load, reducing the waste of system resources.

[0050] 5 is a schematic diagram of a direct memory access architecture according to an embodiment of the present application. As shown in FIG. 5, the architecture includes a motherboard (MB) and an artificial intelligence compliance (AI Compliance). Here, the motherboard MB includes a CPU, four sets of PCIe switches (only two sets are shown in Figure 5), eight sets of FPGAs (only four sets are shown in Figure 5), and two sets of MCIO (Mini Cool Edge IO) connectors (only one set is shown in Figure 5).

[0051] Here, the CPU must be an X86 platform and a model number that supports the CXL function. The four sets of PCIe switches include PCIe Switch1, Switch2, Switch3, and Switch4. The PCIe switch downlink port must support operation in either CXL mode or PCIe mode, and is suitable for data-intensive workloads such as AI. The model number is Broadcom's Atlas 3 series PCIe switch.

[0052] In Figure 5, eight sets of FPGAs are connected to the PCIe switch downlink ports, enabling high-speed, low-latency, and highly efficient data transmission and communication between the CPU and FPGA. Each FPGA has multiple sets of independent x16 PCIe endpoints. Each set of independent PCIe endpoints can be considered as a single PCIe device. The computing device has a built-in 8-channel DMA (Direct Memory Access) controller, supports DDR5 and LPDDR5 interfaces, and supports CXL.

[0053] Two pairs of MCIO x16 connectors are used for memory expansion (scale-up). In Figure 5, UP stands for upstream (uplink port), DP stands for downstream (downlink port), F stands for fabric port, and EP stands for endpoint. Here, the root complex is one of the key components in the PCI Express (PCIe) bus architecture. It is a logical node used to manage the entire PCIe system. The root complex is generally implemented by a CPU, a group of computing devices, or an FPGA. In a PCIe system, each device must be connected to a PCIe bus. The root complex is the starting point and central node of the PCIe bus architecture, and is responsible for managing all PCIe devices and endpoints, including bus bandwidth allocation and management, transmission control, and data routing.

[0054] Each pair of PCIe switch downlink ports is configured with dual GPUs and dual NICs. This configuration is used in the machine learning training phase, where models need to be optimized and trained using large training datasets. The training phase requires relatively high network bandwidth speeds because a large amount of training data needs to be transmitted from a storage device (e.g., cloud storage or local storage) to a training server or device. The amount of data transmitted during the training process is generally very large, and particularly in distributed training, data needs to be frequently exchanged between multiple devices. Therefore, high-speed, low-latency network bandwidth is very important and can significantly affect training speed and efficiency. In large-scale machine learning training, high-bandwidth network connections and dedicated network architectures are generally used to support large-scale data transmission.

[0055] The machine learning training phase has relatively high requirements for network bandwidth speed. When designing and configuring a machine learning system, it is necessary to consider the network bandwidth demands based on specific application scenarios and data scales to ensure system stability and high efficiency.

[0056] The PCIe switch hardware in Figure 5 itself has point-to-point (Peer-to-Peer) communication capabilities and supports shortest-path transmission between points. (2) in Figure 5 allows the GPU to transmit data directly between the NIC and Ethernet using the switch's P2P function. The shortest-path transmission passes only through the PCIe switch and does not burden the CPU, reducing system resource waste. (3) in Figure 5 also uses a PCIe switch to support point-to-point (P2P) communication between the NVMe SSD and the GPU, allowing data to be transmitted directly between the NVMe solid-state drive (NVMe SSD) and the graphics processor (GPU) without going through the host CPU. This direct communication path significantly reduces data transmission latency and CPU overhead, thereby improving overall system performance for a given workload. In a conventional PCIe configuration, data transmission between the NVMe SSD and the GPU requires data to first be sent from the NVMe SSD to host memory, and then from the host memory to the GPU's memory. This process involves multiple jumps, increasing latency. However, with PCIe Switch's P2P communication, the NVMe SSD and GPU can exchange data directly through the PCIe Switch without going through the host memory or CPU. This P2P communication is particularly applicable to tasks that require frequent data exchange between the NVMe SSD and GPU, such as data-intensive workloads such as artificial intelligence, machine learning, and high-performance computing. By enabling direct communication, the PCIe Switch's P2P function can enhance the overall efficiency and performance of data-intensive applications and reduce data movement bottlenecks.

[0057] Figure 5 (1) is an application of CXL Type 1, which allows the GPU to directly access the system's host memory. (4)This configuration allows the GPU and FPGA to communicate directly through point-to-point (P2P) communication, enabling them to access each other's memory directly without the involvement of the host CPU. Specifically, when the GPU needs to access the FPGA's memory, it can directly transmit data to the FPGA's memory locations via the PCIe switch. Similarly, when the FPGA needs to access the GPU's memory, it can directly read data from the GPU's memory via the PCIe switch. This direct communication method reduces data transmission delays and CPU involvement, improving memory access efficiency and performance. Such P2P memory access between the GPU and FPGA is generally applied to applications requiring efficient data exchange and sharing, such as artificial intelligence, machine learning, and high-performance computing. Enabling memory access between the GPU and FPGA via the PCIe switch provides a high-bandwidth, low-latency communication channel, further enhancing the system's computational and data processing capabilities.

[0058] Also, in Figure 5, a unified memory architecture can be realized between the FPGA and host memory by applying CXL Type 2, and the FPGA accelerates two-way memory sharing between the device and the host CPU. This allows devices such as FPGAs to have high-performance independent memory (e.g., GDDR or HBM memory) and share the memory with the CPU, achieving more efficient data transmission and processing. The CPU's host memory and the FPGA's optional memory can be unified into a converged memory, creating a huge memory pool.

[0059] In some embodiments, the computing device further comprises a first MCIO connector; The first MCIO connector is connected to the Fabric port of the first PCIe switch and is used to connect to the second MCIO connector of another computing device.

[0060] The main function of the Fabric port is to support mutual transmission between PCIe switches and to provide I / O sharing and DMA functions with characteristics such as non-blocking and linear acceleration.

[0061] FIG. 6 is an expanded schematic diagram of a memory direct access architecture according to an embodiment of the present application. (5) This is the process of realizing point-to-point transmission of GPUs between two systems using the characteristics of Fabric link, and achieving dynamic adjustment and workload balancing between multiple GPU devices. When performing parallel computing, it is possible to allocate workloads to multiple GPUs to accelerate processing, but the performance and resources of different GPUs may differ, so dynamic adjustment is required to ensure optimal performance and efficiency. (6) As shown in Figure 1, the application of Fabric Link and CXL also enables point-to-point communication between the FPGAs of System 1 and System 2, and integrates the memory pools of the two systems. This design enables efficient data exchange and sharing, thereby improving the performance and efficiency of the entire system. This means that the FPGAs of the two systems can directly access each other's memory, realizing sharing and memory resource sharing, and thereby managing and utilizing memory more effectively.

[0062] This application provides yet another architecture for memory expansion. In some embodiments, the memory expansion unit includes a memory pool, the memory pool including at least one second processing unit having an independent memory and at least one second PCIe switch; The downlink port of the second PCIe exchanger is connected to the first endpoint port of the second processing unit, and the second endpoint port of the second processing unit is connected to the fifth downlink port of the first PCIe exchanger.

[0063] 7 is a schematic diagram 1 of a memory pool extension architecture according to an embodiment of the present application. The memory pool extension architecture includes: 1. MB (Motherboard) Here, the CPU must be an X86 platform and a model number that supports the CXL function.

[0064] 2. AI Compliance It includes four sets of PCIe Switches: PCIe Switch1, Switch2, Switch3, and Switch4. Each pair of PCIe Switch downlink ports is configured with dual GPUs and dual NICs. Each pair of PCIe Switches is paired with four pairs of NVMe SSDs, making the integrated system have a total of 16 pairs of NVMe SSD storage units.

[0065] 3. Memory Pool: Four sets of FPGAs (e.g., FPGA0, FPGA1, FPGA2, FPGA3, which are connected to PCIe Switch downlink ports), Two pairs of PCIe Switches (e.g., Switch5, Switch6), Includes two sets of MCIO x16 connectors used for FPGA scale-up.

[0066] As shown in Figure 7, the FPGA in this architecture has three sets of PCIe endpoints. This architecture uses the downlink ports of PCIe Switch 0, PCIe Switch 1, PCIe Switch 2, and PCIe Switch 3 to connect to one set of endpoint ports among FPGA 0, FPGA 1, FPGA 2, and FPGA 3, respectively, and is connected to two sets of endpoints among FPGA 0, FPGA 1, FPGA 2, and FPGA 3 via the downlink ports of PCIe Switch 5 and PCIe Switch 6. PCIe Switch 5 and PCIe Switch 6 each have one set of fabric ports connected to an MCIO connector, allowing system connection via a cable and serving as a bridge for memory scale-up.

[0067] In an embodiment of the present application, the memory expansion unit may be a memory pool, which includes at least one second processing unit having an independent memory and at least one second PCIe exchange, and the accelerator can access the local memory of the second processing unit in the memory pool and a unified memory composed of the local memory of the first processing unit and the host memory, thereby expanding the memory available to the accelerator, providing more efficient memory access and sharing, accelerating data access, and improving performance.

[0068] Preferably, the second processing unit includes any one or combination of a field programmable gate array (FPGA), a complex program logic device (CPLD), a programmable logic device (PLD), an application specific integrated circuit (ASIC), a general purpose array logic (GAL), a system on chip (SOC), a software defined architecture (SDI) device, and an artificial intelligence (AI) device.

[0069] Preferably, the memory pool further includes a third MCIO connector, which is connected to a Fabric port of the second PCIe switch and is used to connect to a fourth MCIO connector of another computing device.

[0070] 8 is a schematic diagram 2 of a memory pool expansion architecture according to an embodiment of the present application. As shown in FIG. 8, four computing devices (also called systems) can be connected in series in a daisy-chain fashion using MCIO to form a huge memory pool, and the four computing devices can expand shared memory and dynamically procure memory. In this case, four host systems can be interconnected via a star link, realizing a topology in which large memory demands are allocated to any of the systems via the Fabric port, achieving dynamic resource allocation, further increasing the computing power of the system computing units, and maximizing and optimizing resources.

[0071] Preferably, the FPGA uses dynamic partitioning technology to divide the internal resources of the FPGA into different regions, or uses a direct memory access DMA controller to realize dynamic memory allocation and data transfer. The FPGA can realize the function of memory allocation, which means dynamically allocating and managing memory resources in the FPGA, and different modules or subsystems can share and use these memories. The memory allocation function of the FPGA should be realized in the following manner.

[0072] a) Dynamic partition: FPGA can use dynamic partition technology to divide its internal resources into different areas, some of which can be used as FPGA main memory, and the remaining areas can be partitioned and allocated to each GPU or CPU according to the system demand during operation.

[0073] b) DMA Controller: FPGA can use DMA (Direct Memory Access) controller to realize dynamic memory allocation and data transfer. The DMA controller can transfer data from the memory area of ​​FPGA to other devices or modules, and can also transfer data from other devices or modules to the memory area of ​​FPGA, realizing dynamic memory sharing and use.

[0074] The FPGA used in this architecture has Multi-Channel DMA IP for PCI Express, which mainly consists of H2DDM (Host-to-Device Data Mover) and D2HDM (Device-to-Host Data Mover) modules, which also provide DMA-bypass functionality for the host, thereby performing PIO read / write operations to the device memory.

[0075] The MCDMA engine operates on software DMA queues, which are used to transfer data between the local FPGA and the host. Each queue element is a software descriptor written by the driver / software. The hardware reads and executes the queue descriptors. The hardware can support up to 2K DMA channels. For each channel, an independent queue is used to perform DMA read and write operations.

[0076] The H2DDM module transfers data from host memory to local memory via PCIe hardware IP and Avalon-MM Write Master / Avalon-ST Source interface. The D2HDM module transfers data from device memory to host memory. It receives data from the user logic via the Avalon-MM Read Master / Avalon-ST Sink interface, and generates Mem Wr TLPs based on the descriptor information (e.g., PCIe address (destination), data size, and MPS value), moves the data to the host, and transfers it to the receive buffer area of ​​the host memory.

[0077] In some embodiments, the accelerator is a graphics processing unit (GPU). FIG. 9 is a schematic diagram of a connection of eight GPUs, GPU0 to GPU7, according to an embodiment of the present application, which is called a star-link topology connection method, and the interconnection is realized by PCIe Switch0, PCIe Switch1, PCIe Switch2, and PCIe Switch3, which are not shown in FIG. The downlink ports of PCIe Switch 0 are connected to GPU 0 and GPU 1. The PCIe Switch hardware itself supports point-to-point (Peer-to-Peer) communication, supporting point-to-point shortest-path transmission. GPU 0 and GPU 1 can share their own memory directly through P2P. GPU 2 and GPU 3 are downlink ports of PCIe Switch 1, GPU 4 and GPU 5 are downlink ports of PCIe Switch 2, and GPU 6 and GPU 7 are downlink ports of PCIe Switch 3. They can share each other's memory through P2P. Therefore, as shown in Figure 9, each group has interconnected paths. These paths can share the eight sets of GPU memory. GPU memory sharing is the process of sharing the same block of memory among multiple computing units (typically threads) when performing general-purpose computing on a GPU. GPUs generally have a large number of computing cores and can run multiple computing units simultaneously to accelerate processing.

[0078] GPU memory sharing is achieved by using shared memory or global memory in GPUs. These memory areas can be accessed and operated by different computing units, allowing them to share data during computation. This design can avoid unnecessary data copying and transmission, thereby improving computation efficiency and performance.

[0079] However, GPU memory sharing requires special attention to synchronization and race condition issues, because simultaneous access of multiple computing units to the shared memory can lead to data races and inconsistency situations. To ensure data correctness, access to the shared memory must be controlled using synchronization mechanisms (e.g., exclusive locks, signaling, etc.) to ensure data consistency.

[0080] GPU memory sharing is widely applied in many GPU computing applications, such as machine learning, deep learning, scientific computing, image processing, etc. By properly designing and managing memory sharing, GPU computing efficiency can be maximized, thereby achieving faster and more efficient operations.

[0081] The computing device according to the embodiment of the present application has the following beneficial effects: 1) Resource sharing: The GPU and host CPU share the same memory space and can directly access data from the host memory, avoiding redundant copying of data from the host memory to the GPU-dedicated memory. 2) Cost savings: The GPU and CPU can directly expand and allocate memory capacity through CXL. 3) Cache acceleration: Using host memory as cache memory can accelerate GPU data access and improve performance. 4) The introduction of CXL technology further promotes collaboration between the GPU and host CPU, providing more efficient memory access and sharing, further optimizing high-performance computing applications. 5) Using the CXL protocol, memory between the host CPU and accelerators can be dynamically allocated. This means that if an accelerator needs more memory space to process a specific task, it can request more memory from the host CPU, and the host CPU can allocate some memory to the accelerator, allowing memory resources to be shared in an optimal manner. 6) The implementation of memory pooling through the CXL protocol can significantly improve the overall efficiency of the system. The accelerator can directly access the memory of the host CPU, avoiding the cumbersome data transmission and copying process, reducing the calculation and data processing time, and thereby accelerating the running speed of the system.

[0082] 10 is a structural schematic diagram of a server according to an embodiment of the present application. As shown in FIG. 10, a server 1010 includes a computing device 1020. For an understanding of the computing device, please refer to the description in the previous embodiment, and the description will be omitted here. For example, the computing devices provided in the above embodiments of the present application can be applied to AI servers, solving the problem of insufficient memory in computing units and meeting the demands of increasingly complex and massive AI models.

[0083] FIG. 11 is a schematic flowchart of a data processing method according to an embodiment of the present application. As shown in FIG. 11, the data processing method includes step 1110 and step 1120. In step 1110, the accelerator sends a first data request message to the host memory according to the CXL protocol.

[0084] In this step, the accelerator is connected to the CPU through a first PCIe exchange, so that the accelerator directly performs an access operation to the host memory of the system based on the CXL protocol, so that when necessary, the accelerator sends a first data request message to the host memory based on the CXL protocol, and the first data request message is used to request first task data stored in the host memory.

[0085] Preferably, the CXL protocols herein are specifically the CXL.io and CXL.cache protocols. In step 1120, the host memory transmits the first task data to the accelerator via the first PCIe switch in response to the first data request message.

[0086] In this step, the host memory receives a first data request message sent from the accelerator, where the first data request message is used to request first task data stored in the host memory, and then, in response to the first data request message, obtains the first task data and sends the first task data to the accelerator via the first PCIe exchange.

[0087] In an embodiment of the present application, the accelerator can directly obtain data from the host memory based on the CXL protocol, expanding the accelerator's available memory, providing more efficient memory access and sharing, accelerating data access, and improving performance.

[0088] In some embodiments, the method comprises: the accelerator executing the computational task based on the first task data to generate first result data; The method further includes the step of the accelerator receiving the second data request message sent from the CPU and sending the first result data to the host memory.

[0089] As can be understood, the accelerator is connected to the CPU through a first PCIe exchange, which allows the accelerator to perform an access operation to the host memory according to the CXL protocol; after receiving the first task data, the accelerator can execute the corresponding calculation task, generate first result data, and store the first result data in the host memory for use by the CPU or other devices when executing other calculation task demands; the accelerator can store the first result data in its local memory; the CPU actively obtains the first result data from the accelerator when necessary; the accelerator receives a second data request message sent from the CPU and sends the first result data to the host memory, which allows the CPU to also use the accelerator's memory, thereby providing more efficient memory access and sharing.

[0090] In an embodiment of the present application, the accelerator performs a computation task based on the first task data, generates first result data, and sends the first result data to the host memory, and the CPU can also use the computation result of the accelerator or the data stored in the memory, thereby providing more efficient memory access and sharing.

[0091] In some embodiments, the method comprises: the accelerator sending a third data request message to the memory expansion unit according to the CXL protocol; The method further includes the step of the memory expansion unit transmitting the second task data to the accelerator via the first PCIe switch in response to the third data request message.

[0092] The operation of the method of this embodiment requires the presence of a memory expansion unit in the computing device. The memory expansion unit is connected to a second downlink port of the first PCIe switch, the local memory of the memory expansion unit and the host memory form a unified memory, and the accelerator is configured to perform an access operation to the unified memory based on the CXL protocol. It can be understood that the accelerator can perform an access operation to the unified memory based on the CXL protocol, that is, the accelerator can obtain data from the local memory of the memory expansion unit. Specifically, the accelerator sends a third data request message to the memory expansion unit based on the CXL protocol, and the memory expansion unit sends second task data to the accelerator via the first PCIe switch in response to the third data request message.

[0093] In an embodiment of the present application, the local memory of the memory expansion unit and the host memory constitute a unified memory, and the accelerator can perform access operations to the unified memory based on the CXL protocol, allowing data to be more effectively shared and moved between different levels of memory hierarchy structures, reducing data transmission demands between different memory types and potentially reducing latency.

[0094] In some embodiments, the method comprises: The method further includes the accelerator storing the third task data in the memory expansion unit via the first PCIe exchange.

[0095] The accelerator can perform an access operation on the unified memory based on the CXL protocol, that is, the accelerator can store data in the local memory of the memory expansion unit according to the CXL protocol, specifically, the accelerator stores the third task data in the memory expansion unit through the first PCIe switch.

[0096] In an embodiment of the present application, the accelerator can access the memory space shared by the CPU and the memory expansion unit, reducing data transmission delays and copy overhead. Since the CPU and the memory expansion unit share the same memory resources, the physical memory capacity required for the system can be reduced, hardware costs can be saved, memory management can be more unified, and the overall performance of the system can be improved.

[0097] In another aspect, the present application further provides a computer-readable instruction product, which includes computer-readable instructions, and the computer-readable instructions can be stored in a non-transitory computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the computer can perform the embodiments of each of the above data processing methods, and the description thereof will be omitted here.

[0098] In yet another aspect, the present application further provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, perform the embodiments of the data processing methods described above, and further description thereof is omitted here.

[0099] It should be noted that the embodiments of the present application can be freely combined, interchanged, or performed independently, and do not need to be relied upon or depended upon in a fixed order of execution.

[0100] The above-described device embodiments are merely illustrative, and the units described as separate components may or may not be physically separate, and the components represented as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Depending on actual needs, some or all of the modules may be selected to achieve the objectives of the solutions of the present embodiments. Those skilled in the art can understand and implement the present embodiments without any creative effort.

[0101] From the above description of the embodiments, it is clear to those skilled in the art that each embodiment can be realized in the form of software and a necessary general-purpose hardware platform, and naturally can be realized in hardware. Based on this understanding, the above technical solutions can essentially or the part that contributes to the related art can be expressed in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and can contain multiple instructions and be used to make a computer device (which may be a personal computer, a server, a network device, etc.) execute the methods described in each embodiment or part of the embodiments.

[0102] Finally, it should be mentioned that the above embodiments are only for illustrating the technical solutions of the present application, and are not limiting thereof. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art will understand that the technical solutions described in the above embodiments may still be modified or equivalently substituted for some technical features thereof, and these modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. 1. A computing device, comprising: a central processing unit (CPU), an accelerator, and a first high-speed serial computer expansion bus standard (PCIe) switch; the accelerator is connected to a first downlink port of a first PCIe switch; an uplink port of the first PCIe switch connected to the CPU; Each port of the first PCIe switch supports a Compute High Speed ​​Link (CXL) protocol; and 10. The computing device, wherein the accelerator is configured to perform access operations on a host memory based on a CXL protocol.

2. further including a memory expansion unit; the memory expansion unit is connected to a second downlink port of the first PCIe switch; The local memory of the memory expansion unit and the host memory constitute a unified memory; and The computing device of claim 1 , wherein the accelerator is configured to perform access operations on the unified memory based on a CXL protocol.

3. 3. The computing device of claim 2, wherein the memory expansion unit includes at least one first processing unit having an independent memory.

4. 4. The computing device of claim 3, wherein the first processing unit comprises any one or combination of a field programmable gate array (FPGA), a complex program logic device (CPLD), a programmable logic device (PLD), an application specific integrated circuit (ASIC), a general purpose array logic (GAL), a system on a chip (SOC), a software defined architecture (SDI) device, and an artificial intelligence (AI) device.

5. 3. The computing device of claim 2, wherein the memory expansion unit comprises a memory expansion board card.

6. 6. The computing device of claim 5, wherein the memory expansion board card is equipped with a dynamic random access memory (DRAM) or a storage class memory (SCM).

7. further including a Non-Volatile Memory Host Controller Interface Specification (NVMe) solid state drive, the NVMe solid state drive connected to a third downlink port of the first PCIe switch; and 2. The computing device of claim 1, wherein the accelerator is configured to perform access operations on the NVMe solid-state drive based on a CXL protocol.

8. further comprising a network interface controller (NIC); the NIC is connected to a fourth downlink port of the first PCIe switch; and 10. The computing device of claim 1, wherein the accelerator is configured to exchange data with the NIC based on a CXL protocol.

9. further including a first MCIO connector; 2. The computing device of claim 1, wherein the first MCIO connector is connected to a Fabric port of the first PCIe switch and is used to connect to a second MCIO connector of another computing device.

10. The memory expansion unit includes a memory pool, the memory pool including at least one second processing unit having an independent memory and at least one second PCIe switch; and 3. The computing device of claim 2, wherein a downlink port of the second PCIe exchange is connected to a first endpoint port of the second processing unit, and a second endpoint port of the second processing unit is connected to a fifth downlink port of the first PCIe exchange.

11. 11. The computing device of claim 10, wherein the memory pool further includes a third MCIO connector, the third MCIO connector being connected to a Fabric port of the second PCIe switch and being used to connect to a fourth MCIO connector of another computing device.

12. 11. The computing device of claim 10, wherein the second processing unit comprises any one or combination of a field programmable gate array (FPGA), a complex program logic device (CPLD), a programmable logic device (PLD), an application specific integrated circuit (ASIC), a general purpose array logic (GAL), a system on a chip (SOC), a software defined architecture (SDI) device, and an artificial intelligence (AI) device.

13. 13. The computing device of claim 12, wherein the FPGA is used to partition internal resources of the FPGA into different regions using a dynamic partitioning technique or to realize dynamic allocation of memory and data transfer using a direct memory access (DMA) controller.

14. 14. The computing device of claim 1, wherein the accelerator is a graphics processing unit (GPU).

15. A server comprising a computing device according to any one of claims 1 to 14.

16. A data processing method based on a computing device according to any one of claims 1 to 14, comprising: the accelerator sending a first data request message to the host memory according to a CXL protocol; and transmitting first task data to the accelerator via the first PCIe switch in response to the first data request message from the host memory.

17. the accelerator executing a computational task based on the first task data to generate first result data; 17. The data processing method of claim 16, further comprising the step of: the accelerator receiving a second data request message sent from a CPU and sending the first result data to a host memory.

18. the accelerator sending a third data request message to the memory expansion unit according to the CXL protocol; 17. The data processing method of claim 16, further comprising: the memory expansion unit transmitting second task data to the accelerator via the first PCIe exchange in response to the third data request message.

19. 20. The data processing method of claim 18, further comprising the step of the accelerator storing third task data in the memory expansion unit via the first PCIe exchange.

20. 20. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions, the computer-readable instructions implementing the data processing method of any one of claims 16 to 19 when executed by a processor.

Images