Resource access method and computing device

Abstract

The embodiment of the invention relates to a resource access method and computing equipment. The method comprises the steps that an equipment discovery request is sent to second computing equipment through NTB; receiving a response of the second computing device to the first computing device, wherein the response comprises device information of computing resources on the second computing device; creating a virtual PCIe device corresponding to the computing resource on a PCIe bus of the first computing device based on the device information; and based on the virtual PCIe device and the NTB, accessing a computing resource on the second computing device to execute a corresponding computing task. The first computing device can transparently access and use the physical computing resources on the second computing device based on the virtual device and the NTB link so as to execute corresponding computing tasks, so that the first computing device can seamlessly bring far-end resources into the local for unified use without sensing a complex distributed architecture, and the implementation is convenient. Low-delay and high-bandwidth resource expansion can be realized, and meanwhile, the system complexity and the overall hardware cost are remarkably reduced.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to resource access methods and computing devices. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the dramatic increase in the scale of large language model parameters, extremely high demands are placed on computing devices, especially GPUs (graphics processing units).

[0003] When the graphics card resources of a single server cannot handle the full set of model parameters, related technologies typically employ multiple servers interconnected via a high-speed network to perform collaborative inference in a distributed system manner. However, this architecture relies on expensive high-bandwidth network switching equipment, resulting in high overall hardware costs. Summary of the Invention

[0004] This application provides a resource access method and a computing device that can reduce overall hardware costs.

[0005] According to a first aspect of the embodiments of this application, a resource access method is provided. The method is applied to a first computing device, the first computing device being communicatively connected to a second computing device via an NTB based on the PCIe protocol. The method includes: Send a device discovery request to the second computing device via NTB; Receive a response from the second computing device to the first computing device, the response including device information of computing resources on the second computing device; Based on the device information, a virtual PCIe device corresponding to the computing resources is created on the PCIe bus of the first computing device; wherein, the device information may include at least one of: vendor identifier, device identifier and base address register information.

[0006] Based on the virtual PCIe device and NTB, access computing resources on the second computing device to execute corresponding computing tasks.

[0007] In this scheme, the first computing device can transparently access and use the physical computing resources on the second computing device based on the virtual PCIe device and NTB link to perform corresponding computing tasks. This allows the first computing device to seamlessly integrate remote resources into local unified use without being aware of the complex distributed architecture, achieving low-latency, high-bandwidth resource expansion while significantly reducing system complexity and overall hardware costs.

[0008] In one possible implementation, based on device information, a virtual PCIe device corresponding to computing resources is created on the PCIe bus of the first computing device, including: Based on the NTB driver and device information running on the first computing device, the PCIe device is registered with the operating system kernel of the first computing device to obtain a virtual PCIe device.

[0009] In one possible implementation, registering the PCIe device with the operating system kernel of the first computing device includes: Based on the device information, create the structure of the virtual PCIe device; The device registration interface provided by the operating system kernel of the first computing device is called to register the initialized structure as a PCIe device and add the PCIe device to the PCIe device tree of the first computing device.

[0010] After creating and initializing the structure, this scheme uses the standard device registration interface provided by the operating system kernel of the first computing device to formally register the initialized structure as a PCIe device and add the newly created PCIe device to the PCIe device tree of the first computing device. This brings the virtual PCIe device under the unified device management scope of the operating system, allowing the kernel to allocate and manage resources, thus completing device integration at the system level and enabling the virtual device to be correctly recognized and driven by the system.

[0011] In one possible implementation, the method further includes: In response to a resource allocation request, allocate the memory-mapped MMIO space of the first computing device to the virtual PCIe device; Based on the access request to the MMIO space, the NTB driver intercepts and redirects the request to the computing resources of the second computing device. Allocate interrupt resources for the first computing device to the virtual PCIe device. The interrupt resources are used to process interrupt signals sent by the second computing device.

[0012] This solution establishes a stable and efficient data and control path from the first computing device to the second computing device, ensuring that remote resources can be accessed and used completely and transparently as if they were local devices, while maintaining system stability and responsiveness.

[0013] In one possible implementation, the device information also includes BAR information of the second computing device; the size of the MMIO space matches the memory space size of the second computing device indicated by the BAR information.

[0014] This solution ensures the reliability and transparency of the virtualization mechanism from the root of resource allocation by clearly defining and achieving precise matching between the MMIO space and the BAR information of the remote device, thereby enabling seamless, efficient and correct access to remote computing resources.

[0015] In one possible implementation, the method further includes: The computing resources of the first computing device and the computing resources of the second computing device are combined to form a computing resource pool; Computational tasks are executed on the computing resource pool using tensor parallelism (TP).

[0016] By constructing a unified logical resource pool and adopting a TP parallel strategy, the system fully leverages the computing power advantages brought by hardware expansion while avoiding the inherent complexity and performance loss of distributed systems.

[0017] In one possible implementation, the method further includes: When the first computing device starts up, the NTB driver of the local host is loaded; The NTB driver is used to perform operations such as sending device discovery requests, receiving responses, and creating virtual PCIe devices.

[0018] This solution clarifies the automated execution mechanism of the NTB driver during the system startup phase. By integrating device discovery and virtualization processes into the system initialization flow, it achieves "plug-and-play" access to remote computing resources. This design greatly simplifies the deployment and usage complexity of the system, allowing users to obtain expanded computing power without manual intervention, while ensuring the reliability and stability of system resources.

[0019] In one possible implementation, computing resources on the second computing device are visible to the first computing device via NTB, while computing resources on the first computing device are not visible to the second computing device.

[0020] Thanks to its one-way access feature, the second computing device can use a more simplified hardware configuration (e.g., no separate network card or simplified storage system), further reducing the overall system hardware cost.

[0021] According to a second aspect of the embodiments of this application, a resource access device is provided. This device is applied to a first computing device, which is communicatively connected to a second computing device via an NTB based on the PCIe protocol. The device includes: The request sending module is used to send a device discovery request to the second computing device via the NTB. The response receiving module is used to receive the response from the second computing device to the first computing device, and the response includes device information of the computing resources on the second computing device. The virtual device creation module is used to create a virtual PCIe device corresponding to the computing resources on the PCIe bus of the first computing device based on device information. The resource access module is used to access computing resources on the second computing device based on the virtual PCIe device and NTB in order to execute corresponding computing tasks.

[0022] According to a third aspect of the embodiments of this application, a computing device is provided. The computing device includes a memory and a processor, the memory storing a computer program, and the processor executing the program to implement the method as described above.

[0023] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in the embodiments of this application.

[0024] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above in the embodiments of this application. Attached Figure Description

[0025] More details, features, and advantages of embodiments of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of the system architecture provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating a resource access method provided in an exemplary embodiment of this application; Figure 3 A schematic block diagram of the functional modules of a resource access device provided in an exemplary embodiment of this application; Figure 4 A structural block diagram of a computing device provided for an exemplary embodiment of this application. Detailed Implementation

[0026] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that embodiments of this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the embodiments of this application. It should be understood that the accompanying drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0027] It should be understood that the various steps described in the method implementation of this application may be performed in different orders and / or in parallel. Furthermore, the method implementation may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in the embodiments of this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0029] It should be noted that the terms "one" and "more" mentioned in the embodiments of this application are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more". The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0030] like Figure 1 As shown, Figure 1 The system architecture provided in this application embodiment may include a first computing device 10 and a second computing device 20.

[0031] The first computing device 10 is used as a local host and can be the control center of the system.

[0032] In this embodiment, the first computing device 10 can be, for example, a standard x86 server, configured with a high-performance CPU (central processing unit), 64GB of memory, and an NTB (non-transparent bridge) 13 host card. The first computing device 10 can run a standard Linux operating system and NTB device drivers, and the addresses of computing resources in the second computing device can be mapped to address space 11 through NTB 13.

[0033] The second computing device 20 is used as a peer host and can serve as an extended resource pool.

[0034] In this embodiment, the second computing device 20 can be a hardware-simplified computing node. For example, the main components of the second computing device 20 may include a CPU 21 and a GPU 22. The second computing device 20 may also have an NTB endpoint card inserted and be directly connected to the NTB card 13 in the first computing device 10 via a dedicated NTB cable.

[0035] In this embodiment, to achieve cost optimization, the second computing device 20 is not configured with a separate network interface card and hard drive, and its system performs network proxy and remote storage access through the first computing device 10.

[0036] After the first computing device 10 is powered on, the NTB driver is loaded, and a device discovery request is sent to the second computing device 20 through the NTB link.

[0037] The endpoint card of the second computing device 20 returns PCIe (peripheral component interconnect express, high-speed serial computer expansion bus standard) device information of the internal GPU 21.

[0038] The NTB driver of the first computing device 10 dynamically creates and registers a virtual PCIe device for each remote GPU within its local address space 11 based on PCIe device information. The operating system of the first computing device 10 recognizes these virtual devices as local GPUs.

[0039] When a user starts an inference task of a large language model (such as a 70B parameter model) on the first computing device 10, the model loading software identifies, for example, four available GPUs (two real local GPUs and two virtualized remote GPUs).

[0040] The software automatically employs a tensor parallel (TP) strategy to distribute the model parameters across these four GPUs.

[0041] When computation requires access to the GPU 22 on the second computing device 20, the CPU 12 of the first computing device 10 initiates an access to a specific segment in its own address space 11. The NTB 13 hardware automatically intercepts this access, converts the access request into an NTB transaction according to its internally maintained address translation table, and sends it to the second computing device 20 via the link. Finally, the corresponding GPU 22 performs the computation, and the computation result can be returned along the original path.

[0042] This embodiment eliminates the need for expensive high-speed switches and the network interface cards (NICs) and storage of the second computing device, significantly reducing overall hardware costs. Furthermore, the embodiment utilizes NTB direct connection based on the PCIe protocol, reducing communication latency. The entire system is treated as a unified single-machine system by the operating system, eliminating the need for complex distributed task scheduling and state synchronization frameworks, thus greatly reducing operational complexity. Additionally, the first computing device can fully utilize the GPU memory and other computing resources of the second computing device, enabling the running of larger models without model pruning, ensuring optimal inference performance.

[0043] It should be noted that the address translation table mentioned above is implemented in the NTB hardware, and its function is similar to that of the memory management unit, but this table manages system-level physical address translation.

[0044] In the embodiments, combined with Figure 1 As shown, the workflow of this system architecture can specifically include: (1) System power-on and hardware initialization: The first computing device (Host) and the second computing device (Peer) are powered on respectively, and their respective BIOS / UEFI firmware completes the basic initialization of hardware such as CPU and memory.

[0045] (2) PCIe bus enumeration: Before the operating system is loaded, the Host and Peer enumerate the PCIe bus respectively. The Host recognizes its local single-sided NTB device, while the Peer recognizes its GPU as a local PCIe device.

[0046] (3) NTB link establishment: The host's NTB card and the peer's NTB endpoint card establish a stable communication link through a physical cable. At this time, the peer's endpoint card enters passive working mode and waits for instructions from the host.

[0047] (4) Device discovery and information acquisition: The host's operating system loads its one-sided NTB driver. This driver scans and activates the NTB link, and then sends a device discovery request to the peer's endpoint card. The peer endpoint card responds to this request and returns the device information of the peer's GPU to the host driver.

[0048] (5) Virtual GPU Device Creation: Based on the obtained peer-end GPU information, the Host's NTB driver dynamically creates a virtual GPU device (GPU') on the Host's local PCIe bus. This process includes: Simulate the GPU's PCIe configuration space and copy its key identifiers and configuration parameters.

[0049] Allocate a BAR (base address register) window and interrupt resources for the virtual device in the host's local address space for memory-mapped I / O (MMIO).

[0050] (6) Peer ready: The Peer operating system starts up and its local GPU driver loads normally. At this time, the Peer's NTB endpoint card and GPU enter a silent service state, do not actively initiate access, and only respond to and forward access requests sent by the Host through the NTB link.

[0051] Through the above process, a complete communication link is established for the Host to access the Peer-side GPU device. Afterward, applications and system software on the Host can transparently use the Peer-side's physical GPU resources to perform computing tasks by operating the virtual GPU device, just as if using a local device.

[0052] Specifically, during the configuration phase, when the Host's NTB driver discovers the GPU device on the peer, it will query its PCIe BAR space to find out the physical address range (e.g., 0x4000-0x8000) occupied by the GPU in the Peer system.

[0053] During the table creation phase, the driver then allocates a free region in the Host's address space (e.g., 0x90000-0x94000) and establishes a mapping rule in the NTB translation table, such as: "All accesses sent to the Host address 0x90000 are redirected to the Peer address 0x4000".

[0054] During the virtual device creation phase, when creating the virtual GPU device, the Host sets its BAR space to the mapped address (0x90000) on the Host side.

[0055] During the access phase, the GPU driver on the host performs a write operation to address 0x90000 as if accessing a local device. This access request from the CPU / RC is routed to the NTB. The NTB looks up the address in its translation table, finds that 0x90000 needs translation, and replaces the target address in the access request with 0x4000 on the peer. This request is sent to the peer via the NTB link, and the peer's endpoint card ultimately submits it to its local GPU. Additionally, the GPU's response can be returned along the same path.

[0056] In this embodiment, the Host NTB driver dynamically creates a virtual GPU device (GPU) on the local PCIe bus, simulating its PCIe configuration space (copying Peer GPU information) and allocating Host local resources for MMIO (BAR space) and interrupts. Specifically, "allocating Host local resources for MMIO (BAR space)" could mean allocating an address like 0x90000 in the Host address space and configuring a translation table entry in the NTB. "Simulating its PCIe configuration space" could mean setting the virtual GPU's BAR value to 0x90000, thus deceiving the operating system and driver. In this embodiment, the Host's operating system and driver are completely unaware of the remote device's existence; they operate on local, legitimate physical addresses. All complex address translations are handled by the NTB hardware in the background.

[0057] Based on the above embodiments, the system architecture provided in this application consists of two hosts and a peer. The host is local and the peer is remote. The host and peer are connected through a one-sided NTB. Peer resources are visible to the host through the NTB, but the host is not visible to the peer resources. The entire system is regarded as a single-machine system, not a distributed system. During inference, only TP is needed, and no host parallelism is required.

[0058] The peer host provides a physical connection to the host NTB card, such as via cable or backplane. As a proxy on the peer's PCIe bus, it can forward requests from the host to the target device (such as a GPU) and return responses or interrupts to the host.

[0059] Based on the above embodiments, this application also provides a resource access method. This method is applied to a first computing device, which is communicatively connected to a second computing device via an NTB based on the PCIe protocol. Figure 2 As shown, the method may include the following steps: In step S210, a device discovery request is sent to the second computing device via NTB.

[0060] The first computing device sends a device discovery request to the second computing device via the NTB driver and the established NTB physical link. This step aims to proactively probe and obtain information about the computing resources available on the other end.

[0061] In step S220, a response from the second computing device to the first computing device is received, the response including device information of computing resources on the second computing device.

[0062] In this embodiment, the first computing device receives a response from the second computing device to the discovery request. This response contains key device information about the computing resources (e.g., GPUs) mounted on the second computing device. This information typically includes, but is not limited to, the PCIe device's vendor ID, device ID, class code, and the number and size of its base address registers.

[0063] In step S230, based on the device information, a virtual PCIe device corresponding to the computing resources is created on the PCIe bus of the first computing device.

[0064] In this embodiment, based on the device information received in step S220, the first computing device dynamically creates a virtual PCIe device on its local PCIe bus corresponding to the physical computing resources on the second computing device. This creation process is led by the NTB driver of the first computing device and is achieved by registering a new PCIe device with the operating system kernel. The virtual device simulates the PCIe configuration space of the remote physical device and allocates it a local MMIO address space and interrupt resources, thereby enabling the operating system of the first computing device to recognize it as a real local hardware device.

[0065] In step S240, based on the virtual PCIe device and NTB, the computing resources on the second computing device are accessed to execute the corresponding computing task.

[0066] After the virtual PCIe device is successfully created, applications or system software on the first computing device can transparently access and use the physical computing resources on the second computing device based on the virtual device and the NTB link to perform corresponding computing tasks (e.g., large model inference). All access requests are intercepted by the NTB driver via the virtual device and converted into cross-device access transactions by the NTB hardware. Finally, the target computing resources on the second computing device are executed, and the results are returned through the original path.

[0067] The first computing device in this embodiment of the application does not need to be aware of the complex distributed architecture to seamlessly incorporate remote resources into local unified use, thereby achieving low-latency, high-bandwidth resource expansion, while significantly reducing system complexity and overall hardware costs.

[0068] Based on the above embodiments, in another embodiment provided in this application, when creating a virtual PCIe device corresponding to computing resources on the PCIe bus of the first computing device based on device information, the above step S230 may further include: Step S231: Based on the NTB driver and device information running on the first computing device, register the PCIe device with the operating system kernel of the first computing device to obtain a virtual PCIe device.

[0069] In this embodiment, the first computing device initiates a PCIe device registration request to the operating system kernel of the first computing device based on its locally running NTB driver and device information obtained from the second computing device, thereby obtaining the corresponding virtual PCIe device.

[0070] Specifically, the registration process includes: the NTB driver preparing the structure information of the virtual PCIe device, where key parameters (such as vendor ID, device ID, device class code, etc.) are set based on the actual device information of the physical computing resources on the second computing device. Then, the driver calls the standard PCIe device registration interface provided by the operating system kernel (for example, in the Linux kernel, this can be achieved through pci_register_driver or a related device addition interface) to formally register the prepared virtual device information into the PCIe device tree of the first computing device.

[0071] Through this registration process, the operating system kernel incorporates the virtual PCIe device into its unified device management framework, allocates system resources to it, and loads the corresponding standard device drivers, thereby completing the creation and activation of the virtual device. This implementation ensures that the virtual device can be correctly identified and managed by the system to enable subsequent resource access.

[0072] Based on the above embodiments, in another embodiment provided in this application, during the process of registering the PCIe device with the operating system kernel of the first computing device, step S231 may further include: Step S21: Based on the device information, create the structure of the virtual PCIe device.

[0073] In this embodiment, the NTB driver allocates and represents the data structure required by the virtual device in memory. The device identification parameters, class codes, and resource configuration information contained in this structure are strictly set according to the actual device information of the physical computing resources on the second computing device. This step essentially establishes a "digital image" of the virtual device at the software level, ensuring that its hardware characteristics are consistent with the remote physical device.

[0074] Step S22: Call the device registration interface provided by the operating system kernel of the first computing device to register the initialized structure as a PCIe device and add the PCIe device to the PCIe device tree of the first computing device.

[0075] After the structure is created and initialized, the first computing device calls the standard device registration interface provided by its operating system kernel to officially register the initialized structure as a PCIe device and add the newly created PCIe device to the first computing device's PCIe device tree. This brings the virtual PCIe device under the unified device management scope of the operating system, with the kernel handling resource allocation and management. This completes device integration at the system level, enabling the virtual device to be correctly recognized and driven by the system.

[0076] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S250, in response to the resource allocation request, the memory-mapped MMIO space of the first computing device is allocated to the virtual PCIe device.

[0077] In response to a resource allocation request from the operating system kernel, a local memory-mapped MMIO space is allocated to the virtual PCIe device. This MMIO space serves as a "window" or "proxy region" for the first computing device to access computing resources on the second computing device. Specifically, this allocation process is managed by the NTB driver, and the size of the allocated MMIO space is typically matched to the actual BAR requirements of the physical computing resources on the second computing device.

[0078] In step S260, based on the access request to the MMIO space, the NTB driver intercepts and redirects the request to the computing resources of the second computing device.

[0079] Based on an access request initiated by the CPU or device on the first computing device to the allocated MMIO space, the NTB driver intercepts the access operation and redirects it to the corresponding physical computing resource on the second computing device. Specifically, the NTB driver works in conjunction with the NTB hardware, using its internally maintained address translation table to transparently convert read and write operations targeting the local MMIO address into transactions sent via the NTB link to a specific physical address on the second computing device. This mechanism is the core of achieving transparent remote access to resources.

[0080] In step S270, interrupt resources of the first computing device are allocated to the virtual PCIe device. The interrupt resources are used to process the interrupt signals sent by the second computing device.

[0081] The first computing device allocates local interrupt resources (such as MSI or MSI-X vectors) to the aforementioned virtual PCIe device. These interrupt resources are used to receive, map, and process interrupt signals sent from the physical computing resources of the second computing device. When a computing task on the second computing device completes or needs to notify the first computing device, the resulting interrupt is transmitted to the first computing device via the NTB link. The NTB driver then associates this interrupt with the interrupt resources allocated to the virtual device and ultimately reports it to the operating system and driver of the first computing device, thus completing the entire transaction processing flow.

[0082] This embodiment details the allocation and management mechanism for two key system resources (MMIO space and interrupts) required by virtual PCIe devices. Through steps S250 to S270, a stable and efficient data and control path is established from the first computing device to the second computing device, ensuring that remote resources can be accessed and used completely and transparently as if they were local devices, while maintaining system stability and responsiveness.

[0083] It should be noted that the above device information also includes the BAR information of the second computing device, and the size of the MMIO space matches the memory space size of the second computing device indicated by the BAR information.

[0084] In this embodiment, the BAR information explicitly indicates the size and type of memory or I / O space required by the physical computing resource locally (in the second computing device). In step S250, the memory-mapped input / output (MMIO) space allocated by the first computing device for the virtual PCIe device is strictly matched to the memory space size of the computing resource on the second computing device indicated by the BAR information.

[0085] Specifically, after parsing the response from the second computing device, the NTB driver precisely allocates an MMIO window of the same size in the address space of the first computing device, based on the space requirements reported by the physical GPU's BAR (e.g., a request for 64MB of prefetch memory). This precise matching ensures that: The accuracy of address mapping provides the correct range for subsequent address translation by NTB hardware, avoiding the problem of wasted address space due to over-mapping or incomplete access to remote device resources due to under-mapping.

[0086] Driver compatibility: This ensures that standard device drivers (such as GPU drivers) loaded on the first computing device behave exactly the same when configuring and accessing virtual devices as when accessing a local physical device with equivalent resource requirements, guaranteeing driver stability and compatibility without any modifications.

[0087] Resource access integrity: This ensures that the first computing device can fully access all registers and memory areas exposed by the computing resources on the second computing device through the MMIO window, thereby achieving complete control over its functions.

[0088] By clearly defining and implementing precise matching between the MMIO space and the BAR information of the remote device, the implementation can ensure the reliability and transparency of the virtualization mechanism from the root of resource allocation, thereby enabling seamless, efficient and correct access to remote computing resources.

[0089] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S280, the computing resources of the first computing device and the computing resources of the second computing device are combined to form a computing resource pool.

[0090] The computing resources local to the first computing device (such as a local GPU) are logically integrated with the computing resources of the second computing device (such as a remote GPU) mapped through a virtual PCIe device, forming a unified pool of computing resources. Under this architecture, the operating system and upper-layer applications will recognize a complete set of computing units without being aware that these units are physically distributed across different devices.

[0091] In step S290, the computation task is executed on the computing resource pool in a tensor-parallel (TP) manner.

[0092] In this embodiment, computational tasks, particularly large AI model inference tasks, are executed on the aforementioned computing resource pool using a TP (Transaction Processing) approach. Since all computing resources are logically equivalent to a single-machine system, there is no need for cross-host task slicing and state synchronization. Tensor partitioning and parallel computation of model parameters can be performed directly across all GPUs (including local and remote GPUs), thereby significantly improving computational efficiency and reducing inference latency.

[0093] This embodiment clarifies the specific application methods in high-performance computing scenarios such as AI inference. By constructing a unified logical resource pool and adopting a TP parallel strategy, it fully leverages the computing power advantages brought by hardware expansion while avoiding the inherent complexity and performance loss of distributed systems.

[0094] Based on the above embodiments, in another embodiment provided in this application, the method may further include the following steps: In step S291, when the first computing device starts up, the NTB driver of the local host is loaded.

[0095] When the first computing device boots up, during the initialization phase of its operating system kernel, the local host's NTB driver is automatically loaded. This driver, as a core software component enabling remote resource access, manages all interactions with the NTB hardware and provides device virtualization capabilities.

[0096] In step S292, operations such as sending a device discovery request, receiving a response, and creating a virtual PCIe device are performed based on the NTB driver.

[0097] In this embodiment, the following key operation sequence can be automatically executed based on the loaded NTB driver: Send Device Discovery Request: Actively initiate device discovery to the second computing device via the NTB link.

[0098] Receive and parse the response: Obtain complete device information for computing resources on the second computing device.

[0099] Create a virtual PCIe device: Based on the acquired device information, dynamically create the corresponding virtual device on the PCIe bus of the first computing device.

[0100] This automated process ensures that once the first computing device has started up, the computing resources of the second computing device are ready and can be directly recognized and used by the operating system and upper-layer applications.

[0101] This embodiment clarifies the automated execution mechanism of the NTB driver during the system startup phase. By integrating the device discovery and virtualization processes into the system initialization flow, it achieves "plug-and-play" access to remote computing resources. This design greatly simplifies the deployment and usage complexity of the system, allowing users to obtain expanded computing power without manual intervention, while ensuring the reliability and stability of system resources.

[0102] It should be noted that, in this embodiment, the computing resources on the second computing device are visible to the first computing device through the NTB, while the computing resources on the first computing device are not visible to the second computing device.

[0103] Specific manifestations include: One-way resource discovery and mapping: After the first computing device initiates the device discovery process, the device information of computing resources (such as GPUs) on the second computing device can be obtained by the first computing device through the NTB link, and the first computing device can then create the corresponding virtual PCIe device. However, this process is one-way; the second computing device cannot discover or access any computing resources on the first computing device through the same NTB link.

[0104] Asymmetric System Roles: This architecture establishes an asymmetric role for the first computing device as the host and the second computing device as the peer. The first computing device has a complete view and control over the system resources, while the second computing device acts only as a passive resource provider, without needing to be aware of the system status or resource composition of the first computing device.

[0105] This embodiment avoids complex bidirectional resource management and address translation, greatly simplifying the design and driver implementation complexity of the NTB. By centrally scheduling and managing all resources on the first computing device, there is no need to deploy complex resource coordination software on the second computing device, reducing its hardware and software requirements. Furthermore, unidirectional visibility isolates resource access, ensuring that potential software issues on the second computing device do not directly affect the stable operation of the first computing device. Benefiting from the unidirectional access characteristic, the second computing device can employ a simpler hardware configuration (e.g., no separate network card or simplified storage system), further reducing the overall system hardware cost.

[0106] By dividing each functional module according to its corresponding function, this application provides a resource access device, which can be a server, a terminal, or a chip applied to a server. Figure 3 This is a schematic block diagram of the functional modules of a resource access device provided in an exemplary embodiment of this application. The device is applied to a first computing device, which is communicatively connected to a second computing device via an NTB based on the PCIe protocol, such as... Figure 3 As shown, the resource access device includes: Request sending module 31 is used to send a device discovery request to the second computing device via NTB; Response receiving module 32 is used to receive the response from the second computing device to the first computing device, the response including device information of computing resources on the second computing device; The virtual device creation module 33 is used to create a virtual PCIe device corresponding to the computing resources on the PCIe bus of the first computing device based on the device information. Resource access module 34 is used to access computing resources on the second computing device based on the virtual PCIe device and NTB to execute corresponding computing tasks.

[0107] In another embodiment provided in this application, based on device information, a virtual PCIe device corresponding to computing resources is created on the PCIe bus of the first computing device, including: Based on the NTB driver and device information running on the first computing device, the PCIe device is registered with the operating system kernel of the first computing device to obtain a virtual PCIe device.

[0108] In another embodiment provided in this application, registering a PCIe device with the operating system kernel of the first computing device includes: Based on the device information, create the structure of the virtual PCIe device; The device registration interface provided by the operating system kernel of the first computing device is called to register the initialized structure as a PCIe device and add the PCIe device to the PCIe device tree of the first computing device.

[0109] After creating and initializing the structure, this scheme uses the standard device registration interface provided by the operating system kernel of the first computing device to formally register the initialized structure as a PCIe device and add the newly created PCIe device to the PCIe device tree of the first computing device. This brings the virtual PCIe device under the unified device management scope of the operating system, allowing the kernel to allocate and manage resources, thus completing device integration at the system level and enabling the virtual device to be correctly recognized and driven by the system.

[0110] In another embodiment provided in this application, the device further includes an interrupt resource allocation module, used for: In response to a resource allocation request, allocate the memory-mapped MMIO space of the first computing device to the virtual PCIe device; Based on the access request to the MMIO space, the NTB driver intercepts and redirects the request to the computing resources of the second computing device. Allocate interrupt resources for the first computing device to the virtual PCIe device. The interrupt resources are used to process interrupt signals sent by the second computing device.

[0111] This solution establishes a stable and efficient data and control path from the first computing device to the second computing device, ensuring that remote resources can be accessed and used completely and transparently as if they were local devices, while maintaining system stability and responsiveness.

[0112] In another embodiment provided in this application, the device information further includes BAR information of the second computing device; the size of the MMIO space matches the memory space size of the second computing device indicated by the BAR information.

[0113] This solution ensures the reliability and transparency of the virtualization mechanism from the root of resource allocation by clearly defining and achieving precise matching between the MMIO space and the BAR information of the remote device, thereby enabling seamless, efficient and correct access to remote computing resources.

[0114] In another embodiment provided in this application, the device further includes a computing task execution module, used for: The computing resources of the first computing device and the computing resources of the second computing device are combined to form a computing resource pool; Computational tasks are executed on the computing resource pool using tensor parallelism (TP).

[0115] By constructing a unified logical resource pool and adopting a TP parallel strategy, the system fully leverages the computing power advantages brought by hardware expansion while avoiding the inherent complexity and performance loss of distributed systems.

[0116] In another embodiment provided in this application, the device further includes an operation processing module for: When the first computing device starts up, the NTB driver of the local host is loaded; The NTB driver is used to perform operations such as sending device discovery requests, receiving responses, and creating virtual PCIe devices.

[0117] This solution clarifies the automated execution mechanism of the NTB driver during the system startup phase. By integrating device discovery and virtualization processes into the system initialization flow, it achieves "plug-and-play" access to remote computing resources. This design greatly simplifies the deployment and usage complexity of the system, allowing users to obtain expanded computing power without manual intervention, while ensuring the reliability and stability of system resources.

[0118] In another embodiment provided in this application, the computing resources on the second computing device are visible to the first computing device through the NTB, while the computing resources on the first computing device are not visible to the second computing device.

[0119] Thanks to its one-way access feature, the second computing device can use a more simplified hardware configuration (e.g., no separate network card or simplified storage system), further reducing the overall system hardware cost.

[0120] This application also provides a computing device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is configured to execute instructions to implement the methods disclosed in the embodiments of this application.

[0121] The aforementioned processor can also be called a CPU, which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this application can be implemented by the integrated logic circuitry in the processor's hardware or by software instructions. The aforementioned processor can be a general-purpose processor, a digital signal processor (DSP), an ASIC (application-specific integrated circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in memory, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0122] Furthermore, various operations / processes according to embodiments of this application, when implemented via software and / or firmware, can be transmitted from a storage medium or network to a computing device with a dedicated hardware architecture, such as... Figure 4 The computing device 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computing device is able to perform various functions, including functions such as those mentioned above. Specifically, the computing device can be a server. Figure 4 A structural block diagram of a computing device provided for an exemplary embodiment of this application.

[0123] The computing device 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, computing devices, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0124] like Figure 4As shown, the computing device 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computing device 1900. The computing device 1900 also includes a GPU 1910. The computing unit 1901, ROM 1902, GPU 1910, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904. The number of GPUs 1910 may include multiple GPUs.

[0125] Multiple components in computing device 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information to computing device 1900. Input unit 1906 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the computing device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disk and optical disk. Communication unit 1909 allows computing device 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0126] The computing unit 1901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in the embodiments of this application can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on a computing device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in the embodiments of this application by any other suitable means (e.g., by means of firmware).

[0127] This application also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by the processor of a computing device, the computing device is able to perform the methods disclosed in the embodiments of this application.

[0128] The computer-readable storage medium in this application embodiment may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0129] The aforementioned computer-readable medium may be included in the aforementioned computing device; or it may exist independently and not assembled into the computing device.

[0130] This application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this application.

[0131] In embodiments of this application, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The modules, components, or units described in the embodiments of this application can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0134] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0135] The above description is merely an embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0136] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A resource access method, characterized in that, The method is applied to a first computing device, which is communicatively connected to a second computing device via an NTB based on the PCIe protocol. The method includes: A device discovery request is sent to the second computing device via the NTB; Receive a response from the second computing device to the first computing device, the response including device information of computing resources on the second computing device; Based on the device information, a virtual PCIe device corresponding to the computing resource is created on the PCIe bus of the first computing device; Based on the virtual PCIe device and the NTB, access the computing resources on the second computing device to execute the corresponding computing tasks.

2. The method according to claim 1, characterized in that, The step of creating a virtual PCIe device corresponding to the computing resources on the PCIe bus of the first computing device based on the device information includes: Based on the NTB driver running on the first computing device and the device information, the PCIe device is registered with the operating system kernel of the first computing device to obtain a virtual PCIe device.

3. The method according to claim 2, characterized in that, Registering the PCIe device with the operating system kernel of the first computing device includes: Based on the device information, create a structure for a virtual PCIe device; The device registration interface provided by the operating system kernel of the first computing device is invoked to register the initialized structure as a PCIe device and add the PCIe device to the PCIe device tree of the first computing device.

4. The method according to claim 2, characterized in that, The method further includes: In response to a resource allocation request, the memory-mapped MMIO space of the first computing device is allocated to the virtual PCIe device; Based on the access request to the MMIO space, the NTB driver intercepts and redirects the request to the computing resources of the second computing device. The first computing device is allocated interrupt resources to the virtual PCIe device, and the interrupt resources are used to process interrupt signals sent by the second computing device.

5. The method according to claim 4, characterized in that, The device information also includes BAR information of the second computing device; the size of the MMIO space matches the memory space size of the second computing device indicated by the BAR information.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The computing resources of the first computing device and the computing resources of the second computing device are combined to form a computing resource pool; Computational tasks are executed on the computing resource pool in a tensor-parallel (TP) manner.

7. The method according to claim 1, characterized in that, The method further includes: When the first computing device starts up, the NTB driver of the local host is loaded; The NTB driver executes the operations of sending a device discovery request, receiving a response, and creating a virtual PCIe device.

8. The method according to claim 1, characterized in that, The device information includes at least one of the following: supplier identifier, device identifier, and base address register information.

9. The method according to claim 1, characterized in that, The computing resources on the second computing device are visible to the first computing device through the NTB, while the computing resources on the first computing device are not visible to the second computing device.

10. A computing device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-9.

Images