Multi-gpu processor communication method and related devices

Abstract

The application is suitable for the field of computer communication technology, and provides a multi-GPU processor communication method, real-time state data of direct connection links between each PCIe bus bridge chip is acquired; according to the real-time state data, the direct connection link with the lowest current load is selected as a transmission path for GPU processor communication across different bus bridge chips; when the real-time traffic or cache occupancy rate of any direct connection link exceeds a preset threshold, at least part of cross-bridge communication services transmitted via the direct connection link is switched to other direct connection links; when the error code rate exceeds a preset error code rate threshold, the direct connection link is temporarily removed from the effective path; the following optimization operations are sequentially performed until the error code rate recovers to within the threshold; after successful retraining, the direct connection link is re-added to the effective path; according to the selected or switched transmission path, the routing register of the cross-chip direct connection port of the corresponding bus bridge chip is configured, so that each bus bridge chip performs data packet forwarding according to the configuration.

Description

Technical Field

[0001] This application belongs to the field of computer communication technology, and in particular relates to a multi-GPU processor communication method and related equipment. Background Technology

[0002] With the rapid development of applications such as large-scale AI model training, high-performance computing, and massively parallel processing, multi-GPU parallel computing systems have become a core component of computing infrastructure. Currently, existing multi-GPU processor communication technologies employ centralized switching via a single PCIe bus bridge or a cascaded architecture of multiple bus bridges. A single bus bridge carrying multiple GPU processors is prone to port contention, bandwidth compression, and increased signal crosstalk, with end-to-end communication latency typically ≥2μs, failing to meet the demands of large-scale model training and real-time parallel computing. Cascading multiple bus bridges introduces multi-level forwarding latency, signal attenuation, and additional points of failure, significantly reducing cross-bus bridge data interaction efficiency and compromising system stability. Furthermore, the lack of a unified scheduling and coordination mechanism among multiple bus bridges leads to uneven distribution of GPU processor traffic, long-term overload of local bus bridges, and low overall computing power utilization. Centralized power supply is also prone to voltage fluctuations and uneven current distribution. Therefore, there is an urgent need for a multi-GPU processor communication method that can sense the load status of each directly connected link in real time, dynamically select the optimal transmission path based on the real-time load of the link, and automatically perform service switching when the link is overloaded, in order to solve the problems of low coordination efficiency between multiple bus bridges, uneven distribution of service traffic, and long-term overload of local bus bridges in the existing technology. Summary of the Invention

[0003] This application provides a multi-GPU processor communication method to address the problems of low coordination efficiency among multiple bus bridges, uneven distribution of service traffic, and long-term overload of local bus bridges in existing technologies. By acquiring real-time status data of the direct links between PCIe bus bridge chips, the method selects the direct link with the lowest current load as the transmission path for cross-bridge communication based on the real-time status data. When the real-time traffic or cache occupancy rate of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link are switched to other direct links. When the bit error rate exceeds a preset bit error rate threshold, the direct link is temporarily removed from the effective path. The following optimization operations are performed sequentially until the bit error rate recovers to within the threshold. After successful retraining, the direct link is re-added to the effective path. Based on the selected or switched transmission path, the routing register of the cross-chip direct connection port of the corresponding bus bridge chip is configured so that each bus bridge chip performs data packet forwarding according to the configuration. This solves the problems of low coordination efficiency among multiple bus bridges, uneven distribution of service traffic, and long-term overload of local bus bridges in existing technologies.

[0004] In a first aspect, embodiments of the present invention provide a multi-GPU processor communication method. The method is used in a multi-GPU processor device based on a multi-bus-bridge architecture. The device includes multiple groups of GPU processors, which are interconnected via PCIe bus bridge chips, and different PCIe bus bridge chips are physically connected via on-board point-to-point direct links. The method includes the following steps:

[0005] The real-time status data of the direct links between each PCIe bus bridge chip is obtained, including the real-time traffic, cache utilization, and bit error rate of each direct link.

[0006] Based on the real-time status data, the direct link with the lowest current load is selected as the transmission path for GPU processor communication across different bus bridge chips;

[0007] When the real-time traffic or buffer occupancy rate of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link will be switched to other direct links. Specifically, for the cross-bridge communication service to be switched, first configure the forwarding route of the cross-chip direct connection ports of the bus bridge chips connected at both ends of the service via the target direct link, and then clear the forwarding route via the original direct link. When the bit error rate exceeds the preset bit error rate threshold, the direct link will be temporarily removed from the effective path. The following optimization operations will be performed sequentially until the bit error rate recovers to within the threshold: adjust the adaptive equalizer parameters corresponding to the cross-chip direct connection ports at both ends of the direct link; adjust the on-chip termination matching circuit impedance of the cross-chip direct connection ports at both ends; trigger the link training state machine of the bus bridge chips at both ends to perform joint retraining; after successful retraining, the direct link will be re-added to the effective path.

[0008] Configure the routing register of the cross-chip direct connection port of the corresponding bus bridge chip according to the selected or switched transmission path, so that each bus bridge chip can perform packet forwarding according to the configuration.

[0009] Optionally, obtaining real-time status data of the direct links between each PCIe bus bridge chip includes:

[0010] The real-time traffic is obtained by reading the transmit traffic count register and receive traffic count register of the cross-chip direct connection port of each bus bridge chip;

[0011] The buffer occupancy rate is obtained by reading the transmit buffer level register and receive buffer level register of the cross-chip direct connection port;

[0012] The bit error rate is obtained by reading the bit error rate counter corresponding to the cross-chip direct connection port.

[0013] Optionally, the preset threshold includes 80% of the physical bandwidth of the direct link and 70% of the total cache capacity; the condition for switching to other direct links is that the remaining bandwidth of the target direct link is not less than 120% of the bandwidth of the service to be switched.

[0014] Optionally, before acquiring the real-time status data of the direct links between each PCIe bus bridge chip, the method further includes:

[0015] Enumerate each bus bridge chip and its connected GPU processor, read the link status register of the cross-chip direct connection port of each bus bridge chip, and obtain the negotiation rate and negotiation bit width of each direct connection link.

[0016] Direct links that are in a state of reduced speed or reduced bandwidth are marked as degraded links, and their priority is reduced during path selection.

[0017] Optionally, the method further includes:

[0018] The current load status of each direct link is pushed to the host driver, so that the host communication library can allocate the cross-bridge communication task to the bus bridge chipset corresponding to the direct link with the lowest load.

[0019] Secondly, embodiments of the present invention also provide a multi-GPU processor device based on a multi-bus bridge architecture, comprising: multiple groups of GPU processors, wherein each group of GPU processors is interconnected via PCIe bus bridge chips, and different PCIe bus bridge chips are physically connected via on-board point-to-point direct links to implement the method described in the embodiments of the present invention.

[0020] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the multi-GPU processor communication method provided in embodiments of the present invention.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the multi-GPU processor communication method provided in the embodiments of the present invention.

[0022] In this embodiment of the invention, real-time status data of the direct links between PCIe bus bridge chips is obtained. This real-time status data includes the real-time traffic, cache utilization, and bit error rate of each direct link. Based on the real-time status data, the direct link with the lowest current load is selected as the transmission path for GPU processor communication across different bus bridge chips. When the real-time traffic or cache utilization of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link are switched to other direct links. When the bit error rate exceeds a preset bit error rate threshold, the direct link is temporarily removed from the effective path. The following optimization operations are performed sequentially until the bit error rate recovers to within the threshold. After successful retraining, the direct link is re-added to the effective path. Based on the selected or switched transmission path, the routing register of the cross-chip direct connection port of the corresponding bus bridge chip is configured so that each bus bridge chip performs packet forwarding according to the configuration. This invention solves the problems of low coordination efficiency among multiple bus bridges, uneven distribution of service traffic, and long-term overload of local bus bridges in the prior art.

[0023] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a multi-GPU processor communication method provided in one embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] like Figure 1 As shown, Figure 1 This is a flowchart of a multi-GPU processor communication method provided in an embodiment of the present invention. The multi-GPU processor communication method includes the following steps:

[0034] 101. Obtain real-time status data of the direct links between each PCIe bus bridge chip.

[0035] In this embodiment of the invention, the aforementioned multi-GPU processor communication method is used in a multi-GPU processor device based on a multi-bus-bridge architecture. The device includes multiple groups of GPU processors, which are interconnected via PCIe bus bridge chips. Different PCIe bus bridge chips are physically connected via on-board point-to-point direct links. These on-board point-to-point direct links can be physical links that directly connect the cross-chip direct connection ports of two PCIe bus bridge chips at the physical layer via high-speed differential traces on the motherboard PCB, without passing through any cascaded switching chips or forwarding devices, thus achieving direct physical communication between the two bus bridge chips.

[0036] The aforementioned PCIe bus bridge chip can be integrated on the motherboard, used to expand PCIe ports and enable data exchange between multiple PCIe devices. Each bus bridge chip provides multiple PCIe ports for mounting a GPU processor and has at least one cross-chip direct connection port for point-to-point direct connection with other bus bridge chips. The aforementioned PCIe bus bridge chip can be a Broadcom BCM957508 or similar PCIe 5.0 bus bridge chip, each providing 8 PCIe 5.0 x16 ports; multiple bus bridges are directly connected via PCIe 5.0 x16 at a speed of 32GT / s, achieving non-blocking cross-chip forwarding.

[0037] The aforementioned direct link can be a point-to-point high-speed serial connection established directly between different PCIe bus bridge chips through on-board traces. The direct link is used to carry data communication between GPU processors across different bus bridge chips.

[0038] The aforementioned real-time status data can be a set of real-time parameters that are read from the registers of each bus bridge chip at the current operating moment and reflect the current operating status of each direct link.

[0039] The aforementioned real-time status data includes the real-time traffic, buffer occupancy rate, and bit error rate for each directly connected link. The real-time traffic of the directly connected link represents its data transmission rate at the current moment, reflecting the current load intensity of the link. The buffer occupancy rate reflects the fill level of the transmit and receive buffers within the bus bridge chip corresponding to the directly connected link, indicating the processing congestion status upstream or downstream. The bit error rate reflects the probability of bit errors occurring during data transmission in the directly connected link, indicating the signal transmission quality of the link.

[0040] It should be noted that by acquiring real-time status data of the direct links between each PCIe bus bridge chip, real-time perception and quantitative monitoring of the operating status of the direct links between multiple bus bridges are achieved, enabling precise understanding of the current load level and transmission quality of each direct link.

[0041] 102. Based on real-time status data, select the direct link with the lowest current load as the transmission path for GPU processor communication across different bus bridge chips.

[0042] In this embodiment of the invention, the direct link with the lowest current load can be the one with the lightest load among all available direct links, determined comprehensively based on status data such as real-time traffic and cache utilization. For example, the link with the lowest real-time traffic and the lowest cache utilization can be selected.

[0043] It should be noted that by selecting the direct link with the lowest current load as the transmission path for GPU processor communication across different bus bridge chips based on real-time status data, dynamic load balancing of communication services between multiple bus bridges is achieved. This enables cross-bridge communication services to be reasonably allocated to each direct link, effectively avoiding the problem of local direct links becoming communication bottlenecks due to carrying too much cross-bridge communication.

[0044] 103. When the real-time traffic or buffer occupancy rate of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link will be switched to other direct links. Specifically, for the cross-bridge communication service to be switched, first configure the forwarding route of the cross-chip direct connection ports of the bus bridge chips at both ends of the service via the target direct link, and then clear the forwarding route via the original direct link. When the bit error rate exceeds the preset bit error rate threshold, the direct link will be temporarily removed from the effective path. The following optimization operations will be performed sequentially until the bit error rate recovers to within the threshold: adjust the adaptive equalizer parameters corresponding to the cross-chip direct connection ports at both ends of the direct link; adjust the on-chip termination matching circuit impedance of the cross-chip direct connection ports at both ends; trigger the link training state machine of the bus bridge chips at both ends to perform joint retraining; after successful retraining, the direct link will be re-added to the effective path.

[0045] In this embodiment of the invention, the preset threshold can be a pre-set safety threshold. The preset threshold is used to determine whether the direct link has entered an overload state. The preset threshold can be 70%, 80%, etc.

[0046] The aforementioned switch to other direct links may involve rerouting some or all of the cross-bridge communication services carried by a direct link that is identified as overloaded to other direct links with lower current load, in order to achieve load dispersal and rebalancing.

[0047] It should be noted that by switching at least a portion of the cross-bridge communication services transmitted via any direct link to other direct links when the real-time traffic or cache utilization of any direct link exceeds a preset threshold, the system has the capability to proactively intervene in and dynamically adjust overloaded links. It can migrate some services to other lightly loaded links in advance before a link becomes a bottleneck, thereby achieving proactive load balancing among multiple direct links and effectively improving communication throughput and overall resource utilization.

[0048] In this embodiment of the invention, the cross-bridge communication service to be switched may be a cross-bus bridge chip data communication service that is currently being transmitted via an overloaded direct link and needs to be switched to another direct link.

[0049] The aforementioned cross-chip direct connection port can be a PCIe port on a bus bridge chip used for point-to-point direct communication with another bus bridge chip.

[0050] The aforementioned target direct link can be a direct link used to replace the original overloaded link to carry the cross-bridge communication service to be switched.

[0051] The aforementioned forwarding routes can be configuration information used to determine the path from which data packets are forwarded from the port.

[0052] It should be noted that for the cross-bridge communication service to be switched, by first configuring the cross-chip direct connection ports of the bus bridge chips at both ends of the service with forwarding routes via the target direct connection link, the new route becomes effective before the old route is cleared. This ensures that data packets always have a usable forwarding path throughout the entire switching process, fundamentally avoiding data packet drop due to missing routes. By clearing the forwarding route via the original direct connection link only after confirming the successful configuration of the new route, the problem of inconsistent intermediate states caused by asynchronous configuration updates of the bus bridge chips at both ends is eliminated, ensuring that no routing loops are generated during the switching process.

[0053] The aforementioned preset bit error rate threshold can be a pre-set bit error rate threshold. The preset bit error rate threshold is used to determine whether the signal transmission quality of the direct link has deteriorated to the point that active repair is required.

[0054] The aforementioned valid paths can be the set of directly connected links currently available to carry cross-bridge communication services. Each directly connected link in this set is in a state where it can normally forward data packets.

[0055] The aforementioned temporary removal can be a process whereby a directly connected link is temporarily removed from the set of valid paths when the bit error rate of any directly connected link exceeds a preset bit error rate threshold. It is understood that during the temporary removal period, the directly connected link does not participate in path selection, and new cross-bridge communication services will no longer be scheduled to this directly connected link. However, the directly connected link can still carry existing services already in transmission or maintain the basic connectivity of the link to support subsequent optimization processes.

[0056] The aforementioned adaptive equalizer parameters can be circuit parameters within the physical layer of the PCIe bus bridge chip used to compensate for high-frequency attenuation caused by skin effect and dielectric loss during high-speed signal transmission through PCB traces. By adjusting the equalizer parameters, the eye diagram opening of the receiver signal can be optimized for different link loss characteristics, thereby reducing the bit error rate. Equalizer parameters include the gain coefficient and pole frequency of continuous-time linear equalizers and decision feedback equalizers.

[0057] The impedance of the on-chip termination matching circuit mentioned above can be the impedance value of the termination resistor network used to match the characteristic impedance of the transmission line in the physical layer inside the PCIe bus bridge chip.

[0058] The aforementioned link training state machine can be a finite state machine in the physical layer inside the PCIe bus bridge chip, responsible for performing link initialization and link state management.

[0059] The aforementioned joint retraining can be a process that simultaneously triggers the link training state machine of the bus bridge chips at both ends of the direct link, so that both ends enter the retraining process at the same time.

[0060] Understandably, by temporarily removing directly connected links with bit error rates exceeding a preset threshold from the effective path, and isolating them from the service transmission path during link repair, the degradation of communication quality caused by high-error-rate links continuing to carry cross-bridge communication services is avoided, including data retransmission, increased latency, and decreased throughput. A three-stage progressive repair process—adaptive equalizer parameter adjustment, on-chip termination matching circuit impedance adjustment, and joint retraining of the link training state machine—attempts repair step-by-step, from low-cost to high-cost and from low-impact to high-impact. This avoids unnecessary disruption to other services currently being transmitted by immediately performing global retraining, while ensuring that link reconstruction is a fallback solution if mild repair methods fail, thus guaranteeing the completeness of the repair process. By re-adding the directly connected link to the effective path after successful retraining, a complete closed loop from isolation repair to verification recovery is completed, enabling the repaired link to automatically return to service carrying and preventing the waste of overall communication bandwidth resources caused by long-term idleness after the link is removed due to abnormal bit error rates.

[0061] 104. Configure the routing register of the cross-chip direct connection port of the corresponding bus bridge chip according to the selected or switched transmission path, so that each bus bridge chip can perform packet forwarding according to the configuration.

[0062] In this embodiment of the invention, the aforementioned routing register may be a control register inside each bus bridge chip used to configure the packet forwarding path. By configuring the routing register of the cross-chip direct connection port, it is possible to specify which outgoing port or direct connection link should be used to forward a data packet received via that port and whose destination address points to a specific GPU processor.

[0063] The above configuration can be achieved by writing register values ​​through the management interface of the bus bridge chip, ensuring that the hardware-level packet forwarding path is consistent with the transmission path decided by the software. This management interface can be a PCIe configuration space, a dedicated management bus, etc.

[0064] It should be noted that by configuring the routing register of the cross-chip direct connection port of the corresponding bus bridge chip according to the selected or switched transmission path, each bus bridge chip can perform data packet forwarding according to the configuration, and the path decision result is transformed into the actual forwarding behavior at the hardware level of the bus bridge chip. This enables the cross-bridge communication data flow to reach the target GPU processor through the selected path, realizing the full-link closed-loop control from state awareness, path decision to hardware configuration execution.

[0065] In this embodiment of the invention, real-time status data of the direct links between PCIe bus bridge chips is obtained. This real-time status data includes the real-time traffic, cache utilization, and bit error rate of each direct link. Based on the real-time status data, the direct link with the lowest current load is selected as the transmission path for GPU processor communication across different bus bridge chips. When the real-time traffic or cache utilization of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link are switched to other direct links. When the bit error rate exceeds a preset bit error rate threshold, the direct link is temporarily removed from the effective path. The following optimization operations are performed sequentially until the bit error rate recovers to within the threshold. After successful retraining, the direct link is re-added to the effective path. Based on the selected or switched transmission path, the routing register of the cross-chip direct connection port of the corresponding bus bridge chip is configured so that each bus bridge chip performs packet forwarding according to the configuration. This invention solves the problems of low coordination efficiency among multiple bus bridges, uneven distribution of service traffic, and long-term overload of local bus bridges in the prior art.

[0066] It is understood that in the specific implementation of this application, data such as status data, communication data, and load data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0067] Optionally, in the step of obtaining real-time status data of the direct links between each PCIe bus bridge chip, real-time traffic can be obtained by reading the transmit traffic count register and receive traffic count register of the cross-chip direct connection port of each bus bridge chip; buffer occupancy rate can be obtained by reading the transmit buffer level register and receive buffer level register of the cross-chip direct connection port; and bit error rate can be obtained by reading the bit error rate counter corresponding to the cross-chip direct connection port.

[0068] In this embodiment of the invention, the aforementioned cross-chip direct connection port can be a port on a bus bridge chip used for point-to-point direct communication with another bus bridge chip. The aforementioned cross-chip direct connection port can be a PCIe 5.0 standard hot-swappable connector, supporting hot-plugging and automatic hardware identification.

[0069] The aforementioned transmit flow count register can be a register located inside the bus bridge chip and associated with a cross-chip direct connection port. The transmit flow count register is used to accumulate and record the total number of data bytes transmitted from that port.

[0070] The aforementioned receive traffic count register can be a register located inside the bus bridge chip and associated with a cross-chip direct connection port. The receive traffic count register is used to accumulate and record the total number of data bytes received from that port.

[0071] The aforementioned real-time traffic can be calculated by reading the transmit traffic count register and receive traffic count register of the cross-chip direct connection port of each bus bridge chip, which is the total amount of data transmitted on the direct connection link per unit time.

[0072] The aforementioned transmit buffer level register can be a register located inside the PCIe bus bridge chip and associated with the transmit buffer of the cross-chip direct connection port. The transmit buffer level register is used to indicate the percentage or absolute value of the amount of data to be transmitted currently stored in the transmit buffer relative to the total buffer capacity.

[0073] The aforementioned receive buffer level register can be a register located inside the PCIe bus bridge chip and associated with the receive buffer of the cross-chip direct connection port. The receive buffer level register is used to indicate the percentage or absolute value of the amount of data to be processed currently stored in the receive buffer relative to the total buffer capacity.

[0074] The aforementioned buffer occupancy rate can be obtained by reading the transmit buffer level register and receive buffer level register of the cross-chip direct connection port to obtain the percentage of the current data stored in the transmit buffer and receive buffer relative to the total buffer capacity.

[0075] The aforementioned bit error rate counter can be a register located inside the bus bridge chip and associated with the physical layer of the cross-chip direct connection port. The bit error rate counter is used to accumulate and record the number of bit errors that occur on this link.

[0076] The aforementioned bit error rate can be obtained by reading the bit error rate counter corresponding to the cross-chip direct connection port, and counting the ratio of the number of bits that are erroneous during data transmission to the total number of transmitted bits, which is used to characterize the signal transmission quality of the link.

[0077] It should be noted that real-time traffic is obtained by reading the transmit and receive traffic count registers of the cross-chip direct connection ports of each bus bridge chip. Utilizing the cumulative traffic counter natively provided by the bus bridge chip hardware, accurate link transmission rate data is obtained with extremely low read overhead, avoiding the accuracy loss and bandwidth consumption caused by obtaining traffic information through software probing or estimation. The buffer occupancy rate is obtained by reading the transmit and receive buffer level registers of the cross-chip direct connection ports. The real-time update characteristics of the hardware registers are used to obtain an instantaneous snapshot of the buffer level, enabling precise perception of the real-time occupancy status of the transmit and receive buffers at both ends of the link, providing early warning information for predicting impending link overload. The bit error rate (BER) is obtained by reading the corresponding BER counter of the cross-chip direct connection port. Utilizing the automatic BER counting capability of the hardware physical layer during data verification, real-time quantitative monitoring of link transmission quality is achieved without additional overhead, allowing detection of abnormal trends even when the link signal quality deteriorates but the link is still operational.

[0078] Optionally, the preset thresholds include 80% of the physical bandwidth of the direct link and 70% of the total cache capacity; the condition for switching to other direct links is that the remaining bandwidth of the target direct link is not less than 120% of the bandwidth of the service to be switched.

[0079] In this embodiment of the invention, the aforementioned physical bandwidth may be the maximum data transmission rate that a direct link can support at the physical layer.

[0080] Understandably, when the real-time traffic on a directly connected link reaches 80% of its physical bandwidth, it can be determined that the link has entered a high-load warning zone. The preset threshold retains a 20% bandwidth margin to handle traffic surges while triggering intervention before the link becomes truly congested, providing a time window for service switching. By setting the traffic overload preset threshold to 80% of the directly connected link's physical bandwidth, triggering overload detection and service switching when the directly connected link still has 20% remaining bandwidth, it avoids routing disturbances caused by premature switching and provides sufficient safety buffer space for service migration during the switching process, preventing packet loss and latency spikes due to insufficient bandwidth during switching.

[0081] The total cache capacity mentioned above can be the total storage capacity of the transmit and receive buffers within the cross-chip direct connection port, used to temporarily store data packets to be sent or processed.

[0082] Furthermore, when the cache utilization rate reaches 70% of the total cache capacity, it can be determined that the buffer resources of the directly connected link are in a high storage state. If the load continues to increase, it may lead to cache overflow and data packet loss, requiring a service switchover to be triggered. By setting the cache overload preset threshold to 70% of the total cache capacity, the cache utilization rate is used as a proactive early warning indicator of link congestion. Overload determination and service switchover are triggered in advance before the buffer actually overflows, providing sufficient time for the switchover operation and effectively avoiding data packet loss caused by buffer overflow.

[0083] The bandwidth of the service to be switched mentioned above can be the bandwidth occupied by the bridge communication service that needs to be switched from the overloaded link to other direct links.

[0084] The remaining bandwidth of the aforementioned target direct link can be the additional bandwidth capacity that the target direct link can still carry, taking into account the current existing load.

[0085] Understandably, by setting the remaining bandwidth of the target direct link to be no less than 120% of the bandwidth of the service to be switched as the switching condition, an additional 20% bandwidth safety margin is reserved for the target link to ensure that the target link after the switch will not immediately enter an overload state due to the addition of new services.

[0086] Optionally, before obtaining the real-time status data of the direct links between each PCIe bus bridge chip, it is also possible to enumerate each bus bridge chip and its connected GPU processor, read the link status register of the cross-chip direct link of each bus bridge chip, obtain the negotiation rate and negotiation bit width of each direct link; mark the direct links in the state of reduced speed or reduced bit width as degraded links, and reduce their priority when selecting paths.

[0087] In this embodiment of the invention, the above enumeration of each bus bridge chip and its connected GPU processor can be performed by traversing and identifying all PCIe bus bridge chips on the motherboard, determining the device number and position of each bus bridge chip in the PCIe bus topology, and further identifying and recording the bus device function number and link status of each GPU processor connected to each bus bridge chip by reading the downlink port configuration space of each bus bridge chip.

[0088] The link status register for the aforementioned cross-chip direct connection port can be a register located inside the PCIe bus bridge chip and associated with the cross-chip direct connection port. The link status register is used to indicate the real-time negotiation status of the current link, which includes the currently negotiated PCIe transfer rate, the currently negotiated link width, etc.

[0089] The aforementioned negotiated rate can be the PCIe data transmission rate at which the bus bridge chips at both ends of the link ultimately negotiate and establish a link based on physical conditions such as signal quality, trace length, and connector status during the PCIe link training process.

[0090] The aforementioned negotiated bit width can be the PCIe link width that the bus bridge chips at both ends of the link ultimately negotiate and establish based on the physical connection status during the PCIe link training process.

[0091] The aforementioned speed reduction or bandwidth reduction state can be caused by the current negotiated rate of the direct link being lower than the highest nominal rate supported by the direct link, or the current negotiated bandwidth being lower than the maximum physical bandwidth supported by the direct link.

[0092] The aforementioned degraded links can be direct links that are in a state of reduced speed or bandwidth. It is understood that the actual carrying capacity of a degraded link is lower than its nominal physical bandwidth, and it should be given lower priority when selecting a path.

[0093] It should be noted that by enumerating each bus bridge chip and its connected GPU processor and reading the link status registers of the cross-chip direct connection ports, a comprehensive identification and status confirmation of the hardware configuration of each bus bridge chip, each GPU processor, and each direct connection link was achieved. This established a complete hardware topology view for subsequent link status monitoring and path selection, avoiding erroneous routing decisions due to incomplete topology understanding. By reading the link status registers of the cross-chip direct connection ports of each bus bridge chip, the negotiation rate and negotiation bit width of each direct connection link were obtained. Utilizing the link status awareness capability natively provided by PCIe hardware, the true physical negotiation status of each direct connection link was obtained with extremely low overhead. This accurately identified non-ideal links that were operating at reduced speed or bit width due to signal integrity and other reasons. By marking direct connections in a reduced speed or bit width state as degraded links and lowering their priority during path selection, performance losses caused by scheduling cross-bridge communication services to degraded links, resulting in actual bandwidth being far lower than expected, were avoided. This ensured the consistency between the expected bandwidth and the actual available bandwidth of cross-bridge communication.

[0094] Optionally, the method also includes pushing the current load status of each direct link to the host driver, so that the host-side aggregation communication library can allocate the cross-bridge communication task to the bus bridge chipset corresponding to the direct link with the lowest load.

[0095] In this embodiment of the invention, the current load status of the directly connected link can be the comprehensive load quantification information of the directly connected link at the current moment. The current load status can be calculated by the host driver from the relevant status registers read from the cross-chip direct connection ports of each bus bridge chip. The current load status includes the real-time traffic value and buffer utilization value of the current directly connected link.

[0096] The aforementioned host-side driver can be a PCIe device driver running in the host operating system kernel mode. The host-side driver is responsible for device management, resource allocation, interrupt handling, and I / O control of each PCIe bus bridge chip and GPU processor in a multi-GPU processor device.

[0097] The aforementioned collective communication library can be a parallel communication software library running in user space on the host side. This communication software library can be NVIDIA NCCL, MPI, Gloo, etc. The collective communication library can provide collective communication operations such as All-Reduce, All-Gather, and Broadcast for multi-GPU parallel computing tasks.

[0098] The aforementioned bus bridge chipset can be a pair of two bus bridge chips corresponding to a direct link. It can be understood that a direct link connects two bus bridge chips, and when the communication library allocates cross-bridge communication tasks, the task is assigned to this bus bridge chipset. The communication task will then occur between the GPU processors connected to the two bus bridge chips.

[0099] Furthermore, after receiving the current load status information of each direct link, the collective communication library can prioritize the allocation of new communication tasks to the chipset consisting of two bus bridge chips corresponding to the direct link with the lowest current load when allocating cross-bridge communication tasks. This allows newly initiated cross-bridge communication traffic to fall directly on the lightly loaded link, thereby achieving load balancing at the source during the task generation stage.

[0100] It should be noted that by pushing the current load status of each directly connected link to the host-side driver, the information isolation barrier between hardware-level link status awareness and software-level task orchestration is broken down. This enables the host-side aggregated communication library to perceive the real-time load of the underlying links for the first time. The host-side aggregated communication library then allocates cross-bridge communication tasks to the bus bridge chipset corresponding to the directly connected link with the lowest load. This shifts the load balancing decision-making time from passive switching after a link is overloaded to proactive avoidance before a communication task is initiated, reducing the probability of link overload at the source and decreasing the frequency of dynamic switching.

[0101] This invention provides a multi-GPU processor communication device, comprising: multiple groups of GPU processors, wherein each group of GPU processors is interconnected via a PCIe bus bridge chip, and different PCIe bus bridge chips are physically connected via on-board point-to-point direct links to realize the multi-GPU processor communication method provided in this invention.

[0102] In this embodiment of the invention, the multi-GPU processor device further includes a motherboard, a power supply module, a signal optimization module, a cross-chip direct connection port, and a control module. The motherboard may be a high-end server motherboard, equipped with an Intel Xeon Platinum 8480C processor, providing ≥256 PCIe 5.0 lanes to meet the bandwidth requirements of multiple bus bridges and expanded devices. The power supply module may be a 2400W redundant platinum power supply, with power supplied in zones according to bus bridges, and each GPU processor configured with an independent TI TPS54620 PMIC to achieve precise voltage / current regulation. The signal optimization module may use a 100MHz low-jitter crystal oscillator, with each channel configured with an adaptive equalizer and termination matching circuit, strictly controlling the differential line length and impedance to suppress crosstalk. The cross-chip direct connection port may be a port on the bus bridge chip used for point-to-point direct communication with another bus bridge chip, and may be a PCIe 5.0 standard hot-swappable connector, supporting hot-swapping and automatic hardware identification. The aforementioned control module can be an ARM Cortex-M7 MCU with an integrated IPMI remote management interface to achieve status monitoring, load scheduling, fault alarms, and remote operation and maintenance.

[0103] The aforementioned GPU processors are equipped with independent liquid cooling components. Liquid cooling is an active heat dissipation technology that uses liquid as a cooling medium to absorb and remove heat from electronic devices through circulation. The core of liquid cooling is to utilize the fact that liquid has a much higher specific heat capacity and thermal conductivity than air to achieve efficient, low-noise, and high-density thermal management.

[0104] The point-to-point direct connection link on the board can be a physical link that directly connects the cross-chip direct connection ports of two PCIe bus bridge chips at the physical layer through high-speed differential traces on the motherboard PCB, without passing through any cascaded switching chips or forwarding devices, thus realizing direct physical communication between the two bus bridge chips.

[0105] like Figure 2 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described multi-GPU processor communication methods.

[0106] Specifically, it includes a processor 201 and a memory 202, as well as a computer program stored in the memory 202 and capable of running on the processor 201, which executes a multi-GPU processor communication method, wherein:

[0107] The real-time status data of the direct links between each PCIe bus bridge chip is obtained, including the real-time traffic, cache utilization, and bit error rate of each direct link.

[0108] Based on the real-time status data, the direct link with the lowest current load is selected as the transmission path for GPU processor communication across different bus bridge chips;

[0109] When the real-time traffic or buffer occupancy rate of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link will be switched to other direct links. Specifically, for the cross-bridge communication service to be switched, first configure the forwarding route of the cross-chip direct connection ports of the bus bridge chips connected at both ends of the service via the target direct link, and then clear the forwarding route via the original direct link. When the bit error rate exceeds the preset bit error rate threshold, the direct link will be temporarily removed from the effective path. The following optimization operations will be performed sequentially until the bit error rate recovers to within the threshold: adjust the adaptive equalizer parameters corresponding to the cross-chip direct connection ports at both ends of the direct link; adjust the on-chip termination matching circuit impedance of the cross-chip direct connection ports at both ends; trigger the link training state machine of the bus bridge chips at both ends to perform joint retraining; after successful retraining, the direct link will be re-added to the effective path.

[0110] Configure the routing register of the cross-chip direct connection port of the corresponding bus bridge chip according to the selected or switched transmission path, so that each bus bridge chip can perform packet forwarding according to the configuration.

[0111] Optionally, the process of acquiring real-time status data of the direct links between each PCIe bus bridge chip, performed by the processor 201, includes:

[0112] The real-time traffic is obtained by reading the transmit traffic count register and receive traffic count register of the cross-chip direct connection port of each bus bridge chip;

[0113] The buffer occupancy rate is obtained by reading the transmit buffer level register and receive buffer level register of the cross-chip direct connection port;

[0114] The bit error rate is obtained by reading the bit error rate counter corresponding to the cross-chip direct connection port.

[0115] Optionally, the preset threshold executed by the processor 201 includes 80% of the physical bandwidth of the direct link and 70% of the total cache capacity; the condition for switching to other direct links is that the remaining bandwidth of the target direct link is not less than 120% of the bandwidth of the service to be switched.

[0116] Optionally, before acquiring the real-time status data of the direct links between each PCIe bus bridge chip, the method executed by the processor 201 further includes:

[0117] Enumerate each bus bridge chip and its connected GPU processor, read the link status register of the cross-chip direct connection port of each bus bridge chip, and obtain the negotiation rate and negotiation bit width of each direct connection link.

[0118] Direct links that are in a state of reduced speed or reduced bandwidth are marked as degraded links, and their priority is reduced during path selection.

[0119] Optionally, the method executed by processor 201 further includes:

[0120] The current load status of each direct link is pushed to the host driver, so that the host communication library can allocate the cross-bridge communication task to the bus bridge chipset corresponding to the direct link with the lowest load.

[0121] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the multi-GPU processor communication method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0122] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multi-GPU processor communication method, characterized in that, The method is used for a multi-GPU processor device based on a multi-bus bridge architecture. The device includes multiple groups of GPU processors, which are interconnected via PCIe bus bridge chips, and different PCIe bus bridge chips are physically connected via on-board point-to-point direct links. The method includes the following steps: The real-time status data of the direct links between each PCIe bus bridge chip is obtained, including the real-time traffic, cache utilization, and bit error rate of each direct link. Based on the real-time status data, the direct link with the lowest current load is selected as the transmission path for GPU processor communication across different bus bridge chips; When the real-time traffic or buffer occupancy rate of any direct link exceeds a preset threshold, at least a portion of the cross-bridge communication services transmitted via that direct link will be switched to other direct links. Specifically, for the cross-bridge communication service to be switched, first configure the forwarding route of the cross-chip direct connection ports of the bus bridge chips connected at both ends of the service via the target direct link, and then clear the forwarding route via the original direct link. When the bit error rate exceeds the preset bit error rate threshold, the direct link will be temporarily removed from the effective path. The following optimization operations will be performed sequentially until the bit error rate recovers to within the threshold: adjust the adaptive equalizer parameters corresponding to the cross-chip direct connection ports at both ends of the direct link; adjust the on-chip termination matching circuit impedance of the cross-chip direct connection ports at both ends; trigger the link training state machine of the bus bridge chips at both ends to perform joint retraining; after successful retraining, the direct link will be re-added to the effective path. Configure the routing register of the cross-chip direct connection port of the corresponding bus bridge chip according to the selected or switched transmission path, so that each bus bridge chip can perform packet forwarding according to the configuration.

2. The multi-GPU processor communication method as described in claim 1, characterized in that, The acquisition of real-time status data of the direct links between each PCIe bus bridge chip includes: The real-time traffic is obtained by reading the transmit traffic count register and receive traffic count register of the cross-chip direct connection port of each bus bridge chip; The buffer occupancy rate is obtained by reading the transmit buffer level register and receive buffer level register of the cross-chip direct connection port; The bit error rate is obtained by reading the bit error rate counter corresponding to the cross-chip direct connection port.

3. The multi-GPU processor communication method as described in claim 1, characterized in that, The preset threshold includes 80% of the physical bandwidth of the direct link and 70% of the total cache capacity; the condition for switching to other direct links is that the remaining bandwidth of the target direct link is not less than 120% of the bandwidth of the service to be switched.

4. The multi-GPU processor communication method as described in claim 1, characterized in that, Before acquiring the real-time status data of the direct links between each PCIe bus bridge chip, the method further includes: Enumerate each bus bridge chip and its connected GPU processor, read the link status register of the cross-chip direct connection port of each bus bridge chip, and obtain the negotiation rate and negotiation bit width of each direct connection link. Direct links that are in a state of reduced speed or reduced bandwidth are marked as degraded links, and their priority is reduced during path selection.

5. The multi-GPU processor communication method as described in claim 1, characterized in that, The method further includes: The current load status of each direct link is pushed to the host driver, so that the host communication library can allocate the cross-bridge communication task to the bus bridge chipset corresponding to the direct link with the lowest load.

6. A multi-GPU processor device based on a multi-bus bridge architecture, characterized in that, For use including: multiple groups The GPU processors are interconnected through PCIe bus bridge chips, and different PCIe bus bridge chips are physically connected through on-board point-to-point direct links to achieve the method described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the multi-GPU processor communication method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-GPU processor communication method as described in any one of claims 1 to 5.

Images