An inter-core and inter-chip interconnection device and method

Abstract

This application provides an inter-core and inter-chip interconnection device and method, applied to a system containing multiple computing cores or multiple chips. The interconnection device comprises: a DMA channel array unit containing multiple DMA channels, each DMA channel having a built-in microcode cache and microcode program counter for independently executing microcode programs pre-downloaded to the central processing unit to directly complete atomic operations and data forwarding along the data transport path; a data flow-aware dynamic bandwidth allocation unit coupled to the DMA channel array unit and an external bus interface for real-time monitoring of the flow status of each DMA channel and dynamically adjusting the binding relationship between the DMA channels and the external bus interface; and a synchronization management unit coupled to each computing core or chip for maintaining inter-core or inter-chip dependencies and controlling the release timing of each computing core or chip. This improves the overall efficiency of multi-chip collaborative operation.

Description

Technical Field

[0001] This application relates to the field of chip technology, and more specifically, to an interconnection device and interconnection method between cores and between chips. Background Technology

[0002] As computing or inference tasks continue to scale, it is often necessary to combine multiple computing chips or cores into a single system for collaborative operation via interconnect technology. In systems involving multiple cores or chips, data transfer and collaborative computation are required between different chips or cores. Currently, data transfer between cores or chips is typically performed by a DMA controller. Each DMA transfer requires CPU involvement in configuring transfer parameters, and the DMA then notifies the CPU via an interrupt after the transfer is complete, resulting in frequent CPU interrupt responses and significant software overhead. Furthermore, multiple data streams in the system share bus bandwidth, and existing fixed bandwidth allocation methods struggle to flexibly handle real-time traffic changes. Inter-core synchronization often employs a global barrier mechanism, leading to resource waste. Summary of the Invention

[0003] The purpose of this application is to provide an interconnection device and method between cores and chips to improve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide an inter-core and inter-chip interconnect device, applied to a system containing multiple computing cores or multiple chips. The interconnect device comprises: a DMA channel array unit containing multiple DMA channels, each DMA channel having a built-in microcode cache and microcode program counter for independently executing microcode programs pre-downloaded to the central processing unit to directly complete atomic operations and data forwarding on the data transport path; a data flow-aware dynamic bandwidth allocation unit coupled to the DMA channel array unit and an external bus interface for real-time monitoring of the flow status of each DMA channel and dynamically adjusting the binding relationship between the DMA channel and the external bus interface according to the priority of the data flow; and a synchronization management unit coupled to each computing core or chip for maintaining inter-core or inter-chip dependencies and controlling the release timing of each computing core or chip according to the dependencies; wherein the DMA channel array unit, the dynamic bandwidth allocation unit, and the synchronization management unit are coupled to each other.

[0005] In the above implementation process, by embedding microcode cache and microcode program counter in the DMA channel, the DMA channel can independently execute the microcode program pre-downloaded by the CPU. Subsequent data transfer and computation are all completed autonomously by the DMA channel, without the CPU needing to participate in configuration and interrupt response sequentially. This significantly reduces the CPU's involvement frequency, lowers system software overhead, and improves the overall efficiency of multi-chip collaborative work. By setting up a data flow-aware dynamic bandwidth allocation unit, the flow status and backpressure signal of each DMA channel are monitored in real time, and the binding relationship between the DMA channel and the external bus interface is dynamically adjusted according to the data flow priority, improving the overall utilization efficiency of bus bandwidth. By setting up a synchronization management unit, the dependencies between cores or chips are maintained, and the release timing of each computing core or chip is controlled according to the dependencies. Synchronization only occurs between cores with dependencies. Cores without dependencies can continue execution in advance after meeting their own conditions, without waiting for other unrelated cores, thereby reducing the idle waiting time of computing resources and improving the parallel processing efficiency of multi-core or multi-chip systems. Through the coordinated work of the DMA channel array unit, dynamic bandwidth allocation unit, and synchronization management unit, the hardware autonomously completes cross-chip data transfer, computation, and synchronization operations after a single CPU configuration, thereby improving the overall operating efficiency of the multi-chip system.

[0006] Optionally, in this embodiment, each DMA channel includes a first sub-channel and a second sub-channel, which are used to interact with the chips on both sides in the cascade direction, respectively. Each sub-channel includes: an atomic operation decoder and dispatcher for parsing microcode instructions, generating operation control signals and remote atomic operation lock management commands, and dispatching data streams; a remote atomic operation module for processing atomic operation requests from other chips and for determining the atomicity of access; a first read data channel and a second read data channel for reading data in parallel from on-chip or off-chip memory; a configurable operation array for performing operation modes on the two input data streams; and a write data channel for writing out the operation results or pass-through data.

[0007] In the above implementation, the DMA channel uses a first sub-channel and a second sub-channel to interact with the chips on both sides of the cascaded path, isolating the data flow on both sides and reducing mutual interference. Atomic operation decoding and dispatching, along with the parsing of microcode instructions and data stream allocation by the dispatcher, and the handling of atomic operation requests from other chips by the remote atomic operation module, allow cross-chip atomic operations to be completed autonomously by hardware without CPU intervention, reducing latency and software overhead. Data is read in parallel through the first and second read data channels, and a configurable arithmetic array dynamically selects the operation mode based on the microcode instructions to perform fusion calculations on the two data streams. This allows data transfer and computation to be completed within the same hardware pipeline, reducing the number of data transfers. The write data channel writes the computation results and supports cross-chip forwarding, enabling the processed data to be directly transferred to the next-level chip without CPU intervention, completing the data transfer in the chained protocol.

[0008] Optionally, in this embodiment, the microcode program includes an extensible microcode instruction set. Each microcode instruction includes at least one of the following fields: an opcode field, used to specify the instruction type, which includes at least one of transport instructions, arithmetic instructions, synchronization instructions, and routing instructions; an arithmetic mode field, used to specify the type of atomic operation; a next-hop identifier field, used to specify the chain forwarding target; a chain endpoint flag field, used to identify whether the current node is the chain endpoint; a broadcast address field, used to specify the unified base address where each chip stores the final result for backward broadcast writing; and a quality of service priority label field, used by the dynamic bandwidth allocation unit to identify the latency sensitivity of the data stream.

[0009] In the above implementation, by defining an extensible microcode instruction set, the DMA channel can perform various types of operations. The opcode field allows a single microcode instruction to indicate different operations such as transport, computation, synchronization, or routing, without requiring sequential CPU intervention; the operation mode field enables the same arithmetic array hardware to support multiple atomic operations; the next-hop identifier field and the chain endpoint flag field allow for flexible configuration of the forwarding path in the chained reduction; the broadcast address field simplifies the write configuration of backward broadcasts; and the quality of service priority label field enables the dynamic bandwidth allocation unit to identify the latency sensitivity of data streams, thereby ensuring the timely transmission of high-priority data streams. These fields work together to enable microcode programs to flexibly describe complex cross-chip collaborative tasks, reducing the frequency of CPU intervention.

[0010] Optionally, in this embodiment, the dynamic bandwidth allocation unit includes: a bandwidth monitoring subunit, used to poll the utilization rate of each external bus interface and the backpressure status of the DMA channel at a preset period; a quality of service tag identification subunit, used to extract the quality of service priority tag from the microcode instruction; a dynamic arbitration and binding adjustment subunit, used to redirect the DMA channel of the data stream with the required priority to an idle external bus interface according to the monitoring result of the bandwidth monitoring subunit and the identification result of the quality of service tag identification subunit, and restore the original binding relationship of the DMA channel after the preset validity period expires; when the bandwidth monitoring subunit captures the write backpressure signal of the peer chip, the dynamic arbitration and binding adjustment subunit reduces the arbitration weight of the corresponding DMA channel on the current external bus interface, and restores the arbitration weight after the backpressure signal disappears.

[0011] In the above implementation process, the bandwidth monitoring subunit monitors the utilization rate and backpressure status of each external bus interface in real time, enabling the system to detect congestion promptly. The service quality tag identification subunit extracts priority tags from the microcode, allowing the system to distinguish the latency sensitivity of different data streams. The dynamic arbitration and binding adjustment subunit temporarily redirects high-priority data streams to idle interfaces and restores the original binding relationship after the validity period, solving the problem of critical stream blocking while maintaining the determinism of default binding. Furthermore, the backpressure linkage mechanism proactively reduces the sender's weight when the receiver is congested, reducing unnecessary bus resource usage. These mechanisms work together to dynamically allocate bus bandwidth according to the real-time needs of the data stream, reducing the waiting time of high-priority tasks and improving the utilization efficiency of bus resources.

[0012] Optionally, in this embodiment, the synchronization management unit includes: an asynchronous synchronization and out-of-order release subunit, used to maintain a dependency mask configured by software and a wait counter for each computing core or chip; when a computing core or chip completes its local task, it decrements the wait counters of other cores or chips that depend on the completed core or chip by one, and releases the corresponding computing core or chip when the wait counters reach zero, without waiting for other undependent cores or chips; and a synchronization signal merging and interrupt subunit, used to collect the completion signals of each core or chip, and send an interrupt signal to the central processing unit when the preset synchronization conditions are met.

[0013] In the above implementation, the asynchronous synchronization and out-of-order release subunits maintain dependency masks and wait counters. When a computing core completes its local task, the wait counters of other cores that depend on that core are decremented by one, and the corresponding core or chip is immediately released when the counters reach zero, without waiting for other undependent cores, thus reducing the idle time of computing resources. The synchronization signal merging and interrupt subunits collect completion signals and send interrupts uniformly when synchronization conditions are met, reducing the need for separate interrupt triggers when each core completes. This transforms the synchronization operation from the forced global wait of traditional barriers to fine-grained waits that depend only on the path, improving the parallel processing efficiency of multi-core or multi-chip systems and reducing the CPU's interrupt response frequency.

[0014] Optionally, in this embodiment, the synchronization management unit supports multiple synchronization points being active simultaneously, with each synchronization point independently maintaining a set of dependency masks and wait counters for parallel synchronization.

[0015] In the above implementation, by supporting multiple synchronization points to be active simultaneously, each synchronization point independently maintains a dependency mask and a wait counter, ensuring that synchronization at different levels does not block each other. Once a core completes its current level task, it can independently determine whether to continue execution based on the dependencies of the next level, without waiting for other cores to complete their current level tasks. This parallel synchronization mechanism reduces the serial waiting time between cores, enabling pipelined execution of each stage in the entire computation process, thereby improving the overall execution efficiency of multi-core or multi-chip systems.

[0016] Optionally, in the embodiments of this application, when the asynchronous synchronization and out-of-order release subunit detects that any computing core or chip has entered a blocked state due to unmet dependencies, it actively notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core or chip.

[0017] In the above implementation process, when the asynchronous synchronization and out-of-order release subunit detects that a computing core or chip has entered a blocked state due to unmet dependencies, it actively notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core or chip. This allows bus resources that would otherwise be wasted during the blocking period to be released and allocated to other cores or chips that are still working normally, reducing the ineffective occupation of bandwidth resources. When the blocked core is released and resumes operation, its bandwidth quota is restored accordingly. This linkage mechanism improves the overall utilization efficiency of bus resources and reduces resource idleness caused by synchronization waiting.

[0018] Secondly, embodiments of this application also provide an inter-core and inter-chip interconnection method, applied to a system containing multiple computing cores or multiple chips. The method includes: a central processing unit (CPU) initiating a DMA channel of an initiating chip, executing a microcode program in a microcode cache through the DMA channel of the initiating chip, encapsulating local data and microcode instructions into a request packet and sending it to an adjacent chip; wherein the microcode program is written by the CPU to the microcode cache of the DMA channel of the initiating chip; the adjacent chip receives the request packet, parses the attached microcode instructions, performs atomic operations on the local data and intermediate data in the request packet to generate a new request packet and forwards it until the destination chip; after completing the operation, the destination chip returns a return packet containing the final result along the original path, and each chip writes the final result to its local address; after completing the final result writing, the initiating chip sends a completion interrupt signal to the CPU; wherein the system monitors the data flow status and priority in real time to dynamically adjust the data transmission channel, and asynchronously releases each computing core or chip according to pre-configured inter-core or inter-chip dependencies.

[0019] In the above implementation process, the microcode program written by the CPU once is executed through the DMA channel of the initiating chip. Local data and microcode instructions are encapsulated into a request packet and forwarded tier by tier along a chained path. Each adjacent chip parses the microcode instructions attached to the packet, performs atomic operations, and continues forwarding. The destination chip completes the operation and returns the result along the original path. Each chip writes the final result to its local address. The initiating chip sends only one interrupt signal to the CPU after completion. Throughout the data stream transmission process, the system synchronously implements dynamic bandwidth adjustment to reduce the blocking of high-priority data streams; simultaneously, it asynchronously releases each core according to pre-configured dependencies, reducing the forced waiting of cores without dependencies. These steps work together to enable cross-chip reduction operations to be completed autonomously by the hardware after a single CPU configuration, reducing the frequency of CPU involvement and improving multi-chip collaborative efficiency.

[0020] Optionally, in this embodiment, the step of dynamically adjusting the data transmission channel includes: real-time monitoring of the utilization rate of each external bus interface and the backpressure status of the DMA channel to obtain statistical results; extracting the quality of service priority tag from the microcode instruction; based on the statistical results and the quality of service priority tag; when the current binding interface of the high-priority data stream is in a congested state, redirecting the DMA channel of the data stream, and restoring the original binding relationship of the DMA channel after the preset validity period expires; the step of asynchronously releasing each computing core or chip according to the pre-configured inter-core or inter-chip dependency relationship includes: configuring the dependency mask of each computing core or chip, and initializing the wait counter of each core or chip according to the dependency mask; when any computing core or chip completes its local task, decrementing the wait counter of other cores or chips that depend on the completed core or chip by one; immediately releasing the core or chip whose wait counter is zero, allowing it to continue executing subsequent computing tasks; when all dependencies corresponding to any synchronization point are satisfied, sending a stage completion interrupt signal to the central processing unit.

[0021] In the above implementation process, by real-time monitoring of the utilization and backpressure status of the external bus interface, combined with the service quality priority tag extracted from the microcode instructions, the DMA channel is redirected to an idle interface when high-priority data streams encounter congestion and automatically restored after the expiration period, thus achieving dynamic allocation of bus resources. Simultaneously, by configuring dependency masks and initializing wait counters, the counters of other cores dependent on a core are decremented when the core completes its local task, and the corresponding core is immediately released when the counter reaches zero. An interrupt is sent only when all dependencies at the synchronization point are satisfied, achieving fine-grained asynchronous synchronization. The combination of these two methods allows data transmission and synchronization operations to be completed autonomously by the hardware, reducing CPU intervention and improving overall system efficiency.

[0022] Thirdly, embodiments of this application also provide a computer program product, including computer program instructions, which are executed by a processor to perform the method provided in the first aspect or any implementation thereof.

[0023] Fourthly, embodiments of this application also provide a multi-chip cascade system, which includes multiple chips, at least one of which includes the inter-core and inter-chip interconnection device provided in the first aspect or any implementation thereof.

[0024] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, perform the method provided in the first aspect or any implementation thereof.

[0025] This application provides an inter-core and inter-chip interconnection device and method. By embedding microcode cache and microcode program counter in the DMA channel, the DMA channel can independently execute microcode programs pre-downloaded by the CPU. Subsequent data transfer and computation are all completed autonomously by the DMA channel, eliminating the need for the CPU to participate in configuration and interrupt responses sequentially. This significantly reduces the CPU's involvement frequency, lowers system software overhead, and improves the overall efficiency of multi-chip collaborative work. By setting up a data flow-aware dynamic bandwidth allocation unit, the flow status and backpressure signals of each DMA channel are monitored in real time, and the binding relationship between the DMA channel and the external bus interface is dynamically adjusted according to the data flow priority, improving the overall utilization efficiency of bus bandwidth. By setting up a synchronization management unit, the dependencies between cores or chips are maintained, and the release timing of each computing core or chip is controlled according to the dependencies. Synchronization only occurs between cores with dependencies; cores without dependencies can continue execution in advance after meeting their own conditions without waiting for other unrelated cores, thereby reducing the idle waiting time of computing resources and improving the parallel processing efficiency of multi-core or multi-chip systems. Through the coordinated work of the DMA channel array unit, dynamic bandwidth allocation unit, and synchronization management unit, the hardware autonomously completes cross-chip data transfer, computation, and synchronization operations after a single CPU configuration, thereby improving the overall operating efficiency of the multi-chip system. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This application provides a schematic diagram of the structure of an inter-core and inter-chip interconnect device. Figure 2 This is a structural diagram of the DMA subchannel module provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a dynamic bandwidth allocation unit provided in an embodiment of this application; Figure 4 This application provides a schematic diagram of the structure of a synchronization management unit according to an embodiment of the present application. Figure 5 A dependency relationship diagram provided for embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0030] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0031] As the scale of large-scale artificial intelligence models continues to expand, a single chip can no longer independently handle complete model inference tasks. Typically, multiple computing chips or cores are combined into a system through interconnection technology to work collaboratively. In such multi-chip or multi-core systems, data transfer, computation, and synchronization between chips are critical factors affecting overall performance.

[0032] Currently, data transfer between chips is typically handled by a DMA controller. Traditional DMA controllers require CPU intervention to configure the source address, destination address, and transfer length for each data transfer, and the DMA then notifies the CPU via an interrupt after the transfer is complete. When there are numerous cross-chip data transfers in the system, the CPU needs to repeatedly intervene in configuration and interrupt responses, resulting in frequent interrupts and significant software overhead. Although linked-list DMA systems exist that can chain multiple transfer tasks together, each node in the list remains a simple transfer operation and lacks the ability to perform calculations on the data during the transfer process.

[0033] In multi-chip collaborative communication scenarios, such as All-Reduce operations, each chip needs to aggregate its local data and broadcast the results. Existing implementations typically rely on multiple DMA calls at the software layer to complete data round trips. The number of communications increases with the number of chips, leading to higher latency. Furthermore, system bus bandwidth allocation often uses fixed priority or simple polling. When multiple data streams are transmitted together, high-latency-sensitive data streams may be blocked due to low-priority background tasks occupying bandwidth. Regarding synchronization, existing solutions often employ a global barrier mechanism, requiring all participating cores or chips to reach the synchronization point before execution can continue. This forces some cores, even after completing their tasks, to wait for other cores, resulting in idle computing resources.

[0034] This application discloses an inter-core and inter-chip interconnect device. By embedding a microcode cache and microcode program counter within the DMA channel, the DMA channel can independently execute microcode programs pre-downloaded by the CPU. Subsequent data transfer and computation are all completed autonomously by the DMA channel, eliminating the need for the CPU to participate in configuration and interrupt responses sequentially. This significantly reduces the CPU's involvement frequency, lowers system software overhead, and improves the overall efficiency of multi-chip collaborative operation. By setting up a data flow-aware dynamic bandwidth allocation unit, the flow status and backpressure signals of each DMA channel are monitored in real time, and the binding relationship between the DMA channel and the external bus interface is dynamically adjusted according to the data flow priority, improving the overall utilization efficiency of bus bandwidth. By setting up a synchronization management unit, inter-core or inter-chip dependencies are maintained, and the release timing of each computing core or chip is controlled according to these dependencies. Synchronization only occurs between cores with dependencies; cores without dependencies can continue execution ahead of schedule after meeting their own conditions, without waiting for other unrelated cores. This reduces the idle waiting time of computing resources and improves the parallel processing efficiency of multi-core or multi-chip systems. Through the coordinated work of the DMA channel array unit, dynamic bandwidth allocation unit, and synchronization management unit, the hardware autonomously completes cross-chip data transfer, computation, and synchronization operations after a single CPU configuration, thereby improving the overall operating efficiency of the multi-chip system.

[0035] Please see Figure 1 The diagram shown is a structural schematic of an inter-core and inter-chip interconnect device provided in an embodiment of this application. This inter-core and inter-chip interconnect device is applied to a system containing multiple computing cores or multiple chips. The interconnect device comprises a DMA channel array unit, a data flow-aware dynamic bandwidth allocation unit, and a synchronization management unit, wherein the DMA channel array unit, the dynamic bandwidth allocation unit, and the synchronization management unit are coupled to each other. Each unit is described below.

[0036] (a) DMA channel array unit, which contains multiple DMA channels. Each DMA channel has a built-in microcode cache and microcode program counter, which are used to independently execute microcode programs pre-downloaded to the central processing unit to directly complete atomic operations and data forwarding on the data transport path.

[0037] In this embodiment, the DMA channel array unit is a module responsible for data transport and processing. Its basic function is to upgrade the traditional DMA from a "CPU-controlled transporter" to an "independently programmable data processing engine".

[0038] First, we will introduce the overall architecture of the DMA channel array unit. The DMA channel array unit is integrated inside the interconnect device, independent of multiple computing cores (such as NPU cores) on the chip. This unit contains multiple sets of DMA channels, as well as multiple Master read subunits and multiple Master write subunits.

[0039] The multi-Master read subunit includes multiple sets of read Master ports supporting the AXI bus protocol, used to interface with the read data channels in each DMA sub-channel. In one specific implementation, a full-connection mode for the read Master ports can be adopted, meaning each read Master port connects to all RDMAs in all DMA channels. The multi-Master write subunit also includes multiple sets of AXI bus protocol Master write ports, using a full-connection mode to interface with the WDMAs in each sub-channel. The full-connection mode is only one exemplary implementation of this application; other connection methods, such as partial connections or cross connections, can also be used.

[0040] The Master read port and Master write port are the physical interfaces between the DMA channel array unit and the external bus. The Master read port is used to issue read requests to read data from off-chip memory (such as DDR or PCIe space); the Master write port is used to issue write requests to write data to the target memory. These Master ports act as a bridge between the DMA channel and the off-chip memory system, allowing the DMA channel to complete data transfer without directly handling bus protocol details, simply by using the Master ports.

[0041] Each DMA channel is a basic execution unit in the DMA channel array. Multiple DMA channels can work in parallel, handling multiple different data transfer and computation tasks simultaneously. The microcode buffer and microcode program counter inside each DMA channel enable it to operate independently of other channels, without interference between them.

[0042] The internal structure of a DMA channel is described below. Each DMA channel contains a first sub-channel and a second sub-channel, which can be used to interact with the chips on the left and right sides of the cascaded path, respectively. For example, the first sub-channel can be used to read and write data with the chip on the left, and the second sub-channel can be used to read and write data with the chip on the right, thereby isolating the data flow on the left and right sides and reducing mutual interference between the data read and write operations of the chips on both sides. Each sub-channel is completely identical in hardware function and has complete read, calculation, and write capabilities.

[0043] Please see Figure 2 The illustrated DMA subchannel module structure diagram provided in this application embodiment, taking a single subchannel as an example, includes the following components: The atomic operation decoder and dispatcher has its input connected to a microcode buffer and its output connected to an arithmetic array and a remote atomic operation module. This module is used to parse the microcode instructions retrieved from the microcode buffer, generate corresponding arithmetic control signals based on the opcode field in the instructions, and dispatch the data stream to the arithmetic unit (when local arithmetic is required) or the forwarding path (when data needs to be directly passed through to the next-level chip) based on the routing information in the instructions.

[0044] The remote atomic operation module handles atomic operation requests from other chips. When this chip acts as an intermediate node in a chained protocol, it receives request packets from the preceding chip, performs address locking to ensure the atomicity of the operation, and after completing the read-modify-write sequence, forwards the result to the next chip. The remote atomic operation module can also maintain a waiting queue. When multiple sources access the same address simultaneously, the queue mechanism ensures the sequential execution and atomicity of each access operation.

[0045] The first read data channel (RDMA0) and the second read data channel (RDMA1) independently read data from on-chip or off-chip memory. Both read data channels support DDR and PCIe memory spaces and can read data from different memory addresses in parallel, providing two simultaneous inputs for subsequent processing arrays. During operation, RDMA0 and RDMA1 can read data from two different physical memory banks (e.g., different banks of DDR, or one from DDR and one from PCIe space), thereby maximizing the parallelism of data reading.

[0046] The configurable arithmetic array has its inputs connected to the first and second read data channels and its output connected to the write data channel. This array performs corresponding operations based on the operation mode field in the microcode instructions, such as atomic addition, atomic up / down operations, and atomic multiplication, to complete the fusion calculation of the two input data channels. The arithmetic array is configurable, meaning the type of operation it performs is dynamically specified by the operation mode field in the microcode instructions; different microcode instructions can trigger different operations.

[0047] Write Data Path (WDMA) is used to write the output of the computation array or pass-through data back to the target memory. In chained reduction scenarios, WDMA is also responsible for encapsulating the accumulated value into a new request packet and forwarding it across chips to the next level node.

[0048] The following describes the working principle of the microcode execution mechanism. Each DMA channel is equipped with an independent microcode cache and microcode program counter. The microcode cache is a non-volatile or volatile storage medium (such as SRAM) used to store the microcode program pre-downloaded by the CPU. The microcode program counter is a hardware register used to point to the address of the currently executing microcode instruction in the microcode cache.

[0049] During operation, the CPU first downloads the written microcode program to the microcode cache of the designated DMA channel via the configuration interface, and then writes a start value to the start register of that DMA channel. After the start register is set, the microcode program counter of the DMA channel begins to increment from a preset starting address (usually the first address of the microcode cache). In each clock cycle or each execution cycle, the microcode instruction pointed to by the program counter is fetched and sent to the atomic operation decoder and dispatcher for parsing.

[0050] After parsing, the decoder and dispatcher perform different operations according to the instruction type: for transport instructions, it controls RDMA or WDMA to perform data read and write; for arithmetic instructions, it configures the arithmetic array to perform the specified operation and triggers the start of the operation; for routing instructions, it determines the next hop target of the data packet; for synchronization instructions, it interacts with the synchronization management unit to wait for or release signals.

[0051] After executing an instruction, the microcode program counter automatically increments by one, pointing to the next microcode instruction, without CPU intervention. When the program counter points to the end of the microcode cache or a program termination instruction is encountered, the DMA channel stops execution and sends a completion interrupt to the CPU (e.g., the initiating chip of a chained protocol sends an interrupt after receiving a return packet and completing local writing).

[0052] Through the above mechanism, the CPU only needs to configure the microcode program once, and the DMA channel can autonomously complete the entire data transfer and calculation process. The CPU does not need to participate in configuration and respond to interrupts one by one during the process.

[0053] In a cross-chip chain atomic reduction scenario, assume that the multi-chip cascaded system includes an initiating chip, at least one adjacent chip, and an end chip. The CPU pre-writes the microcode program into the microcode cache of the initiating chip's DMA channel. The microcode program contains arithmetic instructions, such as specifying atomic addition and arithmetic modes, and routing instructions, such as specifying the next-hop chip identifier and the chain end flag.

[0054] After the DMA channel of the initiating chip is started, it reads local data through RDMA, obtains the initial accumulated value through the arithmetic array, and then constructs a request packet through WDMA and sends it to the next-level chip. The format of the request packet includes a header and a data payload.

[0055] After receiving a request packet, the adjacent chip performs atomic operations to decode and the dispatcher to parse the microcode instructions attached to the packet. It then reads local data via RDMA, performs accumulation via the arithmetic array, and finally constructs a new request packet via WDMA and sends it to the next-level chip. After the final chip completes the final accumulation, it writes the result to its local address and triggers the return flag, returning the result along the original path.

[0056] Throughout the chained reduction process, the DMA channels of each chip execute autonomously according to the microcode instructions. The CPU only configures the microcode program and the start register at the very beginning, and does not need to participate in the intermediate process.

[0057] It is understood that the number of Master ports described in this application (e.g., 4 sets of read Master ports and 2 sets of write Master ports) is merely illustrative. In practical applications, the number of Master ports can be flexibly configured according to factors such as system size and bandwidth requirements. For example, more Master ports can be set in systems with high bandwidth requirements, while fewer Master ports can be set in cost-sensitive systems.

[0058] Similarly, the number of DMA channels and the number of sub-channels (first sub-channel and second sub-channel) contained in each channel are merely examples. In practical applications, the number of DMA channels can be expanded to more as needed; for example, dozens or even hundreds of DMA channels can be set up in large multi-chip systems. The number of sub-channels is also not limited to two; in more complex cascaded topologies, each DMA channel can contain more sub-channels to connect chips in more directions.

[0059] Furthermore, the fully connected approach (i.e., each Master port connects to all RDMA or WDMA channels in all DMA channels) is one implementation method of this application, which has the advantages of flexible routing and multiple path options during redirection. However, in some application scenarios, partial or grouped connection methods can also be used, for example, dividing multiple DMA channels into several groups, with each group sharing a Master port.

[0060] (ii) A dynamic bandwidth allocation unit based on data flow awareness is coupled between the DMA channel array unit and the external bus interface. It is used to monitor the flow status of each DMA channel in real time and dynamically adjust the binding relationship between the DMA channel and the external bus interface according to the priority of the data flow.

[0061] Please see Figure 3 The diagram shown is a structural schematic of a dynamic bandwidth allocation unit provided in an embodiment of this application. The dynamic bandwidth allocation unit may include three sub-units: 1. The bandwidth monitoring subunit polls each DMA channel and each Master port at microsecond intervals. This subunit contains a statistics register that records the number of bytes read / written, average latency, and maximum latency for each DMA channel within a sliding time window. Additionally, this subunit monitors the utilization rate of each Master port and the interface backpressure signal. Through continuous sampling, the bandwidth monitoring subunit can monitor the congestion or idle status of each Master port in the system in real time.

[0062] 2. A Quality of Service (QoS) tag identification subunit (i.e., a QoS tag identification subunit) is used to extract the QoS field (Quality of Service Priority Tag) from the microcode instruction to be transmitted. In one specific embodiment of this application, the QoS value ranges from 0 to 15, with a higher value indicating a more latency-sensitive data stream. The QoS tag identification subunit compares this value with a preset threshold (e.g., 8) to determine whether the data stream is a high-latency-sensitive stream.

[0063] 3. The Dynamic Arbitration and Binding Adjustment Subunit is used to decide whether to perform Master port redirection based on the input information from the Bandwidth Monitoring Subunit and the QoS Tag Identification Subunit, and then executes the specific redirection operation. This subunit internally maintains a binding table that records which Master port each DMA subchannel's RDMA and WDMA are currently bound to.

[0064] In an optional embodiment, assuming that at a certain moment, the bandwidth monitoring subunit detects that the utilization of a Master port is close to saturation, for example, reaching or exceeding a preset utilization threshold, and at the same time, the QoS tag identification subunit identifies a DMA channel to be transmitted with a high QoS value, such as an All-Reduce protocol data stream. At this time, the dynamic arbitration and binding adjustment subunit executes the following decision-making process: First, the binding table is queried to confirm the current default Master port bound to the DMA channel of this high QoS data stream. If the default Master port is currently congested, a redirection decision is made.

[0065] Secondly, the arbitration subunit scans the bandwidth monitoring data of other Master ports to find a Master port with low current utilization or that is completely idle. In an optional embodiment, the arbitration subunit preferentially selects a completely idle Master port for redirection to ensure that high-priority data streams can be transmitted without blocking; if no completely idle Master port is available, the Master port with the lowest utilization can also be selected for redirection.

[0066] Finally, the arbitration subunit establishes a temporary mapping relationship for the high QoS data stream, redirecting one or two read data channels of its DMA channel to an idle Master port, and sets the validity period of the temporary mapping. This validity period can be specified by the time field in the microcode instruction, or it can be dynamically calculated by the system based on the expected transmission time of the high-priority data stream.

[0067] During the temporary mapping period, data for this DMA channel is transmitted through the redirected Master port. When the validity period expires, the arbitration subunit automatically clears the redirection rule, and the DMA channel resumes using the default binding relationship. If there are new data requests during the validity period, the redirected Master port will continue to be used; if the redirected Master port also becomes congested, the arbitration subunit can re-determine and redirect the data stream to another idle Master port.

[0068] The following example illustrates the complete process of redirection and recovery. Assume the system contains multiple DMA channels and a four-port read Master subunit, where read Master port 0 is bound to the RDMA of multiple DMA channels by default. At a certain moment, a low-priority background task consumes most of the bandwidth of read Master port 0. At this time, a high-priority All-Reduce task initiates a request, and its default-bound read Master port 0 is congested.

[0069] The arbitration subunit detects that Read Master Port 1 and Read Master Port 2 are relatively idle. Therefore, it redirects one read data channel of the high-priority task to Read Master Port 1 and the other to Read Master Port 2, setting a validity period T. During this period, the two read data channels of the high-priority task read data in parallel through Read Master Port 1 and Read Master Port 2, completing the transmission at full bandwidth. The low-priority background task continues to run on its original Master port unaffected. After the validity period expires, the high-priority task's read data channel resumes transmission through Read Master Port 0.

[0070] The dynamic bandwidth allocation unit also includes a backpressure linkage mechanism. Backpressure refers to the pause signal sent by the receiver to the sender when the receiver buffer of the peer chip is full or unable to receive more data. When the sender's write Master port continuously receives backpressure signals, it means that the processing capacity of the peer chip is saturated, and continuing to send data may lead to retransmission or buffer overflow.

[0071] After capturing the backpressure signal on the Master port, the bandwidth monitoring subunit transmits this information to the dynamic arbitration and binding adjustment subunit. The arbitration subunit determines whether the backpressure has continuously exceeded a preset number of cycles. If so, it proactively reduces the arbitration weight of the corresponding DMA channel on the current Master port, thereby reducing its bandwidth usage and transferring the released bandwidth quota to other channels that can still transmit normally.

[0072] Once the backpressure signal disappears and the system returns to normal for several consecutive cycles, the arbitration subunit restores the arbitration weight of the DMA channel to its default value. The number of normal cycles M detected during recovery is typically less than the number of continuous backpressure cycles N detected when weight reduction is triggered, in order to avoid frequent weight adjustment and recovery operations due to occasional short-term backpressure, thereby improving system stability.

[0073] Through the aforementioned counter-pressure linkage mechanism, the dynamic bandwidth allocation unit can not only cope with congestion caused by multiple data streams competing for data, but also with blocking caused by insufficient processing capacity at the receiving end, further improving the utilization efficiency of bandwidth resources.

[0074] (iii) Synchronization management unit, coupled to each computing core or chip, is used to maintain the dependency relationship between cores or chips and control the release timing of each computing core or chip according to the dependency relationship.

[0075] The synchronization management unit consists of two sub-units: 1. The synchronization signal merging and interrupt subunit receives synchronization and completion signals from each computing core or chip via its input interface, and connects to the CPU's interrupt controller via its output interface. This subunit supports multi-channel signal merging and can simultaneously monitor the status of multiple cores or chips. When preset conditions are met (e.g., all dependencies corresponding to a synchronization phase are satisfied), this subunit sends an interrupt signal to the CPU; when an exception occurs (e.g., timeout or deadlock), this subunit also sends an exception interrupt to the CPU to notify the software to handle the exception. This subunit ensures compatibility with the standard CPU interrupt interface.

[0076] 2. The Asynchronous Synchronization and Out-of-Order Release Subunit is the core module of the synchronization management unit, responsible for maintaining dependency relationships and making release decisions. Internally, this subunit maintains a dependency mask table and a set of wait counters. The dependency mask table, written by the CPU during the configuration phase, describes the dependencies between cores or chips. The wait counters record the number of incomplete cores that each core or chip is currently waiting for. When the wait counter for a core or chip reaches zero, the subunit immediately releases that core or chip. This subunit also includes a communication interface with the dynamic bandwidth allocation unit, used to proactively notify the bandwidth allocation unit to reduce the bandwidth quota when blocked.

[0077] The following describes the configuration and maintenance of dependencies. During system initialization or at the start of each computation phase, the CPU writes a dependency mask to the synchronization management unit via the configuration interface. A dependency mask is a data structure used to represent dependency graphs between cores or chips. In one specific implementation, each core or chip corresponds to a dependency mask, where each bit in the mask represents another core or chip, and a value of "1" indicates that the represented core or chip needs to wait for the represented core or chip to complete.

[0078] For example, in a system containing four cores, A, B, C, and D, if B depends on A, C depends on A, and D has no dependency, then the dependency mask written by the CPU is: A's mask is 0 (representing no dependency), B's mask is A, C's mask is A, and D's mask is 0.

[0079] After receiving the dependency mask, the synchronization management unit initializes a wait counter for each core or chip. The initial value of the wait counter is equal to the number of other cores or chips that the core or chip depends on. In the example above, A's wait counter is 0 (dependency count is 0), B's wait counter is 1 (depends on A), C's wait counter is 1 (depends on A), and D's wait counter is 0 (dependency count is 0).

[0080] During operation, when a core or chip completes its local computation task, it sends a completion signal to the synchronization management unit. Upon receiving the completion signal, the synchronization management unit, based on the dependency mask, identifies all other cores or chips that depend on the completed core or chip and decrements their wait counters by one. When the wait counter of a core or chip changes from 1 to 0, the synchronization management unit immediately releases that core or chip, allowing it to begin subsequent computations.

[0081] Traditional barrier synchronization requires all cores to reach the synchronization point simultaneously before they can be released uniformly. However, the out-of-order release in this application allows each core to be released sequentially after satisfying its own dependency conditions. Let's continue with the example of the four cores mentioned above. The wait counters for A and D are initially set to 0, so A and D begin execution immediately after system startup without waiting for anyone. The wait counters for B and C are initially set to 1 (both waiting for A), so B and C enter a waiting state and do not consume computing resources.

[0082] When A completes, the synchronization management unit decrements the wait counters of B and C by one, bringing them both to 0. B and C are then immediately released and begin execution. D, since it depends on no one, had already begun execution before A and may already be complete. In this process, each core is released immediately after satisfying its own dependency conditions, without blocking each other.

[0083] In more complex multi-level dependency scenarios, a single synchronization point may involve dependencies on multiple cores. The synchronization management unit supports multiple synchronization points being active simultaneously, with each synchronization point independently maintaining a set of dependency masks and wait counters. Synchronization at different levels does not block each other. For example, synchronization at the first level (when the expert calculation at this level is completed) and synchronization at the second level (when the next level of calculation begins) can be active simultaneously. When a core is released after satisfying its dependency conditions at the first level, if it also has dependencies at the second level, it continues to wait or release at the synchronization point at the second level.

[0084] When the synchronization management unit detects that a core or chip has entered a blocked state due to unmet dependencies, although the core or chip has completed its own calculations, it cannot continue to execute subsequent tasks temporarily because the other cores it depends on have not yet completed. During this blocking period, the DMA channel of the core or chip may be in an idle state and no effective data transfer will occur.

[0085] The synchronization management unit sends a notification to the dynamic bandwidth allocation unit via the communication interface, containing the identifier of the blocked core or chip and its corresponding DMA channel information. Upon receiving the notification, the dynamic bandwidth allocation unit reduces the bandwidth quota of the DMA channel corresponding to the blocked core or chip. The released bandwidth resources can be allocated to other unblocked cores or chips, thereby improving the overall utilization of bus resources.

[0086] When the dependency conditions of a blocked core or chip are finally met, the synchronization management unit releases the core or chip and sends a notification to the dynamic bandwidth allocation unit again. Upon receiving the recovery notification, the dynamic bandwidth allocation unit restores the bandwidth quota of the DMA channel of the core or chip to the normal level, ensuring that it can obtain the necessary bandwidth in subsequent data transmissions.

[0087] The synchronization signal merging and interrupt subunit continuously collects completion signals from each core or chip. When all dependencies corresponding to a certain synchronization stage are satisfied (for example, in the MoE model in the document, the next layer synchronization point requires both B and C to be completed), the subunit determines that the current stage's computation task has ended, sends a stage completion interrupt signal to the CPU, and reports that the computation of that layer has been completed.

[0088] Throughout the process, the CPU only participates in writing the dependency mask and starting the synchronization management unit during the initialization configuration phase. No software intervention is required during the runtime phase, and it only receives an interrupt notification at the end of each synchronization phase.

[0089] In the above implementation process, by embedding microcode cache and microcode program counter in the DMA channel, the DMA channel can independently execute the microcode program pre-downloaded by the CPU. Subsequent data transfer and computation are all completed autonomously by the DMA channel, without the CPU needing to participate in configuration and interrupt response sequentially. This significantly reduces the CPU's involvement frequency, lowers system software overhead, and improves the overall efficiency of multi-chip collaborative work. By setting up a data flow-aware dynamic bandwidth allocation unit, the flow status and backpressure signal of each DMA channel are monitored in real time, and the binding relationship between the DMA channel and the external bus interface is dynamically adjusted according to the data flow priority, improving the overall utilization efficiency of bus bandwidth. By setting up a synchronization management unit, the dependencies between cores or chips are maintained, and the release timing of each computing core or chip is controlled according to the dependencies. Synchronization only occurs between cores with dependencies. Cores without dependencies can continue execution in advance after meeting their own conditions, without waiting for other unrelated cores, thereby reducing the idle waiting time of computing resources and improving the parallel processing efficiency of multi-core or multi-chip systems. Through the coordinated work of the DMA channel array unit, dynamic bandwidth allocation unit, and synchronization management unit, the hardware autonomously completes cross-chip data transfer, computation, and synchronization operations after a single CPU configuration, thereby improving the overall operating efficiency of the multi-chip system.

[0090] Optionally, in this embodiment, each DMA channel includes a first sub-channel and a second sub-channel, which are used for data interaction with chips on both sides in the cascade direction, respectively. For example, the first sub-channel can be used for data read / write with the left-side cascaded chip, and the second sub-channel can be used for data read / write with the right-side cascaded chip. By allocating the data flow on the left and right sides to different sub-channels, the data read / write operations of the chips on both sides are hardware-isolated, thereby reducing mutual interference between the data read / write operations on the left and right sides and improving the parallelism of data transfer.

[0091] Please continue reading Figure 2 The diagram shown below illustrates the DMA subchannel module structure provided in this application embodiment. Each subchannel includes: The atomic operation decoder and dispatcher parses microcode instructions, generates operation control signals and remote atomic operation lock management commands, and dispatches data streams. Parsing microcode instructions: Microcode instructions contain opcode fields, operation mode fields, next-hop identifier fields, etc. The atomic operation decoder and dispatcher decodes these fields to identify the type of the current instruction and the specific operation requirements. Then, based on the decoding result, it sends operation control signals to the configurable operation array and lock management commands to the remote atomic operation module to ensure atomicity when multiple sources access the same address.

[0092] The remote atomic operation module handles atomic operation requests from other chips to ensure the atomicity of accesses. Specifically, upon receiving a request from another chip, it first locks the target address, then executes a read-modify-write sequence to read the current value, performs a calculation with the input value, writes the result back, and finally forwards it to the next-level chip. When multiple sources simultaneously request access to the same address, this module maintains a waiting queue to ensure that requests are executed in order.

[0093] The first and second read data channels are used to read data from on-chip or off-chip memory in parallel. They operate independently, reading data simultaneously from different memory locations, and serve as two inputs to the computation array. Each read data channel includes address generation logic, read request sending logic, and a data receive buffer. The advantage of parallel operation of the two read data channels is that when two-way data fusion calculations are required, data can be read from two different memory banks simultaneously, shortening the data read time.

[0094] A configurable arithmetic array is used to perform arithmetic operations on two input data channels. The two inputs of the configurable arithmetic array are connected to the first and second read data channels respectively, and the output is connected to the write data channel. The type of operation performed by the arithmetic array is dynamically specified by the operation mode field in the microcode instruction, such as atomic addition, atomic scaling, and atomic multiplication. In the hardware implementation, the arithmetic array contains multiple arithmetic logic units, and a selector selects different operation paths according to the operation mode field.

[0095] The write data channel is used to write out the operation results or pass-through data. It is also used to write the operation results or pass-through data back to the target memory. The write data channel input is connected to the output of the configurable arithmetic array or directly to the read data channel. The write data channel includes address generation logic and write request sending logic. In a chained reduction scenario, the write data channel is also responsible for encapsulating the accumulated value into a request packet and forwarding it across chips to the next-level node.

[0096] In the implementation of the above embodiments: each DMA channel is configured with a first sub-channel and a second sub-channel, respectively used for data interaction with the chips on both sides in the cascade direction, so that the data flow on the left and right sides is isolated from each other, reducing mutual interference. Atomic operation decoding and dispatching are used to parse microcode instructions and dispatch data streams, in conjunction with a remote atomic operation module to handle atomic operation requests from other chips. This allows cross-chip atomic operations to be completed autonomously by hardware without CPU intervention, reducing latency and software overhead in cross-chip operations. Data is read in parallel through the first and second read data channels, and the configurable arithmetic array dynamically selects the operation mode according to the microcode instructions to complete the fusion calculation of the two data streams. This allows data transfer and computation to be completed in the same hardware pipeline, reducing the number of data transfers. The write data channel writes the computation results and supports cross-chip forwarding, allowing the processed data to be directly passed to the next-level chip, completing the data transfer in the chained protocol without CPU intervention.

[0097] Optionally, in this embodiment of the application, the microcode program includes an extensible set of microcode instructions, and the microcode instructions include at least one of the following fields: The opcode field specifies the instruction type, which includes at least one of the following: transport instructions, arithmetic instructions, synchronization instructions, and routing instructions. Transport instructions control the simple movement of data from the source address to the destination address; arithmetic instructions trigger atomic operations on the arithmetic array; synchronization instructions interact with the synchronization management unit; and routing instructions determine the forwarding target of the data packet. In actual encoding, the opcode field can use fixed-width binary encoding, with different encoding values ​​corresponding to different instruction types. The decoder then distributes the instructions to different execution units accordingly.

[0098] The operation mode field specifies the type of atomic operation. Operation types include atomic addition, atomic maximum, atomic minimum, and atomic multiplication. In actual encoding, this field uses multi-bit binary encoding; different encoded values ​​correspond to different operation types. The decoder configures the operation array to the corresponding operation mode using a selector.

[0099] The next-hop identifier field specifies the target for chained forwarding. In a multi-chip cascaded system, each chip has a unique identifier. After an adjacent chip completes its local accumulation, it needs to forward the request packet to the next-level chip; this field serves as the identifier for that next-level chip. The decoder extracts this field value, and the write data channel uses it to send the request packet to the corresponding adjacent chip.

[0100] The Chain End Flag field is used to identify whether the current node is the end of the chain. Specifically, it indicates whether the current chip is the last node in the chained reduction. This field is a flag; for example, when this bit is "true," it means the current chip is the end, and the final result only needs to be written to the local address and the return flag triggered; when this bit is "false," it means the current chip is an intermediate node, and request packets need to be forwarded.

[0101] The broadcast address field specifies a unified base address for each chip to store the final result, allowing for backward broadcast writing. During backward broadcasting, each chip writes the final result to its local memory according to the base address indicated by this field. This field stores a base address value. When each chip receives a return packet, it uses this base address as the write target address. Since this address is an offset relative to each chip's local address space, each chip only needs to add it to the starting address of its local memory to obtain the final write address.

[0102] The Quality of Service (QoS) Priority Tag field is used by the Dynamic Bandwidth Allocation Unit (DBLA) to identify the latency sensitivity of a data stream. This field stores a priority value; a higher value indicates a more latency-sensitive data stream. The QoS Tag Identification Subunit within the DBLA extracts this field value to determine the latency sensitivity level of the data stream. When a high-QoS data stream encounters congestion, the arbitration subunit temporarily redirects it to an idle Master port to ensure timely transmission.

[0103] In the implementation of the above embodiments: by defining an extensible microcode instruction set, the DMA channel can perform various types of operations. The opcode field allows a single microcode instruction to indicate different operations such as transport, computation, synchronization, or routing, without requiring sequential CPU intervention; the operation mode field enables the same computing array hardware to support multiple atomic operations; the next-hop identifier field and the chain endpoint flag field allow for flexible configuration of the forwarding path in the chained protocol; the broadcast address field simplifies the write configuration of backward broadcasts; and the quality of service priority label field enables the dynamic bandwidth allocation unit to identify the latency sensitivity of data streams, thereby ensuring the timely transmission of high-priority data streams. These fields work together to enable the microcode program to flexibly describe complex cross-chip collaborative tasks, reducing the frequency of CPU intervention.

[0104] Please continue reading Figure 3 The diagram shown illustrates a structure of a dynamic bandwidth allocation unit provided in an embodiment of this application. Optionally, in this embodiment, the dynamic bandwidth allocation unit includes: The bandwidth monitoring subunit polls the utilization rate of each external bus interface and the backpressure status of the DMA channels at preset intervals. The subunit calculates the current utilization rate by statistically analyzing the effective data transfer volume of each Master port within a sliding time window; simultaneously, it continuously monitors whether there are pause signals from the peer chip on the write data channels of each DMA channel. To accurately reflect the real-time status, this subunit counts the number of read / write bytes and the interface backpressure signal within the sliding window; the window size can be set according to actual needs. When the utilization rate of an external bus interface exceeds a preset threshold, or when a DMA channel continuously receives a backpressure signal, the bandwidth monitoring subunit transmits this information in real time to the dynamic arbitration and binding adjustment subunit.

[0105] The Quality of Service (QoS) Tag Identification Subunit extracts QoS priority tags from microcode instructions. In each microcode instruction, the QoS priority tag field stores a numerical value, which is set by the programmer when writing the microcode program based on the task's latency sensitivity. The QoS Tag Identification Subunit reads this field value from the currently pending microcode instruction and passes it to the Dynamic Arbitration and Binding Adjustment Subunit as the basis for determining the priority of that data stream.

[0106] The dynamic arbitration and binding adjustment subunit is used to redirect the DMA channel of the data stream that meets the priority requirements to an idle external bus interface based on the monitoring results of the bandwidth monitoring subunit and the identification results of the service quality label identification subunit, and restore the original binding relationship of the DMA channel after the preset validity period expires.

[0107] For example, a binding table can be maintained within the dynamic arbitration and binding adjustment subunit to record which external bus interface (i.e., which Master port) each DMA subchannel is currently bound to. When a high-priority data stream transmission request is received, the dynamic arbitration subunit queries the default binding interface for that request. If the bandwidth monitoring subunit reports that the interface is currently congested, it scans the utilization status of other interfaces, selects an idle or least utilized interface to establish a temporary binding relationship, and starts a validity timer. During the validity period, read or write data for this DMA channel is transmitted through the redirected interface. When the timer expires, the arbitration subunit clears the redirection rule, and the DMA channel reverts to the default binding relationship.

[0108] When the bandwidth monitoring subunit detects a write reverse pressure signal from the peer chip, the dynamic arbitration and binding adjustment subunit reduces the arbitration weight of the corresponding DMA channel on the current external bus interface and restores the arbitration weight after the reverse pressure signal disappears.

[0109] Write back pressure refers to a pause signal sent by the receiving end to the transmitting end when the receiving buffer of the peer chip is full or unable to continue receiving data. When the bandwidth monitoring subunit captures this signal and it persists for more than a preset number of cycles, the dynamic arbitration subunit determines that the transmission capacity of the channel is limited and actively reduces its arbitration weight. The arbitration weight determines the priority value of this channel when competing with other channels for bus usage rights. Reducing the weight means that the channel gets less bus time on the current interface, and the freed-up bandwidth can be used by other channels that can still transmit normally. When the back pressure signal disappears and normal operation resumes for several consecutive cycles, the arbitration subunit restores the arbitration weight of the channel to the default value, allowing it to regain its due bus usage priority.

[0110] In the implementation of the above embodiments: the bandwidth monitoring subunit monitors the utilization rate and backpressure status of each external bus interface in real time, enabling the system to detect congestion promptly; the service quality tag identification subunit extracts priority tags from the microcode, allowing the system to distinguish the latency sensitivity of different data streams; the dynamic arbitration and binding adjustment subunit temporarily redirects high-priority data streams to idle interfaces and restores the original binding relationship after the validity period, solving the problem of critical stream blocking while maintaining the determinism of default binding. Furthermore, the backpressure linkage mechanism proactively reduces the weight of the sending end when the receiving end is congested, reducing the unnecessary occupation of bus resources. The combined effect of these mechanisms allows bus bandwidth to be dynamically allocated according to the real-time needs of the data stream, reducing the waiting time of high-priority tasks and improving the utilization efficiency of bus resources.

[0111] Please continue reading Figure 4 The schematic diagram shown below illustrates a synchronization management unit according to an embodiment of this application. Optionally, in this embodiment, the synchronization management unit includes an asynchronous synchronization and out-of-order release subunit and a synchronization signal merging and interruption subunit: The asynchronous synchronization and out-of-order release subunit is used to maintain the dependency mask configured by software and the wait counter of each computing core or chip. When a computing core or chip completes its local task, it decrements the wait counters of other cores or chips that depend on the completed core or chip by one, and releases the corresponding computing core or chip when the wait counters reach zero, without waiting for other undependent cores or chips.

[0112] Dependency masks are written by the CPU during the configuration phase to describe dependencies between cores or chips. For example, "chip B depends on chip A" means that chip B can only continue execution after chip A has finished. The asynchronous synchronization and out-of-order release subunit maintains a set of registers, with each computing core or chip corresponding to a wait counter, whose initial value is equal to the number of other cores or chips that the core or chip depends on.

[0113] During operation, this subunit continuously monitors the completion signals of each computing core or chip. When a computing core or chip completes its local task, the subunit queries the dependency mask and decrements the wait counters of other cores or chips that depend on that completed core or chip by one. When the wait counter of a computing core or chip reaches zero, the subunit immediately releases the corresponding computing core or chip, allowing it to continue executing subsequent tasks. Since the completion times of each core are different, the timing of the wait counters reaching zero also differs. The release order is naturally related to the completion order. For example, cores that complete their dependencies first are released first, without waiting for other undependent cores or chips, thus achieving "out-of-order release".

[0114] In the specific hardware implementation, the dependency mask can be stored in a bitmap format, with each bit corresponding to a core or chip, and a bit value of "1" indicating the existence of a dependency relationship. This sub-unit contains comparator and adder logic. When a completion signal is received, it searches for all other cores that depend on the completed core based on the core's identifier and decrements their wait counters. When a counter value of zero is detected, a release signal is triggered.

[0115] The synchronization signal merging and interrupt subunit is used to collect the completion signals of each core or chip and send an interrupt signal to the central processing unit when the preset synchronization conditions are met.

[0116] The synchronization signal merging and interrupt subunit receives completion signal lines from various computing cores or chips, supporting the merging of multiple signals. Preset synchronization conditions can be that all dependencies corresponding to a certain synchronization stage have been met (e.g., in the MoE model, the next-layer synchronization point requires both B and C to be completed), or they can be triggered by an exception, such as a timeout or deadlock. When the condition is met, this subunit sends an interrupt to the CPU via the interrupt line, reporting the completion of the current stage of the computation task.

[0117] In the implementation of the above embodiments: A dependency mask and wait counters are maintained by the asynchronous synchronization and out-of-order release subunits. When a computing core completes its local task, the wait counters of other cores that depend on that completed core are decremented by one, and the corresponding core or chip is released immediately when the counter reaches zero, without waiting for other undependent cores, thus reducing the idle time of computing resources. The synchronization signal merging and interrupt subunits collect completion signals and send interrupts uniformly when the synchronization condition is met, reducing the need for separate interrupt triggers when each core completes. This transforms the synchronization operation from the forced global wait of traditional barriers to fine-grained waits that depend only on the path, improving the parallel processing efficiency of multi-core or multi-chip systems and reducing the CPU interrupt response frequency.

[0118] Optionally, in this embodiment, the synchronization management unit supports multiple synchronization points being active simultaneously, with each synchronization point independently maintaining a set of dependency masks and wait counters for parallel synchronization.

[0119] A synchronization point refers to a synchronization stage or synchronization level. For example, in a multilayer network of the MoE model, the synchronization boundary after each layer's computation is completed is a synchronization point. In a complex computation graph, multiple synchronization points at different levels may be active simultaneously. For instance, a first-level synchronization point (waiting for A to complete before releasing B and C) and a second-level synchronization point (waiting for B and C to complete before releasing the next layer) can coexist and work at the same time.

[0120] Each synchronization point independently maintains a set of dependency masks and wait counters. The dependency mask describes the dependencies between the cores or chips involved in that synchronization point, and the wait counter records the number of incomplete dependencies for each core or chip at that synchronization point. The dependency masks and wait counters of different synchronization points are independent of each other and stored in different register sets.

[0121] The meaning of "parallel synchronization" is that synchronization at different levels does not block each other, and each synchronization point proceeds independently. For example, when a core is released after satisfying its dependency condition at the first synchronization point, it begins executing the second level of computation; if the core has new dependencies at the second synchronization point, it needs to continue waiting for the release condition of the second synchronization point, but this waiting will not affect other cores at the first synchronization point. The counter update and release judgment of each synchronization point are performed in parallel, and there is no situation where one synchronization point must wait for another synchronization point to complete before it can work.

[0122] In one optional implementation, the synchronization management unit can internally set up multiple sets of register files, each corresponding to a synchronization point. When the CPU configuration depends on a mask, it also specifies which synchronization point the mask belongs to. During operation, the completion signal input and counter update operations of each synchronization point are executed in parallel within the same clock cycle, achieving pipelined parallel synchronization of multiple synchronization points.

[0123] In the implementation of the above embodiments: by supporting multiple synchronization points to be active simultaneously, each synchronization point independently maintains a dependency mask and a wait counter, ensuring that synchronization at different levels does not block each other. When a core completes its current level task, it can independently determine whether to continue execution based on the dependency relationship of the next level, without waiting for other cores to complete their current level tasks. This parallel synchronization mechanism reduces the serial waiting time between cores, enabling pipelined execution of each stage in the entire computation process, thereby improving the overall execution efficiency of multi-core or multi-chip systems.

[0124] Optionally, in this embodiment, when the asynchronous synchronization and out-of-order release subunit detects that any computing core or chip has entered a blocked state due to unmet dependencies, it actively notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core or chip.

[0125] When the wait counter of a computing core or chip has not yet reached zero, the core or chip is in a blocked state. That is, it has completed its own local computing task, but because other cores or chips it depends on have not yet completed, it cannot continue to execute subsequent tasks. During this blocking period, the DMA channel corresponding to the core or chip has no effective data transfer task and is in an idle or low-utilization state.

[0126] When the asynchronous synchronization and out-of-order release subunit detects that any computing core or chip has entered a blocked state due to unmet dependencies, it proactively sends a notification to the dynamic bandwidth allocation unit. This notification contains the identification information of the blocked core or chip. Upon receiving the notification, the dynamic bandwidth allocation unit reduces the bandwidth quota of the DMA channel corresponding to the blocked core or chip, for example, by reducing its arbitration weight on the external bus interface. The released bandwidth resources can then be allocated to other unblocked cores or chips, thereby avoiding the waste of bus resources caused by the idle state of blocked cores.

[0127] In one optional implementation, a dedicated communication interface or signal line exists between the asynchronous synchronization and out-of-order release subunit and the dynamic bandwidth allocation unit. When a subunit determines that a core has entered a blocked state, for example, if its wait counter has not changed within a preset time, it sends a deweighting request signal to the dynamic bandwidth allocation unit through this interface, carrying the channel number of the blocked core. Upon receiving this signal, the dynamic bandwidth allocation unit adjusts the arbitration weight register of the corresponding channel to reduce the bandwidth quota.

[0128] In the implementation of the above embodiments: when the asynchronous synchronization and out-of-order release subunit detects that a computing core or chip has entered a blocked state due to unmet dependencies, it actively notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core or chip. This allows bus resources that would otherwise be wasted during the blocking period to be released and allocated to other cores or chips that are still working normally, reducing the ineffective occupation of bandwidth resources. When the blocked core is released and resumes operation, its bandwidth quota can be restored accordingly. This linkage mechanism improves the overall utilization efficiency of bus resources and reduces resource idleness caused by synchronization waiting.

[0129] Please see Figure 2 The illustrated diagram shows a flowchart of an inter-core and inter-chip interconnection method provided in an embodiment of this application. The inter-core and inter-chip interconnection method provided in this application can be applied to electronic devices, which may include physical devices such as servers, PCs, tablets, or smartphones, or virtual devices such as virtual machines or containers. The electronic device can be a single device, a combination of multiple devices, or a cluster of a large number of devices. The inter-core and inter-chip interconnection method may include: Step S110: The central processing unit starts the DMA channel of the initiating chip, executes the microcode program in the microcode cache through the DMA channel of the initiating chip, and encapsulates the local data and microcode instructions into a request packet and sends it to the adjacent chip; wherein, the microcode program is written by the central processing unit to the microcode cache of the DMA channel of the initiating chip.

[0130] Step S120: The adjacent chip receives the request packet, parses the attached microcode instructions, performs atomic operations on the local data and the intermediate data in the request packet to generate a new request packet and forwards it until the destination chip.

[0131] Step S130: After the endpoint chip completes the calculation, it returns a return packet containing the final result along the original path, and each chip writes the final result to its local address.

[0132] Step S140: After the initiating chip completes the final result writing, it sends a completion interrupt signal to the central processing unit.

[0133] The system monitors the data stream's flow status and priority in real time to dynamically adjust the data transmission channel, and asynchronously releases each computing core or chip according to the pre-configured inter-core or inter-chip dependencies.

[0134] In step S110, the CPU first writes the prepared microcode program into the microcode cache of the designated DMA channel of the initiating chip via the configuration bus. This microcode program contains all the instructions required for subsequent chips to execute the chained reduction, including arithmetic instructions, routing instructions, and corresponding control information. The CPU then writes a startup value to the startup register of the DMA channel. After this, the entire process is autonomously driven by the hardware, and the CPU no longer participates. The started DMA channel retrieves instructions from the microcode cache, decodes them using atomic operations, and parses them using a dispatcher. It then reads data from local memory via the read data channel as the initial accumulated value. The write data channel then encapsulates the local data and the microcode instructions required by subsequent chips into a request packet according to a predetermined format. This request packet contains two parts: a data payload carrying the current accumulated intermediate value and an instruction payload carrying the microcode instructions to be executed by subsequent chips. After encapsulation, the write data channel sends the request packet to the adjacent chip in the cascaded direction via the inter-chip interconnect interface, based on the routing information in the microcode instructions.

[0135] In step S120, after an adjacent chip receives a request packet from the previous level through the inter-chip interconnect interface, its DMA channel's atomic operation decoder and dispatcher parses the microcode instructions attached to the packet to determine the type of operation to be performed (e.g., atomic addition, atomic maximum value, etc.). Then, the chip's read data channel reads local data from local memory, and the configurable arithmetic array performs fusion calculations with the intermediate data in the request packet according to the parsed operation mode to generate a new accumulated value. The write data channel repackages the new accumulated value and the remaining microcode instructions that have not yet been executed into a new request packet. At the same time, the atomic operation decoder and dispatcher determines the forwarding target based on the next-hop identifier field in the microcode instructions, and the write data channel sends the new request packet to the next-level chip. The above parsing, calculation, encapsulation, and forwarding operations are executed sequentially in each level of adjacent chips. The forwarding operation of the previous level chip and the receiving processing operation of the next level chip overlap in time, thereby improving the overall processing efficiency. This process is repeated until the request packet reaches the end chip of the chain topology.

[0136] In step S130, after receiving the request packet, the endpoint chip, during the atomic operation decoder and dispatcher's parsing of the microcode instructions, identifies the chain endpoint flag field indicating that it is the endpoint node. The endpoint chip performs the final atomic operation to obtain the global final result. The endpoint chip writes the final result to its local address and then constructs a return packet—the return packet contains the global final result and a return path identifier. The endpoint chip sends the return packet in reverse along the path from which the request packet originated to the next-level adjacent chip. After receiving the return packet, each adjacent chip writes the final result in the packet to its own local address and then forwards the return packet to the next level until the return packet reaches the initiating chip. During this process, the write data channels of each chip determine the write target address according to the broadcast address field in the microcode instructions, ensuring that each chip stores the result in a uniformly agreed-upon location.

[0137] In step S140, after receiving the return packet, the initiating chip writes the final result to its local address. Subsequently, the initiating chip's DMA channel determines that the entire chained reduction task has been completed and sends a completion interrupt signal to the central processing unit via the interrupt line. Regardless of the number of chips in the cascaded network, the CPU receives only this one interrupt notification throughout the entire process.

[0138] During the execution of steps S110 to S130 above, the system monitors the data flow status and priority in real time to dynamically adjust the data transmission channel, and asynchronously releases each computing core or chip according to the pre-configured inter-core or inter-chip dependencies.

[0139] The dynamic adjustment process includes, for example, the bandwidth monitoring subunit polling the utilization rate of each external bus interface and the backpressure status of each DMA channel at a preset period, and the QoS tag identification subunit extracting priority tags from microcode instructions. When the currently bound interface of a high-priority data stream is congested, the dynamic arbitration subunit temporarily redirects the DMA channel of that data stream to an idle external bus interface, restoring the original binding relationship after the preset validity period expires. Furthermore, when a write backpressure signal from the peer chip is detected, the arbitration weight of the corresponding DMA channel is reduced, and restored after the backpressure disappears.

[0140] The asynchronous release implementation is as follows: Before method execution, the CPU pre-configures the dependency masks for each core or chip through a configuration interface. The synchronization management unit initializes the wait counter for each core or chip based on the dependency masks. During method execution, when any computing core or chip completes its local task, the wait counter of the downstream core driven by that core is decremented by one. Cores with wait counters reaching zero are immediately released to continue executing subsequent tasks. When all dependencies corresponding to any synchronization phase are satisfied, the synchronization management unit sends a phase completion interrupt signal to the CPU. When any core enters a blocked state due to unmet dependencies, the synchronization management unit notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core, which then recovers after release.

[0141] In the above implementation process, the microcode program written by the CPU once is executed through the DMA channel of the initiating chip. Local data and microcode instructions are encapsulated into a request packet and forwarded tier by tier along a chained path. Each adjacent chip parses the microcode instructions attached to the packet, performs atomic operations, and continues forwarding. The destination chip completes the operation and returns the result along the original path. Each chip writes the final result to its local address. The initiating chip sends only one interrupt signal to the CPU after completion. Throughout the data stream transmission process, the system synchronously implements dynamic bandwidth adjustment to reduce the blocking of high-priority data streams; simultaneously, it asynchronously releases each core according to pre-configured dependencies, reducing the forced waiting of cores without dependencies. These steps work together to enable cross-chip reduction operations to be completed autonomously by the hardware after a single CPU configuration, reducing the frequency of CPU involvement and improving multi-chip collaborative efficiency.

[0142] The following example illustrates the complete execution flow of chained atomic reduction based on microcode execution mechanism using a specific application scenario.

[0143] The following application scenario is set: Four chips (A, B, C, D) are interconnected to form a multi-chip cascade system; each chip holds a local floating-point array gA, gB, gC, gD; the goal is to perform an All-Reduce addition operation so that all chips eventually obtain the global sum G = gA + gB + gC + gD; chip A is the initiator of the chain reduction and chip D is the endpoint.

[0144] Step 1: The CPU pre-writes the microcode instructions for implementing the chained atomic reduction into the microcode cache of the DMA channel of chip A. Then, the CPU writes the startup value to the startup register of the DMA channel of chip A to begin the entire operation. Throughout this process, the CPU only participates in this one-time configuration; subsequent operations are all completed automatically by the hardware.

[0145] Step 2: After the DMA channel of chip A is activated, the microcode program counter automatically increments, retrieving and executing instructions from the microcode cache one by one. The DMA channel reads local data gA as the initial accumulated value acc according to the microcode instructions. After the atomic operation decoder and dispatcher parses the microcode instructions, the write data channel constructs a new request packet. This request packet contains the current accumulated value acc and the microcode instructions required by chips B, C, and D. Based on the next-hop identifier field in the microcode instructions, the write data channel sends the request packet to the adjacent chip B through the inter-chip interconnect interface.

[0146] Step 3: After receiving the request packet, chip B's DMA channel's atomic operation decoder and dispatcher decodes the microcode instructions attached to the packet. It reads local data gB through the read data channel, and the configurable arithmetic array performs an atomic addition operation on gB and the accumulated value acc in the request packet, generating a new accumulated value acc = acc + gB. The write data channel repackages the new accumulated value and the remaining microcode instructions into a new request packet and sends it to chip C according to the next-hop identifier field.

[0147] Chip C performs the same operation as chip B: receiving the request packet, parsing the microcode instruction, reading the local gC, accumulating to obtain acc=acc+gC, encapsulating the new request packet, and sending it to chip D.

[0148] Chip D, acting as the chain endpoint, receives a request packet, parses the microcode instructions, and identifies the chain endpoint flag field indicating that it is the endpoint node. Chip D reads its local value gD and performs atomic addition with the accumulated value to obtain the global final result acc = gA + gB + gC + gD. Chip D writes the final result to its local address, sets the return flag, and initiates the broadcast process.

[0149] Step 4: Chip D constructs a return packet containing the global final result and sends it in reverse along the original cascaded path. After receiving the return packet, chip C writes the final result into its local address and then sends the return packet to chip B. After receiving the return packet, chip B writes the final result into its local address and then sends the return packet to chip A. After receiving the return packet, chip A writes the final result into its local address and notifies its DMA channel that the operation is complete.

[0150] Step 5: The DMA channel of chip A determines that the entire chain reduction task has been completed and sends a completion interrupt signal to the CPU through the interrupt line.

[0151] After the above steps, all chips have obtained the global accumulation result (acc). Throughout the process, the CPU only participates in the configuration step one; subsequent forward accumulation and backward broadcasting are autonomously completed by the DMA channels of each chip according to microcode instructions, without the CPU's sequential involvement. The CPU receives only one completion interrupt, independent of the number of cascaded chips, effectively avoiding interrupt storms and reducing software overhead.

[0152] Optionally, in this embodiment of the application, the step of dynamically adjusting the data transmission channel includes: Real-time monitoring of the utilization rate of each external bus interface and the backpressure status of the DMA channel to obtain statistical results; extraction of service quality priority tags from microcode instructions; based on the statistical results and service quality priority tags; when the current binding interface of a high-priority data stream is congested, redirecting the DMA channel of the data stream, and restoring the original binding relationship of the DMA channel after the preset validity period expires.

[0153] During implementation, the bandwidth monitoring subunit continuously samples the effective data transmission volume of each external bus interface at microsecond intervals, calculates the utilization percentage of each interface within the current sliding window, and monitors for backpressure signals from the peer in each DMA channel. These data constitute statistical results. The Quality of Service (QoS) tag identification subunit reads the priority value from the QoS field of the microcode command to be transmitted, determining whether the data stream is a high-latency sensitive stream. After receiving the above two pieces of information, the dynamic arbitration and binding adjustment subunit queries the binding table to confirm the external bus interface currently bound to the high-priority data stream. If the utilization of this interface exceeds a preset threshold, the arbitration subunit scans the statistical results of other external bus interfaces, selects the interface with the lowest utilization or one that is completely idle, temporarily redirects the DMA channel of the data stream to this idle interface, and starts a validity timer. Before the timer expires, all transmissions of the data stream are completed through the redirected interface. After the timer expires, the arbitration subunit clears the redirection rule, and the DMA channel is restored to the original interface bound by default, thus ensuring timely transmission of high-priority data streams while maintaining the determinism of the default binding.

[0154] The following example illustrates the working process of the dynamic bandwidth allocation unit for data flow awareness.

[0155] The following application scenario is set up: The system contains 3 DMA channels. Multiple Master read sub-units and multiple Master write sub-units all use a fully connected approach to connect to the RDMA and WDMA in each sub-channel. Specifically, DMA channel 1 has its first sub-channel (including RDMA0) and second sub-channel (including RDMA1) enabled; DMA channel 2 has its first sub-channel (including RDMA0); and DMA channel 3 has its first sub-channel (including RDMA0). In the following description, DMA channels 2 and 3 will be referred to directly as "DMA channel" to refer to their first sub-channels; DMA channel 1 will be described using RDMA0 and RDMA1 respectively. DMA channel 1 carries the All-Reduce reduction task with a QoS value of 15 (representing high latency sensitivity); DMA channel 2 carries the intermediate activation value transfer task with a QoS value of 8; and DMA channel 3 carries the log export task with a QoS value of 2 (representing low latency sensitivity). The theoretical bandwidth of the Master read channel is 16GB / s. Currently, DMA channel 3 is performing a large log export, occupying 12GB / s, DMA channel 2 is occupying 3GB / s, and the remaining available bandwidth of the Master is only 1GB / s. At this point, DMA channel 1 needs to perform an All-Reduce operation, requiring a burst bandwidth of 8GB / s.

[0156] Step 1: The bandwidth monitoring subunit continuously samples the utilization rate of each read Master port at microsecond intervals. In the current state, the bandwidth monitoring subunit reports that the utilization rate of read Master port 0 is nearing saturation and cannot support further data transmission.

[0157] Step 2: DMA channel 1 initiates an All-Reduce request. Both of its read data channels, RDMA0 and RDMA1, need to read data, and the QoS field value in the microcode instruction is 15. After receiving the request, the dynamic arbitration subunit queries the current binding table and finds that the read Master port 0, which is bound by default to RDMA0, is congested, while the read Master port 1, which is bound by default to RDMA1, is currently idle. The arbitration subunit sends RDMA1 directly to the idle read Master port 1; at the same time, it scans and finds that read Master port 2 is completely idle, so it temporarily redirects RDMA0 to read Master port 2 and sets the temporary mapping validity period to T microseconds.

[0158] Step 3: RDMA0 of DMA channel 1 should have used the read Master port 0, but the arbitration subunit modified its routing through hardware logic, sending the data from the read Master port 2. The two read data channels of the All-Reduce request completed the non-blocking transmission at a rate of 8GB / s on read Master port 1 and read Master port 2 respectively. Meanwhile, DMA channel 3 and DMA channel 2 continued to operate on the original read Master port 0, without affecting each other.

[0159] Step 4: After T microseconds, the temporary mapping expires, and the arbitration subunit automatically clears the redirection rule. The next transmission of RDMA0 in DMA channel 1 will resume using the default bound read Master port 0. If there is a new read request in RDMA0 of DMA channel 1 within the validity period, read Master port 2 will continue to be used; if read Master port 2 is also congested, the arbitration subunit can re-determine whether to redirect the data flow to another idle Master port or keep the current route unchanged.

[0160] As can be seen from the above examples, the data flow-aware dynamic bandwidth allocation unit, through the cooperation of three links—"real-time monitoring," "QoS identification," and "temporary redirection"—can flexibly adapt to sudden high-priority traffic without changing the determinism of static binding, effectively improving the problems of critical flow being blocked and idle bandwidth being wasted in traditional static arbitration.

[0161] In another scenario, suppose the receive buffer of the peer chip corresponding to the All-Reduce operation of DMA channel 1 is temporarily full, causing a write backpressure signal on the write Master port, meaning the peer chip continuously indicates that writing is not allowed. After the bandwidth monitoring subunit captures this backpressure signal, it notifies the dynamic arbitration subunit. If the arbitration subunit determines that the backpressure has persisted for more than N cycles, it proactively reduces the arbitration weight of DMA channel 1 on read Master port 2 from the default value of 16 to 4, transferring the released bandwidth quota to other channels using read Master port 2, thus effectively utilizing bus resources. When the backpressure signal disappears and normal operation resumes for M consecutive cycles, the arbitration subunit restores the arbitration weight of DMA channel 1 to the default value of 16, allowing it to regain its proper bus usage priority.

[0162] The steps for asynchronously releasing each computing core or chip based on pre-configured inter-core or inter-chip dependencies include: Configure the dependency mask for each computing core or chip, and initialize the wait counter for each core or chip according to the dependency mask; when any computing core or chip completes its local task, decrement the wait counters of other cores or chips that depend on the completed core or chip by one; immediately release the core or chip whose wait counter is zero, allowing it to continue executing subsequent computing tasks; when all dependencies corresponding to any synchronization point are satisfied, send a stage completion interrupt signal to the central processing unit.

[0163] During implementation, before the method begins execution, the CPU writes the dependency masks of each computing core or chip to the synchronization management unit through the configuration interface, describing the dependencies between cores. The synchronization management unit initializes the wait counters based on the number of dependencies of each core or chip. For example, if a core depends on two other cores, its wait counter is initially set to 2. During method execution, each computing core or chip sends a completion signal to the synchronization management unit upon completing its local task. After receiving the completion signal, the asynchronous synchronization and out-of-order release subunit queries the dependency mask, identifies all other cores that depend on the completed core, and decrements the wait counters of these cores by one. When the wait counter of a core reaches zero, all cores that the core depends on have completed their tasks, and the synchronization management unit immediately releases the core, allowing it to begin executing subsequent computational tasks. For scenarios with multiple synchronization points, each synchronization point independently maintains its own set of dependency masks and wait counters, and performs the above release judgment independently. When all dependencies corresponding to a synchronization point are satisfied, the synchronization signal merging and interrupt subunit sends a stage completion interrupt signal to the CPU, reporting that the computational tasks of that stage have all been completed.

[0164] In the implementation of the above embodiments: by real-time monitoring of the utilization and backpressure status of the external bus interface, combined with the service quality priority tag extracted from the microcode instructions, the DMA channel is redirected to an idle interface when high-priority data streams encounter congestion and automatically restored after the expiration period, thus achieving dynamic allocation of bus resources. Simultaneously, by configuring a dependency mask and initializing a wait counter, the counters of other cores dependent on that core are decremented when the core completes its local task, and the corresponding core is immediately released when the counter reaches zero. An interrupt is sent only when all dependencies at the synchronization point are satisfied, achieving fine-grained asynchronous synchronization. The combination of these two methods allows data transmission and synchronization operations to be completed autonomously by the hardware, reducing CPU intervention and improving overall system efficiency.

[0165] Please see Figure 5 The diagram illustrates the dependency relationships provided in the embodiments of this application.

[0166] The Hybrid Expert (MoE) model is used as an example to illustrate the asynchronous synchronization and out-of-order release mechanism of the synchronization management unit, aiming to improve the inefficiency of traditional barrier synchronization. In this example, a MoE layer has four experts deployed on chips A, B, C, and D. The output of chip A needs to be sent to chips B and C simultaneously, while chip D does not depend on any expert. The outputs of chips B and C need to be aggregated and sent to the next layer. For ease of explanation, it is assumed that the computation time of each chip follows the order: Time_A > Time_B > Time_C > Time_D, where chip A takes the longest time and chip D takes the shortest.

[0167] The first stage is the configuration phase, which occurs only once and is executed by the CPU.

[0168] Step 1: The CPU writes the dependency mask through the configuration interface. Chip A dependency mask = 0 (no dependency); Chip B dependency mask = [A] (depends on A); Chip C dependency mask = [A] (depends on A); Chip D dependency mask = 0 (no dependency); Next layer synchronization point dependency mask = [B, C] (must wait for B and C to complete, D does not participate).

[0169] Step 2: Enable multi-synchronization-point pipeline. The hardware automatically initializes the wait counter for each core. The wait counters for chips A and D are 0, the wait counters for chips B and C are initially 1 (each depends on one core), and the wait counter for the next synchronization point is initially 2 (it needs to wait for both cores B and C).

[0170] Then comes the running phase, which is automated by the hardware and involves no CPU.

[0171] Step 3: At time T=0, all chips begin calculating their respective expert tasks. Chips A and D have wait counters of 0 and begin execution immediately; chips B and C remain blocked while waiting for chip A.

[0172] Step 4: At time T = Time_D, chip D completes the calculation. Its wait counter is 0, and the synchronization management unit determines that there are no dependencies and immediately releases chip D. Chip D then begins executing the next layer's independent calculation task that does not depend on any other component.

[0173] Step 5: At time T=Time_A, chip A completes its calculation. The synchronization management unit detects that chip A has completed its calculation and, based on the dependency mask, decrements the wait counters of chips B and C, which depend on chip A (both change from 1 to 0). The counters returning to zero triggers a release, and chips B and C immediately begin their respective expert calculations.

[0174] Step 6: At time T = Time_A + Time_C, chip C completes its expert calculation. At this time, the current value of the wait counter of the next synchronization point is 2 (it needs to wait for B and C). After chip C arrives, it decrements the counter to 1 (it still needs to wait for chip B). Chip C enters a blocked state, waiting for chip B to complete.

[0175] Step 7: At time T = Time_A + Time_B, chip B completes its expert calculation. The wait counter of the next-layer synchronization point decreases from 1 to 0, the synchronization management unit immediately releases chip C and the next-layer calculation task, chip C is awakened, and the next layer begins calculation. The blocking time of chip C is only (Time_B - Time_C), that is, it only waited for chip B, which is slower than itself.

[0176] Step 8: In the subsequent pipeline, each chip proceeds naturally according to its dependency relationship, and the entire process does not require any software synchronization instructions.

[0177] Finally, there are the steps for linkage and interruption.

[0178] Step 9: During Step 6 to Step 7, chip C is in a blocked state while waiting for chip B. The synchronization management unit actively notifies the dynamic bandwidth allocation unit to reduce the priority of the DMA channel bandwidth of chip C and use the released bandwidth for data transmission of chip B or chip D.

[0179] Step 10: After chip B completes and chip C is woken up, the synchronization management unit notifies the dynamic bandwidth allocation unit again to restore the bandwidth priority of chip C.

[0180] Step 11: When all dependencies of the next layer synchronization point are satisfied (i.e., both chip B and chip C are completed), the synchronization signal merging and interrupt unit sends a completion interrupt to the CPU, reporting that the inference of this layer has ended and that the next layer can be entered or the result can be output.

[0181] Throughout the process, the CPU only participates in writing dependency masks and starting synchronization management during the configuration phase. During the runtime phase, there are no software synchronization instructions; it only receives a phase completion interrupt once in step 11. Chip D and chip A do not wait at all, while chip C only waits for the unavoidable minimum dependency wait time of (Time_B - Time_C). This invention effectively solves the inefficiency problem of traditional barrier synchronization forcing all cores to wait uniformly, while avoiding the waste of bandwidth resources through a blocking linkage mechanism.

[0182] A multi-chip cascaded system includes multiple chips, at least one of which includes the aforementioned inter-core and inter-chip interconnect devices. For example, the multi-chip cascaded system includes four chips A, B, C, and D, which form a cascaded topology through the aforementioned inter-core and inter-chip interconnect devices. Each of chips A, B, C, and D is equipped with the aforementioned inter-core and inter-chip interconnect devices, and these interconnect devices are sequentially connected via physical links between the chips, forming a cascaded structure of A to B, B to C, and C to D. Each chip holds local data gA, gB, gC, and gD respectively. The system needs to perform an All-Reduce addition operation to ensure that all chips ultimately obtain the global sum G = gA + gB + gC + gD.

[0183] In a multi-chip cascaded system, chip A, acting as the initiator of the chain protocol, has a microcode program pre-written by the CPU in its interconnect device's DMA channel. Chip A's DMA channel independently executes the microcode program, using local data gA as the initial accumulation value, encapsulating it along with the microcode instructions required by subsequent chips into a request packet, and sending it to the adjacent chip B. Upon receiving the request packet, chip B's DMA channel parses the microcode instructions attached to the packet, reads its local gB, performs an atomic addition operation with the accumulated value, generates a new request packet, and forwards it to chip C. Chip C performs the same operation and forwards it to chip D. Chip D, as the endpoint chip, completes the final accumulation to obtain the global result G, writes G to its local address, starts broadcasting, and returns step-by-step along the original path D→C→B→A. Each chip, upon receiving the return packet, writes G to its own local address, and chip A sends a completion interrupt signal to the CPU after completing the write.

[0184] During the above process, the dynamic bandwidth allocation unit in the interconnection device of each chip monitors the traffic status of the links between chips in real time, and dynamically adjusts the data transmission channel according to the QoS tag in the microcode instruction to ensure that high-priority data streams are not blocked. The synchronization management unit in the interconnection device of each chip maintains the waiting counter of each chip according to the pre-configured dependency mask. When chip A completes, it releases chips B and C, which depend on chip A. Chip D, because it has no dependency relationship, is immediately released after completion and continues to execute the next stage of calculation without waiting for other chips.

[0185] In this multi-chip cascaded system, the CPU only participates in microcode writing and synchronous dependency configuration during the initial configuration phase. Subsequent chained accumulation, dynamic bandwidth adjustment, and asynchronous release are all autonomously handled by the interconnecting hardware of each chip, without requiring sequential CPU intervention during operation. This system effectively reduces the frequency of CPU involvement and improves the overall efficiency of multi-chip collaborative processing. Please refer to [link to relevant documentation]. Figure 6The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0186] Figure 6 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a server or PC, or a virtual device, such as a virtual machine or virtualization container. Furthermore, electronic device 300 is not limited to a single device; it can be a combination of multiple devices or a cluster of numerous devices.

[0187] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0188] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0189] This application also provides a computer program product, including computer program instructions, which are executed by a processor to perform the method described above.

[0190] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0191] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0192] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An inter-core and inter-chip interconnect device, applied to a system comprising multiple computing cores or multiple chips, characterized in that, The interconnected devices include: The DMA channel array unit contains multiple DMA channels. Each DMA channel has a built-in microcode cache and microcode program counter for independently executing microcode programs pre-downloaded to the central processing unit to directly perform atomic operations and data forwarding on the data transport path. A data flow-aware dynamic bandwidth allocation unit is coupled between the DMA channel array unit and the external bus interface to monitor the traffic status of each DMA channel in real time and dynamically adjust the binding relationship between the DMA channel and the external bus interface according to the priority of the data flow. A synchronization management unit, coupled to each computing core or chip, is used to maintain the dependencies between cores or chips and control the release timing of each computing core or chip according to the dependencies. The DMA channel array unit, the dynamic bandwidth allocation unit, and the synchronization management unit are coupled to each other.

2. The interconnection device according to claim 1, characterized in that, Each DMA channel includes a first sub-channel and a second sub-channel, the first sub-channel and the second sub-channel being used for data interaction with chips on both sides in the cascaded direction, respectively; each sub-channel includes: Atomic operation decoder and dispatcher is used to parse microcode instructions, generate operation control signals and remote atomic operation lock management commands, and dispatch data streams; The remote atomic operation module is used to process atomic operation requests from other chips and to determine the atomicity of the access. The first and second read data channels are used to read data in parallel from on-chip or off-chip memory. Configurable arithmetic array for performing arithmetic operations on two input data streams; The write data channel is used to write out the calculation results or pass-through data.

3. The interconnection device according to claim 1, characterized in that, The microcode program contains an extensible set of microcode instructions, which include at least one of the following fields: The opcode field is used to specify the instruction type, which includes at least one of the following: transport instructions, arithmetic instructions, synchronization instructions, and routing instructions. The operation mode field is used to specify the type of atomic operation; The next-hop identifier field is used to specify the target of the chained forwarding; The chain end flag field is used to identify whether the current node is the chain end; The broadcast address field is used to specify the unified base address where each chip stores the final result, so that it can be written to the broadcast later. The Quality of Service (QoS) Priority Tag field is used by the dynamic bandwidth allocation unit to identify the latency sensitivity of the data stream.

4. The interconnection device according to claim 1, characterized in that, The dynamic bandwidth allocation unit includes: A bandwidth monitoring subunit is used to poll the utilization rate of each external bus interface and the backpressure status of the DMA channel at a preset period. A service quality tag identification subunit is used to extract service quality priority tags from the microcode instructions; The dynamic arbitration and binding adjustment subunit is used to redirect the DMA channel of the data stream with the required priority to an idle external bus interface based on the monitoring results of the bandwidth monitoring subunit and the identification results of the service quality label identification subunit, and restore the original binding relationship of the DMA channel after the preset validity period expires. When the bandwidth monitoring subunit detects a write reverse pressure signal from the peer chip, the dynamic arbitration and binding adjustment subunit reduces the arbitration weight of the corresponding DMA channel on the current external bus interface, and restores the arbitration weight after the reverse pressure signal disappears.

5. The interconnection device according to claim 1, characterized in that, The synchronization management unit includes: The asynchronous synchronization and out-of-order release subunit is used to maintain the dependency mask configured by software and the wait counter of each computing core or chip. When the computing core or chip completes its local task, the wait counter of other cores or chips that depend on the completed core or chip is decremented by one, and the corresponding computing core or chip is released when the wait counter reaches zero, without waiting for other undependent cores or chips. The synchronization signal merging and interrupt subunit is used to collect the completion signals of each core or chip and send an interrupt signal to the central processing unit when the preset synchronization conditions are met.

6. The interconnection device according to claim 5, characterized in that, The synchronization management unit supports multiple synchronization points being active simultaneously. Each synchronization point independently maintains a set of dependency masks and wait counters for parallel synchronization.

7. The interconnection device according to claim 5, characterized in that, When the asynchronous synchronization and out-of-order release subunit detects that any computing core or chip has entered a blocked state due to unmet dependencies, it actively notifies the dynamic bandwidth allocation unit to reduce the bandwidth quota of the blocked core or chip.

8. A method for interconnecting cores and chips, applied to a system comprising multiple computing cores or multiple chips, characterized in that, The method includes: The central processing unit initiates the DMA channel of the initiating chip, executes the microcode program in the microcode cache through the DMA channel of the initiating chip, and encapsulates local data and microcode instructions into a request packet and sends it to the adjacent chip; wherein, the microcode program is written by the central processing unit to the microcode cache of the DMA channel of the initiating chip; The adjacent chip receives the request packet, parses the attached microcode instructions, performs atomic operations on the local data and the intermediate data in the request packet to generate a new request packet and forwards it until the destination chip. After the endpoint chip completes the calculation, it returns a return packet containing the final result along the original path, and each chip writes the final result to its local address. After completing the final result writing, the initiating chip sends a completion interrupt signal to the central processing unit. The system monitors the data stream's flow status and priority in real time to dynamically adjust the data transmission channel, and asynchronously releases each computing core or chip according to pre-configured inter-core or inter-chip dependencies.

9. The method according to claim 8, characterized in that, The steps for dynamically adjusting the data transmission channel include: The utilization rate of each external bus interface and the backpressure status of the DMA channel are monitored in real time to obtain statistical results; the quality of service priority tag is extracted from the microcode instruction; based on the statistical results and the quality of service priority tag, when the current binding interface of the high priority data stream is in a congested state, the DMA channel of the data stream is redirected, and the original binding relationship of the DMA channel is restored after the preset validity period expires. The step of asynchronously releasing each computing core or chip according to the pre-configured inter-core or inter-chip dependencies includes: Configure the dependency mask for each computing core or chip, and initialize the wait counter for each core or chip according to the dependency mask; when any computing core or chip completes its local task, decrement the wait counters of other cores or chips that depend on the completed core or chip by one; immediately release the core or chip whose wait counter is zero, allowing it to continue executing subsequent computing tasks; when all dependencies corresponding to any synchronization point are satisfied, send a stage completion interrupt signal to the central processing unit.

10. A multi-chip cascade system, characterized in that, The multi-chip cascade system includes multiple chips, and at least one of the multiple chips includes the inter-core and inter-chip interconnect devices as described in any one of claims 1 to 7.

Images