Multi-core integrated chip, computing storage core particle, distributed routing method and electronic device

Abstract

The application provides a multi-core particle integrated chip, a computing storage particle, a distributed routing method and an electronic device. The multi-core particle integrated chip comprises at least one particle array, the particle array comprises a plurality of computing storage particles, the computing storage particle comprises a computing die and a dynamic random access memory vertically stacked with the computing die; the computing die comprises a distributed router and a plurality of inter-particle interconnection ports arranged around the computing die; the plurality of inter-particle interconnection ports comprise intra-group neighbor ports and boundary expansion ports, the intra-group neighbor ports are used for connecting the computing storage particles inside the particle array, and the boundary expansion ports are used for connecting newly-added computing storage particles outside the particle array; the distributed router is connected with the intra-group neighbor ports and the boundary expansion ports respectively, is used for determining a next-hop port in the intra-group neighbor ports and the boundary expansion ports, and forwards a communication message through the next-hop port. Through the application, flexible expansion of a large-scale particle array can be realized.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and in particular to a multi-chip integrated circuit, a computing and storage chip, a distributed routing method, and an electronic device. Background Technology

[0002] With the increasing demands for AI training and inference, higher requirements are being placed on the computing power density and storage bandwidth of computing chips. Limited by wafer manufacturing yield and package size, it is difficult to continuously improve the performance of a single large-area computing die. Chiplet integration technology, by packaging and interconnecting multiple independent chips, has become an effective way to improve overall computing power and storage capacity.

[0003] However, multi-chip architectures primarily rely on a central switching node or a fully interconnected network. As the number of chips increases, architectures relying on a central switching node become physically limited by the shoreline width and port number of the switching components; fully interconnected topologies require each chip to establish point-to-point direct connections with all other chips, resulting in a quadratic increase in the number of packages and wiring. These interconnect architectures limit the ability of chip arrays to scale smoothly to a large scale, leading to problems such as poor scalability, high package and wiring complexity, and excessive interconnect resource consumption when scaling up the array. Summary of the Invention

[0004] This application provides a multi-chip integrated circuit, a computing and storage chip, a distributed routing method, and an electronic device, which can achieve flexible expansion of large-scale chip arrays without the need for a central switching node by introducing a distributed two-level port routing mechanism in vertically stacked computing and storage chips.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a multi-chip integrated circuit, including at least one chip array, the chip array including multiple computing and storage chips, each computing and storage chip including a computing die and at least one dynamic random access memory stacked perpendicularly to the computing die; The computing die includes a distributed router and multiple inter-die interconnect ports disposed around the computing die; The multiple interconnect ports between the cores include intra-group nearest neighbor ports and boundary extension ports. The intra-group nearest neighbor ports are used to connect computing and storage cores inside the core array, and the boundary extension ports are used to connect newly added computing and storage cores outside the core array. The distributed router connects to the nearest neighbor port within the group and the boundary extension port respectively. The distributed router is used to determine the next-hop port among the nearest neighbor port within the group and the boundary extension port, and to forward communication packets through the next-hop port.

[0006] This application provides a computing memory die, which includes a computing die and at least one dynamic random access memory stacked perpendicularly to the computing die; The computing die includes a distributed router and multiple inter-die interconnect ports disposed around the computing die; The computing storage core is used to send a probe signal through the interconnect ports between the multiple cores during initialization, and to generate a valid port bitmap based on the feedback result of the probe signal; The compute storage kernel is used to dynamically divide the interconnect ports between the multiple kernels into intra-group neighbor ports and boundary extension ports based on the effective port bitmap. The intra-group neighbor ports are used to communicate with other compute storage kernels belonging to the same kernel array, and the boundary extension ports are used to connect compute storage kernels outside the kernel array across the array. The distributed router is used to determine the next-hop port among the nearest ports within the group and the boundary extension ports, and to forward communication packets through the next-hop port.

[0007] This application provides a distributed routing method applied to a multi-chip integrated circuit. The multi-chip integrated circuit includes at least one chip array, the chip array including multiple computing and storage chips, each computing and storage chip including a distributed router and multiple inter-chip interconnect ports disposed around the computing die of the computing and storage chip. The method includes: The detection signal is transmitted via the computing storage core through the interconnect ports between the multiple cores; The effective port bitmap is generated based on the feedback result of the probe signal through the computational storage core. The computational storage core is used to divide the interconnect ports between the multiple cores into intra-group neighbor ports for connecting the core array and boundary extension ports for connecting the core array to the outside of the core array based on the effective port bitmap. The next-hop port is determined from the nearest neighbor ports within the group and the boundary extension ports using the distributed router. The distributed router forwards communication messages via the next-hop port.

[0008] In the above scheme, determining the next-hop port among the nearest neighbor ports within the group and the boundary extension ports via the distributed router includes: Obtain the target coordinates of the target computing storage core pointed to by the communication message, and the local coordinates of the computing storage core where the distributed router is located; By comparing the horizontal dimension coordinates of the target coordinates with the horizontal dimension coordinates of the local coordinates, a horizontal comparison result is obtained. In cases where the lateral comparison results indicate inconsistent lateral dimension coordinates, the next-hop port is determined from the nearest neighbor port within the group or the boundary extension port in the lateral interconnection direction to reduce the lateral coordinate difference. If the horizontal comparison results indicate that the horizontal dimension coordinates are consistent but the vertical dimension coordinates are inconsistent, the next-hop port is determined from the nearest neighbor port within the group or the boundary extension port in the vertical interconnection direction.

[0009] In the above scheme, determining the next-hop port among the nearest neighbor ports within the group and the boundary extension ports via the distributed router includes: Obtain multiple alternative forwarding paths configured in the distributed router to reach the target computing and storage core, wherein each alternative forwarding path corresponds to a different interconnection port between cores; Obtain the link credit or transmission queue depth of the inter-core interconnect port corresponding to the multiple alternative forwarding paths; Based on the link credit or the congestion status represented by the sending queue depth, the next-hop port is determined from the inter-core interconnect ports corresponding to the multiple alternative forwarding paths.

[0010] In the above scheme, the communication message includes a priority field, and the forwarding of the communication message through the distributed router via the next-hop port includes: Parse the priority field in the communication message to determine the priority level of the communication message; When the priority level is first priority, an independent virtual channel is allocated to the communication message; wherein the communication message carrying key-value cache data or gradient synchronization data is configured to the first priority. The distributed router, based on the independent virtual channel, prioritizes forwarding the communication packets via the next-hop port.

[0011] In the above scheme, after dividing the plurality of inter-chip interconnect ports into intra-group nearest neighbor ports for connecting the inside of the chip array and boundary extension ports for connecting the outside of the chip array based on the effective port bitmap, the method further includes: Perform termination detection on the boundary extension port that is in an unconnected state; When the termination detection fails, the drive circuit of the boundary extension port in the unconnected state is shut down to perform a shielding operation.

[0012] This application provides an electronic device, which includes a processor, wherein the processor includes a multi-chip integrated chip or a computing storage chip provided in this application.

[0013] The embodiments of this application have the following beneficial effects: By vertically stacking computing dies with at least one dynamic random access memory (DRAM) to form computing-storage granules, efficient integration of computing and storage is achieved, improving the data interaction efficiency within the computing-storage granules. Distributed routers and multiple inter-granule interconnect ports are incorporated into the computing dies. These interconnect ports are divided into intra-group nearest neighbor ports for connecting computing-storage granules within the granule array, and boundary extension ports for connecting newly added computing-storage granules outside the granule array. The distributed router determines the next-hop port from the intra-group nearest neighbor ports and the boundary extension ports, and forwards communication packets through this next-hop port. This allows the routing and forwarding of communication packets to be directly completed by each computing-storage granule using its own distributed router. When increasing the number of computing-storage granules, new granules can be directly connected via the boundary extension ports, thereby improving the flexibility of expanding the array size of multi-granule integrated chips and enhancing the interconnect expansion capability of the granule array. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of the multi-chip integrated chip provided in the embodiments of this application; Figure 2 This is a schematic cross-sectional view of the computing storage chip provided in an embodiment of this application; Figure 3 This is a schematic diagram of the planar topology of the chip array provided in the embodiments of this application; Figure 4 This is a schematic diagram of an eight-chip array structure of a multi-chip integrated chip provided in this application embodiment; Figure 5 This is a schematic diagram of a sixteen-chip array structure of a multi-chip integrated chip provided in this application embodiment; Figure 6 This is a flowchart illustrating the distributed routing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of local and remote DRAM access paths under a unified address space provided in the embodiments of this application; Figure 8 This is a schematic diagram of the data sharding and execution process for AI training and inference provided in the embodiments of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0018] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0019] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0020] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0021] 1) Multi-chip integrated circuit: refers to a single chip system formed by integrating multiple chips with independent or identical functions through advanced packaging technology. In the embodiments of this application, it refers to a scalable chip that includes at least one chip array and achieves communication interconnection by relying on the distributed routing mechanism within the chip.

[0022] 2) Chip array: refers to a collection of multiple chips arranged and spliced ​​according to a specific two-dimensional planar physical topology (such as a two-row, two-column arrangement). In the embodiments of this application, the computing and storage chips inside the chip array are interconnected to form basic units, and can be extended outward to connect to other chip arrays or individual chips through boundary expansion.

[0023] 3) Computation and storage chip: refers to a small integrated circuit entity with independent functions, integrated with other chips through packaging interconnect technology. In the embodiments of this application, it refers to a physical entity formed by vertically stacking and packaging a logic computing die with at least one dynamic random access memory, which has both computing processing and local storage functions.

[0024] 4) Computation die: refers to an unpackaged integrated circuit logic die cut from a semiconductor wafer. In the embodiments of this application, it refers to a basic die containing logic circuits such as a computing engine, distributed router, and interconnection interfaces, used to execute core computing instructions and control data routing and forwarding.

[0025] 5) Three-Dimensional Dynamic Random Access Memory (3D DRAM): refers to high-density random access memory manufactured using three-dimensional stacking technology. In the embodiments of this application, it is vertically stacked with the computing die (e.g., through hybrid bonding) to provide the computing die with high-bandwidth, low-latency local storage space.

[0026] 6) Distributed Router: Refers to a decentralized network routing control hardware module. In the embodiments of this application, the distributed router is independently deployed inside the computing die of each computing storage chip, and is used to independently resolve coordinates, select the next-hop port, and control the forwarding of communication packets, without relying on a global central switching node.

[0027] 7) Die-to-Die (D2D) Port: This refers to the physical communication interface used to enable high-speed data and control signal transmission between different dies. In this embodiment, it is located around the computing die, providing lateral and longitudinal physical connection channels between dies.

[0028] 8) Intra-group neighbor port: refers to a state type in which interconnection ports between chips are logically divided. In the embodiments of this application, it refers to a port facing the inside of the chip array, specifically used to realize the interconnection and communication message forwarding between multiple computing and storage chips belonging to the same chip array.

[0029] 9) Boundary Extension Port: This refers to another logically defined state type of the interconnect port between chips. In the embodiments of this application, it refers to a reserved port facing outwards from the chip array, used to connect to other chip arrays across arrays, or to connect to newly added computing and storage chips outside the chip array. It is a key interface for achieving smooth scaling of multi-chip integrated circuits.

[0030] 10) Communication message: refers to a standardized data unit transmitted between interconnected network nodes. In the embodiments of this application, it refers to a data packet forwarded between ports at various levels via a distributed router, which may encapsulate information including the target computing storage core identifier, global physical address, priority, and data content (such as tensor data, synchronization messages, etc.).

[0031] 11) Global Physical Address Space: This refers to a contiguous address space formed by uniformly connecting storage regions distributed across multiple discrete computing storage chips at the hardware addressing level. This allows upper-layer software to automatically route and address requests to specific target physical addresses within this space, shielding the underlying hardware from the physical transport process across chips.

[0032] 12) Computing Engine: Refers to the hardware logic unit within the computing die specifically used for performing data operations and processing. In the embodiments of this application, it refers to a processing module that performs tasks such as artificial intelligence matrix operations and tensor operations, and is capable of generating storage access requests containing target physical addresses in the global physical address space.

[0033] In the evolution of chip architecture, the increasing demands for computing power and storage bandwidth from large-scale AI model training and inference have made it difficult for single large-area computing dies to achieve linear growth in computing power and storage bandwidth due to limitations in wafer yield, photomask constraints, and package edge length. To address this, the industry has introduced multi-chip integration technology. However, analysis of related architectures has revealed the following design flaws: First, when multi-chip systems interconnect internally, they typically rely on a central switching die or a fully interconnected topology. As the number of chips increases, the central switching component faces physical bottlenecks in terms of port count, shoreline width, and switching bandwidth. A fully interconnected topology causes the physical wiring inside the package to grow exponentially, increasing the difficulty of package routing and making it difficult for the system to scale smoothly to larger scales (such as 8 chips, 16 chips, and above).

[0034] Secondly, in terms of storage access mechanisms in multi-core systems, pure software partitioning schemes increase the complexity of programming and compilation; if a forced system-wide hardware cache consistency is adopted, it is necessary to maintain complex directory states and generate invalid consistency broadcast traffic, thereby occupying the inter-core communication bandwidth.

[0035] Therefore, how to achieve flexible expansion and interconnection memory access of large-scale chip arrays without increasing the central switching node and introducing global cache coherency overhead is a difficult technical challenge to overcome in related technologies.

[0036] To address the aforementioned technical problems, embodiments of this application provide a multi-chip integrated circuit, a computing and storage chip, a distributed routing method, and an electronic device, which can solve the technical problems of limited chip array expansion due to reliance on a central switching node and high communication overhead across chip interconnects.

[0037] Figure 1 This is a schematic diagram of the structure of the multi-chip integrated chip provided in an embodiment of this application. See also... Figure 1 The multi-chip integrated circuit 10 includes at least one chip array 20, the chip array 20 includes a plurality of compute storage chips 100, each compute storage chip 100 includes a compute die and at least one dynamic random access memory (DRAM) stacked perpendicularly to the compute die. Figure 1 (Not shown in the image).

[0038] The computing die includes a distributed router and multiple inter-die interconnect ports 120 disposed around the computing die. The multiple inter-die interconnect ports 120 include intra-group nearest neighbor ports and boundary extension ports. The intra-group nearest neighbor ports are used to connect computing storage dies 100 within the die array 20, and the boundary extension ports are used to connect newly added computing storage dies 110 outside the die array 20. The distributed router connects to the intra-group nearest neighbor ports and the boundary extension ports respectively. The distributed router is used to determine the next-hop port among the intra-group nearest neighbor ports and the boundary extension ports, and to forward communication packets through the next-hop port.

[0039] For a detailed explanation of the hardware architecture of the computing storage chip 100 and the distribution of its two-level ports, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic cross-sectional view of the computing storage chip provided in an embodiment of this application. For example... Figure 2 As shown, each computing storage die 100 includes a computing die 130 and at least one dynamic random access memory 140 stacked vertically to the computing die 130. The computing die 130 includes a distributed router (…). Figure 2 The compute die 130 is integrated within the internal logic layer (not separately labeled) and has multiple inter-die interconnect ports 120 located around its physical sides. These inter-die interconnect ports 120 are specifically hierarchically categorized into intra-group nearest neighbor ports 121 and boundary extension ports 122. A high-density vertical signal channel 150 is formed between the compute die 130 and the underlying first-level dynamic random access memory 140 via a hybrid bonding process. Optionally, the compute die 130 is fixed to a package substrate 160. The specific operation of the compute memory die 100 is described below.

[0040] Here, a computing die refers to the basic semiconductor chip that houses logic computing circuits, control buses, and network routing modules. Dynamic random access memory (DRAM) refers to a semiconductor memory device manufactured using three-dimensional processes that provides high-density data storage space. A vertical signal path refers to a conductive path that penetrates the die-to-die interface in the vertical direction and directly transmits high-frequency electrical signals.

[0041] In some embodiments, a computing die 130 is fixed above a packaging substrate 160. After fixing the computing die 130, at least one layer of dynamic random access memory 140 is forward-stacked and aligned with the top surface of the computing die 130 in the height direction. After stacking and pressing, a short-distance, high-bandwidth local vertical memory loop is established between the computing die 130 and the dynamic random access memory 140 through a vertical signal channel 150. Simultaneously, two levels of physical interfaces are arranged on the four sides of the computing die 130. The interface connected to the adjacent side inside the current chip array 20 is configured as an intra-group nearest neighbor port 121, and the interface reserved on the outward suspended side is configured as a boundary extension port 122.

[0042] In some embodiments, see Figure 3 , Figure 3 This is a schematic diagram of the planar topology of the chip array provided in this application embodiment. The chip array 20 includes four computing and storage chips 100 arranged in two rows and two columns. The four computing and storage chips 100 are respectively identified as the first computing and storage chip C0, the second computing and storage chip C1, the third computing and storage chip C2, and the fourth computing and storage chip C3. Figure 3 Solid double-headed arrows indicate intra-group communication links established by neighboring ports within the group. Dashed double-headed arrows in the figure indicate reserved boundary expansion ports. The following describes the internal physical arrangement of the chip array 20.

[0043] Here, a two-row, two-column arrangement refers to a physical layout structure in a two-dimensional planar space, fixed in a crisscrossing matrix form. An intra-group communication link refers to a physical transmission channel within the same core array 20 used for data exchange between adjacent nodes.

[0044] In some embodiments, the first compute storage chip C0 is placed in the upper left corner. The second compute storage chip C1 is placed in the upper right corner. The third compute storage chip C2 is placed in the lower left corner. The fourth compute storage chip C3 is placed in the lower right corner. After placement, the first compute storage chip C0 and the second compute storage chip C1 are laterally connected using the nearest neighbor ports within the group corresponding to the solid bidirectional arrows. The first compute storage chip C0 and the third compute storage chip C2 are vertically connected using the nearest neighbor ports within the group corresponding to the solid bidirectional arrows. After connection, the boundary extension ports corresponding to the dashed bidirectional arrows are retained in the west and north directions of the first compute storage chip C0.

[0045] In some embodiments, for the array splicing expansion of the chip array 20, external module docking can be performed using the boundary expansion ports corresponding to the dashed bidirectional arrows. The control hardware detection circuit reads the current level connection status of the boundary expansion port. After reading the level connection status, in the horizontal direction, the east-facing boundary expansion port of the second computing storage chip C1 is used to dock with another first newly added chip array with the same structure, forming an eight-chip array. After forming the eight-chip array, in the vertical direction, the south-facing boundary expansion ports of the third computing storage chip C2 and the fourth computing storage chip C3 are used to dock with two other second newly added chip arrays with the same structure, extending and splicing the eight-chip array longitudinally to form a sixteen-chip array.

[0046] For example, four compute storage cores 100 are arranged in a two-row, two-column configuration as C0, C1, C2, and C3. The first compute storage core C0 in the upper left corner is assigned an x-coordinate of 0 and a y-coordinate of 1. The fourth compute storage core C3 in the lower right corner is assigned an x-coordinate of 1 and a y-coordinate of 0. The first compute storage core C0 sends a 256-byte data packet to the second compute storage core C1 via the nearest neighbor port within the group corresponding to the horizontal solid bidirectional arrow. The first compute storage core C0 reserves a separate boundary extension port in the westward direction, corresponding to the dashed bidirectional arrow, for receiving external probe signals.

[0047] This application embodiment designs the chip array as containing four computing and storage chips arranged in a two-row, two-column configuration. Intra-group communication links are established at the internal adjacent edges using solid bidirectional arrows (represented by port numbers), while boundary extension ports (represented by dashed bidirectional arrows) reserve physical expansion interfaces at the external suspended edges. This scheme, through a basic matrix layout combined with clear port status division, ensures at the physical level that the computing and storage chips can interact with adjacent nodes over the shortest distance, avoiding long-distance cabling congestion caused by fully interconnected topologies. The reserved boundary extension ports at the external edges provide a physical basis for modular splicing, solving the technical problems of high packaging and wiring complexity and the difficulty in smoothly scaling the chip interconnect network to a large scale in related technologies.

[0048] In some embodiments, the multi-chip integrated circuit (MCC) sets up a unified global physical address space. The compute die also includes a compute engine, a local address decoder, a local memory controller, and a remote memory access agent. The compute engine is used to generate a memory access request, wherein the memory access request includes a target physical address in the global physical address space. The local address decoder is used to parse the target physical address in the memory access request to obtain a target chip identifier, and compare the target chip identifier with the local chip identifier of the compute die itself to obtain a comparison result. The local address decoder is also used to distribute the memory access request to the local memory controller or the remote memory access agent based on the comparison result.

[0049] Here, the global physical address space refers to the contiguous address range formed by concatenating storage resources distributed in different physical locations at the hardware addressing level. The computing engine refers to the hardware execution unit responsible for executing data operation instructions and actively initiating data read / write tasks. The local address decoder refers to the hardware logic circuitry used for bit-field parsing and physical attribution determination of address data. The local memory controller refers to the control module responsible for translating read / write instructions into the underlying physical operation timing for dynamic random access memory. The remote memory access agent refers to the hardware agent module responsible for encapsulating cross-chip memory access operations into a network communication format and coordinating the transmission protocol. A storage access request refers to a hardware instruction packet requesting the reading or writing of data at a specific address. The target physical address refers to the absolute physical address pointing to the specific data storage location in the global physical address space. The target chip identifier refers to the hardware identification number hidden in the target physical address and used to indicate the actual physical attribution of the target physical address. The local chip identifier refers to the unique hardware identification number bound to the current computing die itself. The comparison result refers to the Boolean status value output by the local address decoder after logically comparing the target chip identifier with the local chip identifier.

[0050] In some embodiments, a unified global physical address space is pre-configured in the multi-chip integrated circuit. After configuring the global physical address space, the computing engine generates a storage access request when performing data processing tasks. After generating the storage access request, the computing engine sends the storage access request to the local address decoder. The local address decoder receives the storage access request and extracts the target physical address contained within it. After extracting the target physical address, the local address decoder parses the target physical address. After parsing, a specific address bit field is truncated as the target chip identifier. After truncating the target chip identifier, the local address decoder reads the internal hardware registers to obtain the local chip identifier. After obtaining the local chip identifier, the local address decoder inputs the target chip identifier and the local chip identifier into a comparator circuit for comparison. After comparison, the comparison result is output. After outputting the comparison result, the local address decoder distributes the storage access request to the local storage controller or remote memory access agent according to the logical state of the comparison result.

[0051] In some embodiments, the process of the local address decoder distributing storage access requests based on the comparison results can be implemented using a multi-channel concurrent distribution mechanism. The local address decoder reads the logical state of the comparison results. When the logical state is a match indicating that the target kernel identifier and the local kernel identifier are consistent, the internal local direct channel is activated. After activating the local direct channel, the local address decoder directly pushes the storage access request to the buffer queue of the local storage controller through the local direct channel. When the logical state is a mismatch indicating that the target kernel identifier and the local kernel identifier are inconsistent, the internal cross-chip proxy channel is activated. After activating the cross-chip proxy channel, the local address decoder forwards the storage access request to the remote storage access proxy through the cross-chip proxy channel.

[0052] For example, a multi-chip integrated circuit has a global physical address space with a total capacity of 32 gigabytes. The address range of the global physical address space is from hexadecimal 0x00000000 to 0x7FFFFFFF. The computing engine generates a memory access request when performing tensor operations. The memory access request includes a target physical address of 0x40001000. The local address decoder parses the high 4 bits of the target physical address 0x40001000. After parsing, the target chip identifier is obtained as hexadecimal 4. The local address decoder reads the local hardware register to obtain the local chip identifier as hexadecimal 2. The local address decoder compares the target chip identifier 4 with the local chip identifier 2. A mismatch is found. Based on the mismatch, the local address decoder distributes the memory access request to a remote memory access agent.

[0053] This application's embodiments establish a unified global physical address space within a multi-chip integrated circuit. A computing engine generates storage access requests containing the target physical address. A local address decoder parses the target chip identifier and compares it with the local chip identifier. Based on the comparison result, the storage access request is distributed to the local storage controller or a remote memory access agent. This scheme, by constructing a continuous and unified global physical address space at the hardware level, shields the upper-layer software from the physical separation characteristics of multiple storage nodes, reducing the complexity of cross-chip programming. Hardware decoding enables the routing and scheduling of inconsistent storage access, allowing local requests to directly access high-bandwidth vertical channels, while remote requests are handled by remote memory access agents. This eliminates the overhead of maintaining system-wide hardware cache consistency, solving the technical problems of complex cross-chip storage access software programming and wasted communication bandwidth due to maintaining global cache consistency in related technologies.

[0054] In some embodiments, the local address decoder is used to extract the kernel identifier field from the target physical address to obtain the target kernel identifier; or, the local address decoder is used to query the programmable region table to obtain the target kernel identifier mapped to the target physical address, wherein the programmable region table is used to map a continuous global physical address space to the dynamic random access memory of the computing storage kernel.

[0055] Here, the chip identifier field refers to the set of bits in the target physical address that are pre-divided according to a specific address width and are specifically used to record the hardware identity number of the computing storage chip. The programmable region table refers to a hardware lookup table logical structure deployed within the local address decoder that supports dynamically configuring the correspondence between address ranges and hardware identity numbers. Mapping refers to transforming an abstract, continuous global physical address space into a correspondence between physical locations within a specific computing storage chip.

[0056] In some embodiments, the local address decoder receives a target physical address. Upon receiving the target physical address, the local address decoder identifies the fixed-width configuration parameters of the target physical address. The local address decoder performs a shift and truncation operation on the target physical address according to the fixed-width configuration parameters. After the shift and truncation operation, the local address decoder extracts the high-order bits of the kernel identifier field from the target physical address. The local address decoder determines the value of the kernel identifier field as the target kernel identifier.

[0057] In some embodiments, for scenarios with irregular address distributions, a hardware lookup table mechanism can be used to obtain the target chip identifier. The local address decoder obtains the target physical address. After obtaining the target physical address, the local address decoder inputs the target physical address into a programmable area table for interval matching. After interval matching, the local address decoder retrieves a contiguous global physical address space containing the target physical address from the programmable area table. The local address decoder reads the hardware number record associated with this contiguous global physical address space. The local address decoder outputs the hardware number record and obtains the target chip identifier mapped to the target physical address.

[0058] For example, the local address decoder receives a target physical address of hexadecimal 0x12345678. When using the method of extracting the kernel identifier field, the local address decoder extracts the high 8 bits of the target physical address 0x12345678. After extracting the high 8 bits, the kernel identifier field is obtained as hexadecimal 12. The local address decoder determines the hexadecimal 12 as the target kernel identifier. When using the method of querying the programmable area table, the programmable area table records that the address range of hexadecimal 0x10000000 to 0x1FFFFFFF maps to the compute memory kernel numbered 3. The local address decoder inputs the target physical address hexadecimal 0x12345678 into the programmable area table for comparison. After comparison, it is determined that the target physical address falls within the range of hexadecimal 0x10000000 to 0x1FFFFFFF. The local address decoder obtains the target kernel identifier 3 mapped to the target physical address by querying the programmable area table.

[0059] This application's embodiments extract the target kernel identifier field from the target physical address using a local address decoder, or query a programmable region table using a local address decoder to obtain the target kernel identifier mapped to the target physical address. The above scheme provides a fast and direct address resolution path through hardware bit-field truncation operations, ensuring low-latency address decoding. Simultaneously, by introducing a programmable region table for continuous global physical address space mapping configuration, it provides flexible non-contiguous memory allocation management capabilities. Utilizing a dual resolution mechanism combining hard-wire truncation and dynamic table lookup improves the flexibility and resolution efficiency of global physical address space addressing and routing, solving the technical problems of rigid cross-kernel storage physical address mapping and inflexible support for complex memory partition scheduling in related technologies.

[0060] In some embodiments, the local storage controller is configured to receive a storage access request when the comparison result indicates that the target kernel identifier matches the local kernel identifier; the local storage controller is configured to extract the kernel internal storage address from the target physical address contained in the received storage access request; the local storage controller is configured to parse the kernel internal storage address to obtain the target storage channel, target memory bank, target row address, and target column address; the local storage controller is configured to perform a data read operation or a data write operation in at least one dynamic random access memory via a vertical storage interface at the physical location corresponding to the target storage channel, target memory bank, target row address, and target column address.

[0061] Here, the internal storage address of a compute memory chip refers to a local physical address specifically used for addressing within the storage space of a single compute memory chip after removing its identity identifier. The target storage channel refers to an independent concurrent data transfer bus path within the dynamic random access memory (DRAM). The target memory block refers to a two-dimensional storage array block within the DRAM capable of independently performing activation and precharge operations. The target row address refers to the row coordinate index used to locate a specific word line within the two-dimensional storage array block. The target column address refers to the column coordinate index used to locate a specific word line within the two-dimensional storage array block. The vertical storage interface refers to the physical channel for data and instruction transmission in the vertical direction, composed of hybrid bonding nodes. Data read or write operations refer to the low-level physical execution process by which hardware extracts charge data from or injects charge data into the physical storage medium.

[0062] In some embodiments, the local storage controller reads the comparison result. When the comparison result indicates that the target kernel identifier matches the local kernel identifier, the local storage controller receives a storage access request. After receiving the storage access request, the local storage controller reads the target physical address carried in the storage access request. The local storage controller performs a masking operation on the target physical address. After the masking operation, the local storage controller extracts the lower-order internal storage address of the kernel. The local storage controller inputs the internal storage address of the kernel into the address mapping logic circuit for parsing. After parsing, the target storage channel, target memory bank, target row address, and target column address are separated. After separation, the local storage controller converts the above parameters into memory access timing instructions. The local storage controller sends the memory access timing instructions to the vertical storage interface. The local storage controller drives at least one dynamic random access memory (DRAM) via the vertical storage interface. In the at least one DRAM, the physical location corresponding to the target storage channel, target memory bank, target row address, and target column address is addressed. Data read operations or data write operations are performed at the physical location.

[0063] In some embodiments, an interleaving mapping mechanism can be used to optimize concurrent memory access efficiency during the process of the local storage controller resolving the internal storage address of the memory chip. The local storage controller obtains the internal storage address of the memory chip. After obtaining the internal storage address, the local storage controller performs an XOR hash operation on a portion of the bits of the internal storage address. The XOR hash operation scrambles the low-order continuity of the internal storage address to generate the target storage channel and target memory bank. After generating the target storage channel and target memory bank, the local storage controller allocates the remaining high-order bits of the internal storage address as the target row address. The local storage controller allocates the least significant byte offset of the internal storage address as the target column address. Based on the allocation result of the scrambled continuity, the local storage controller distributes consecutive storage access requests to different target storage channels and target memory banks for concurrent execution.

[0064] For example, the local storage controller reads the matching comparison result and receives a storage access request containing the target physical address in hexadecimal format 0x2000ABCD. The local storage controller extracts the lower 24 bits to obtain the core internal storage address in hexadecimal format 0x0000ABCD. The local storage controller converts the hexadecimal 0x0000ABCD into a binary sequence. The local storage controller extracts the highest 2 bits of the binary sequence to obtain the target storage channel as 1. It extracts the next 3 bits to obtain the target memory bank as 4. It extracts the middle 14 bits to obtain the target row address as 85. It extracts the last 5 bits to obtain the target column address as 13. The local storage controller performs a data read operation at the physical location corresponding to row 85 and column 13 of the target storage channel numbered 1 and the target memory bank numbered 4 via the vertical storage interface, reading 32 bytes of tensor data.

[0065] This application embodiment receives storage access requests and extracts the internal storage address of the chip during comparison and matching via a local storage controller. The internal storage address is then parsed to separate the target storage channel, target storage block, target row address, and target column address. Data reading or writing is then performed at the corresponding physical location in the dynamic random access memory via the vertical storage interface. This scheme, through precise local address stripping and fine-grained hierarchical parsing, directly translates computation requests into low-level physical timing instructions for the local vertically stacked media. Utilizing the high throughput of the vertical storage interface, a high-bandwidth local closed-loop storage access mechanism with an extremely short physical path is established within the chip, shortening data loading time and solving the technical problems of high memory access latency and low local bandwidth utilization caused by long local memory access paths under unified addressing in related technologies.

[0066] In some embodiments, the remote memory access agent is used to receive a storage access request when the comparison result indicates that the target kernel identifier does not match the local kernel identifier; the remote memory access agent is used to encapsulate the storage access request into a communication message; and the remote memory access agent is used to send the communication message to the distributed router.

[0067] Here, encapsulation refers to the data processing process of packaging and combining raw instructions and data according to the standard format of network transmission protocols, and adding auxiliary transmission fields.

[0068] In some embodiments, the remote memory access agent reads the comparison result. When the comparison result indicates that the target kernel identifier does not match the local kernel identifier, the remote memory access agent receives a storage access request from the internal buffer bus. After receiving the storage access request, the remote memory access agent extracts the opcode, target physical address, and valid data payload carried in the storage access request. After extraction, the remote memory access agent generates a message header according to the format standard of the internal inter-kernel interconnection protocol. After generating the message header, the remote memory access agent embeds the opcode, target physical address, and valid data payload after the message header, and combines and encapsulates them into a communication message. After encapsulation into a communication message, the remote memory access agent pushes the communication message to the sending queue of the distributed router. The distributed router obtains the communication message and performs subsequent forwarding operations across computing storage kernels.

[0069] In some embodiments, the process of a remote memory access agent encapsulating a storage access request into a communication message can employ a fragmentation mechanism to adapt to the underlying network transmission specifications. The remote memory access agent receives the storage access request. Upon receiving the storage access request, the remote memory access agent calculates the total length of the effective data payload carried by the storage access request. The remote memory access agent compares the total length with the maximum transmission unit length allowed for a single communication message. When the total length exceeds the maximum transmission unit length, the remote memory access agent segments the effective data payload into multiple data slices. The remote memory access agent generates an independent message header carrying a slice sequence number for each data slice. The remote memory access agent combines the independent message headers with the corresponding data slices, encapsulating a storage access request into multiple consecutive communication messages.

[0070] For example, the remote memory access agent reads a mismatch and receives a write-type storage access request carrying a 256-byte payload. The remote memory access agent identifies the maximum transmission unit (MTB) of the underlying network protocol as 64 bytes. The remote memory access agent divides the 256-byte payload into four 64-byte data slices. The remote memory access agent generates four independent headers. The remote memory access agent injects sequence numbers 0, 1, 2, and 3 sequentially into the headers, along with the target physical address. The remote memory access agent combines the four independent headers with the corresponding 64-byte data slices to encapsulate four communication messages. The remote memory access agent then continuously sends these four communication messages to the send queue of the distributed router.

[0071] This application embodiment receives storage access requests when the comparison results do not match via a remote memory access agent, encapsulates the storage access requests into standard format communication messages, and finally sends the communication messages to a distributed router for forwarding. The above scheme automatically identifies cross-chip data access requests by independently dividing the local and remote memory access agent logic within the hardware architecture. By utilizing hardware protocol encapsulation to seamlessly convert memory semantic instructions into network interconnect semantic messages, it establishes a data transmission channel between the computing engine and the external chip array, solving the technical problems of high transmission latency and significant system overhead caused by the need for operating system software intervention for cross-node remote memory access in related technologies.

[0072] In some embodiments, the computing die further includes a link layer; a remote memory access agent is used to allocate a transaction identifier for the received storage access request and inject the transaction identifier into the communication message; the link layer is used to encode the communication message output by the distributed router and send the encoded communication message to the computing storage die corresponding to the target die identifier in a serial or parallel manner; the remote memory access agent is used to receive a response message including the transaction identifier and update the transaction completion status corresponding to the transaction identifier based on the response message.

[0073] Here, the link layer refers to the hardware circuit layer responsible for physical channel adaptation of communication messages and signal encoding / decoding logic. A transaction identifier is a hardware digital number used to uniquely track a complete request interaction process in concurrent network transmission. Encoding processing refers to the signal processing process of adding checksums or converting the format of the original bit stream to meet the requirements of physical channel transmission. Serial or parallel mode refers to the physical output mode of sending the bit stream bit by bit sequentially along a single line or simultaneously along multiple lines. A response message is a data packet carrying the execution result returned by the computing and storage core corresponding to the target core identifier after completing the operation. Transaction completion status refers to the flag information recorded in the hardware register indicating whether a specific memory access request has been successfully processed.

[0074] In some embodiments, the remote memory access agent receives a storage access request. Upon receiving the request, the remote memory access agent controls an internal hardware counter to generate a unique transaction identifier. The agent injects this transaction identifier into the header field of the communication message during encapsulation. After injection, the distributed router acquires the communication message and passes it to the link layer. The link layer receives the communication message output by the distributed router. It performs encoding on the message to add verification information. After encoding, the link layer drives the underlying hardware interface to send the encoded message serially or in parallel via physical pins to the compute-storage kernel corresponding to the target kernel identifier. The compute-storage kernel corresponding to the target kernel identifier sends back a response message after processing. The remote memory access agent receives this response message. It extracts the transaction identifier from the response message. The agent looks up the local hardware trace table based on the extracted transaction identifier. After looking up the trace table, the agent updates the transaction completion status in the trace table based on the result carried in the response message.

[0075] In some embodiments, the coordinated process of link-layer encoding and remote memory access agent state tracking can be implemented by combining forward error correction and timeout retransmission mechanisms. The remote memory access agent allocates a transaction identifier and records the allocation timestamp in its internal hardware trace table. The link layer receives communication packets and performs forward error correction encoding to generate redundant error correction codes. After sending the encoded communication packet, the remote memory access agent starts a timeout timer bound to the transaction identifier. When the remote memory access agent receives a response packet including the transaction identifier before the timeout timer expires, it closes the timeout timer and updates the transaction completion status corresponding to the transaction identifier to a successful status. When the timeout timer expires and no response packet including the transaction identifier is received, the remote memory access agent regenerates a communication packet carrying the original transaction identifier based on the unupdated transaction completion status. The remote memory access agent sends the communication packet to the link layer to trigger a re-execution of the encoding and transmission operations.

[0076] For example, the remote memory access agent receives a storage access request. The remote memory access agent assigns a transaction identifier of 1024 to this storage access request. The remote memory access agent injects transaction identifier 1024 into the communication message. The link layer receives the communication message and performs cyclic redundancy check encoding. The link layer sends the encoded communication message in parallel via 16 data lines to the compute storage die corresponding to the target die identifier numbered 3. Three microseconds later, the remote memory access agent receives a response message. The remote memory access agent parses the response message to obtain the transaction identifier 1024. The remote memory access agent finds the transaction identifier 1024 in the hardware trace table and updates the corresponding transaction completion status from pending to completed based on the response message.

[0077] This application embodiment allocates storage access requests through a remote memory access agent and injects transaction identifiers into the communication messages. It utilizes the link layer to encode the communication messages output by the distributed router and sends them serially or in parallel. Finally, the remote memory access agent receives response messages to update the transaction completion status corresponding to the transaction identifier. This scheme achieves asynchronous request tracing at the hardware level through a transaction identifier mechanism, allowing the computing die to concurrently issue multiple out-of-order new requests while waiting for a remote response, thus improving network concurrency throughput. Simultaneously, the link layer's encoding processing ensures the reliability of physical transmission. By utilizing a closed-loop state tracing mechanism, it solves the technical problems in related technologies where the lack of hardware tracing for cross-die asynchronous concurrent access leads to chaotic response matching, and physical channel signal attenuation causes data transmission errors.

[0078] In some embodiments, at least one dynamic random access memory (DRAM) and a computing die are hybrid-bonded to form a vertical signal channel; electrical signals are transmitted between the computing die and at least one DRAM via through-silicon vias (TSVs) or microbumps; the computing die is stacked above or below at least one DRAM; a thermal diffusion sheet or a back-side power supply network is provided on the side of the computing die facing away from the DRAM.

[0079] Here, hybrid bonding refers to a high-density packaging process in semiconductor manufacturing that directly physically bonds metal pads to insulating dielectric layers without the use of traditional solder. Through-silicon vias (TSVs) are tiny vertical conductive channels that penetrate the substrate thickness within a silicon wafer. Microbumps are micrometer-sized metal connection bumps fabricated on the surface of a semiconductor die. Thermal diffusers are high thermal conductivity components used to rapidly conduct and evenly distribute heat within the chip. Backside power supply networks are networks of metal interconnects laid on the back of a semiconductor die to directly supply power to internal transistor circuits.

[0080] In some embodiments, a computing die and at least one dynamic random access memory (DRAM) are obtained. After obtaining the computing die and at least one DRAM, they are physically aligned. After alignment, the computing die is stacked above or below the at least one DRAM according to set chip heat dissipation direction parameters. After stacking, bonding is achieved at the mating interface using a hybrid bonding process. After bonding, a vertical signal channel is formed between the computing die and the at least one DRAM. During the formation of the vertical signal channel, through-silicon vias or microbumps arranged at the mating interface are used as conductive media to transmit high-frequency electrical signals. While transmitting high-frequency electrical signals, a high thermal conductivity heat spreader is mounted on the surface of the computing die facing away from the DRAM, or a back-side power supply network is fabricated on the surface of the computing die facing away from the DRAM.

[0081] In some embodiments, the adaptive structure packaging can be combined with a power detection mechanism for the coordinated configuration process of stacking direction, power supply network, and heat dissipation components. The packaging device reads the rated current density parameters and heat dissipation power parameters of the computing die. After reading the rated current density parameters and heat dissipation power parameters, when the heat dissipation power parameter is greater than a set power threshold, the packaging device controls a robotic arm to invert and stack the computing die on top of at least one dynamic random access memory. After stacking, the packaging device creates a back power supply network on the top-facing back side of the computing die using an etching process. After creating the back power supply network, external power is directly channeled into the back power supply network. After channeling to the back power supply network, the packaging device applies a thermal interface material to the area above the back power supply network. After applying the thermal interface material, a thermal diffusion sheet is attached and placed on top of the thermal interface material. After attaching the thermal diffusion sheet, the heat generated by the computing die operation is dissipated to the external space of the chip through the thermal diffusion sheet.

[0082] For example, a 100-micrometer-thick computing die is stacked on top of four layers of dynamic random access memory (DRAM). 20,000 microbumps are arranged between the computing die and the underlying DRAM using a hybrid bonding process with a 5-micrometer pitch. These 20,000 microbumps are used to collaboratively transmit electrical signals in the vertical direction and form a vertical signal channel. A back-side power supply network providing a constant 1-volt voltage is fabricated on the surface of the computing die facing away from the DRAM. A 2-millimeter-thick copper heat dissipation sheet is then placed over the back-side power supply network to accelerate heat dissipation.

[0083] This application embodiment forms a vertical signal channel through hybrid bonding, transmits electrical signals using through-silicon vias (TSVs) or microbumps, and stacks computing dies above or below at least one dynamic random access memory (DRAM). A thermal diffusion sheet or a back-side power supply network is disposed on the back surface of the computing die. This solution, through high-density three-dimensional hybrid bonding technology, shortens the physical interconnect distance between chips, increases the bandwidth density of signal transmission, and reduces transmission power consumption caused by parasitic capacitance. The combination of flexible physical stacking orientation and the synergistic design of the back-side power supply network and thermal diffusion sheet effectively manages the high-density heat and high-current demands generated by logic operations while improving three-dimensional spatial integration. This solves the technical problems of low interconnect bandwidth density caused by two-dimensional planar packaging and severe heat dissipation bottlenecks and power losses caused by three-dimensional stacking structures in related technologies.

[0084] In some embodiments, when the compute memory chip includes an unconnected boundary extension port, the compute memory chip is used to perform termination detection on the unconnected boundary extension port; the compute memory chip is used to shut down the drive circuit of the unconnected boundary extension port to perform a shielding operation when the termination detection fails.

[0085] Here, an unconnected boundary extension port refers to a physical reserved interface that is currently physically floating and has not yet been connected to any external computing or storage chips. Termination detection is a hardware probing process that determines whether a port pin is connected to a valid load by measuring the impedance or signal reflection on the physical line. Termination detection failure indicates that the port is actually physically open or in a state of severe impedance mismatch. The driver circuit refers to the hardware transmitting component responsible for sending first-order signal outputs to the physical pins and outputting the toggling drive current. Shielding operation refers to cutting off the interface's power supply or logic clock, putting it into a silent or high-impedance isolated state.

[0086] In some embodiments, after the multi-chip integrated circuit is initialized, the computing and storage chip scans the connection status of each boundary extension port. When an unconnected boundary extension port is detected, the control hardware detection module performs termination detection on that unconnected boundary extension port. After performing termination detection, the hardware detection module outputs the impedance parameter status on the line. When the impedance parameter status indicates that the termination detection failed, the computing and storage chip outputs a shutdown control command. The computing and storage chip sends the shutdown control command to the drive circuit corresponding to the unconnected boundary extension port. The drive circuit shuts down its own output channel according to the shutdown control command, cutting off the signal drive to the unconnected boundary extension port, thereby completing the shielding operation for the unconnected boundary extension port.

[0087] In some embodiments, the termination detection and shielding operation process can be precisely detected and controlled by combining timing pulse reflection detection and static power gating mechanisms. During the power-on initialization phase, the computational storage chip control signal generator sends a probe pulse signal of a specific duration to the unconnected boundary extension port. After sending the probe pulse signal, the computational storage chip receives the return pulse level reflected from the physical line. The computational storage chip calculates the equivalent termination impedance based on the voltage peak ratio of the return pulse level. When the equivalent termination impedance exceeds a preset open-circuit impedance upper limit threshold, the termination detection is determined to have failed. After determining that the termination detection has failed, the computational storage chip disconnects the power supply rail corresponding to the drive circuit of the unconnected boundary extension port and sets the interface pin to a high-impedance output state. After setting to a high-impedance output state, the computational storage chip sets the shielding control bit in the local status register to a logic valid bit to inform the distributed router not to allocate communication packets to the unconnected boundary extension port.

[0088] For example, the computing memory chip has four boundary expansion ports. During power-up, the computing memory chip injects a probe pulse signal with a duration of 10 nanoseconds into these four boundary expansion ports. After termination detection, the computing memory chip measures an equivalent termination impedance of 50 ohms for two of the boundary expansion ports, completing normal termination. The equivalent termination impedance measured for the other two boundary expansion ports exceeds 1 megaohm, which is judged as an open circuit and the termination detection is confirmed to have failed. The computing memory chip sends a shutdown command to the two unconnected boundary expansion ports with impedances exceeding 1 megaohm, cutting off the 0.8-volt supply voltage to the output transistor in its own drive circuit. After cutting off the voltage, its physical pin is placed in a high-impedance output state, and the shield control bit in the corresponding physical port status register is set to the number 1.

[0089] This application embodiment performs termination detection on unconnected boundary extension ports via computational memory chips, and performs a shielding operation by shutting down the corresponding drive circuit when the termination detection fails. The above scheme accurately identifies reserved interfaces in a physically floating state through underlying pulse reflection and impedance detection. When the detection fails, the drive circuit is promptly shut down, and unnecessary switching logic and static power supply are cut off. By utilizing proprietary termination detection and shielding operations, this solves the technical problem in related technologies where scalable chips continue to output clocks and signals when reserved interfaces are not connected, resulting in ineffective static and dynamic power consumption waste and severe parasitic radiation interference.

[0090] In some embodiments, the computational storage chip is used to send a probe signal during the initialization of the multi-chip integrated circuit; the computational storage chip is used to obtain a valid port bitmap fed back by the probe signal; the distributed router is used to generate a routing table based on the valid port bitmap, and determine the next-hop port among the nearest neighbor ports and the boundary extension ports within the group based on the routing table.

[0091] Here, initialization refers to the startup preparation phase after the multi-chip integrated circuit powers on, involving hardware self-testing and network topology establishment. The probe signal is a short pulse-level signal used to detect the presence and connection status of adjacent compute-storage chips on the physical link. The valid port bitmap is a set of flag bits used to record the connection status of each physical port around the compute-storage chip using a binary bit sequence. The routing table is a hardware data structure that records the mapping relationship between the physical coordinates of a target node and its local physical ports.

[0092] In some embodiments, the multi-chip integrated circuit enters the initialization phase. During initialization, the computing and storage chip drives the hardware interface circuitry to send probe signals outwards through all surrounding physical ports. After sending the probe signals, the computing and storage chip activates its receiving circuitry to receive external probe feedback signals. Based on the state of the probe feedback signals, the computing and storage chip identifies which physical ports have established valid physical connections. After identifying valid physical connections, the computing and storage chip combines and concatenates the connection states of each physical port to generate a valid port bitmap. After generating the valid port bitmap, the computing and storage chip inputs the valid port bitmap into the distributed router. The distributed router parses the valid port bitmap. After parsing the valid port bitmap, the distributed router generates a hardware routing table based on the parsed valid link directions. After generating the hardware routing table, when a communication packet is received, the distributed router queries the routing table. Based on the query results of the routing table, the distributed router determines the next-hop port from the group's nearest neighbor ports and boundary extension ports that are in a valid connection state.

[0093] In some embodiments, the process of sending probe signals and generating routing tables can be combined with a neighboring node coordinate exchange mechanism for dynamic topology discovery. The computational storage core sends a probe signal. Simultaneously with sending the probe signal, the computational storage core modulates its own local coordinate data into the probe signal and sends it out together. After sending the local coordinate data, the computational storage core receives feedback probe signals carrying neighboring coordinate data from neighboring computational storage cores. The computational storage core extracts neighboring coordinate data. After extracting the neighboring coordinate data, the computational storage core generates a valid port bitmap with neighboring node coordinate information based on the physical port locations of the received neighboring coordinate data. The distributed router obtains the valid port bitmap with neighboring node coordinate information. After obtaining it, the distributed router uses the neighboring coordinate data to calculate the shortest forwarding path. After calculating the shortest forwarding path, the distributed router writes the shortest forwarding path into its routing table so that subsequent packet forwarding can directly look up the table to determine the next-hop port.

[0094] For example, the computing and storage chip has four interconnect ports around its perimeter. During power-on initialization of the multi-chip integrated circuit, the computing and storage chip sends probe signals through these four interconnect ports. The computing and storage chip receives high-level feedback signals on the east and south ports, but not on the west and north ports. The computing and storage chip generates a valid port bitmap consisting of a 4-bit binary sequence, with the bitmap data being 1010. Here, 1 represents a connected port and 0 represents a disconnected port. The distributed router reads the valid port bitmap 1010. Based on the valid port bitmap, the distributed router generates a routing table, activating the east-facing intra-group neighbor ports and the south-facing boundary extension ports and adding them to the available forwarding table entries. When the distributed router receives a communication packet, it queries the routing table to determine the east-facing intra-group neighbor port as the next-hop port.

[0095] This application embodiment obtains a valid port bitmap from the feedback of the probe signal by sending a probe signal during the initialization of the computing storage chip. A distributed router then generates a routing table based on the valid port bitmap and determines the next-hop port from the nearest neighbor ports and boundary extension ports within the group. This scheme, through a dynamic detection mechanism at the hardware level, enables each computing storage chip to autonomously perceive its physical connectivity and dynamically construct the network topology, without the need for pre-coding globally fixed routing rules. The dynamically generated routing table guides port forwarding, automatically identifying new nodes and adapting forwarding paths when the array size of multi-chip integrated circuits expands. This solves the technical problems of rigid static routing configuration and inability to adapt to dynamic expansion of the chip array size in related technologies.

[0096] In some embodiments, the multi-chip integrated chip includes multiple chip arrays, which are spliced ​​together in the horizontal and vertical directions to form a sixteen-chip array, or multiple chip arrays are spliced ​​together in the horizontal direction to form an eight-chip array; in the multiple chip arrays, a distributed router is used to prioritize the use of neighboring ports within the group to forward communication packets; the distributed router is used to obtain the congestion status of the boundary extension ports and distribute communication packets in multiple pairs of parallel boundary extension ports based on the congestion status.

[0097] Here, the horizontal direction refers to the physical arrangement direction extending along the horizontal row dimension within the two-dimensional plane of the chip package. The vertical direction refers to the physical arrangement direction extending along the vertical column dimension within the two-dimensional plane of the chip package. An eight-chip array refers to a two-row, four-column physical array architecture formed by horizontally splicing two basic chip arrays in two rows and two columns. A sixteen-chip array refers to a four-row, four-column physical array architecture formed by horizontally and vertically splicing four basic chip arrays in two rows and two columns respectively. Congestion status refers to the hardware congestion quantification indicator of the transmit buffer queue depth or available link credit of the communication interface. Multiple pairs of parallel boundary extension ports refer to two or more cross-array physical interconnect links that simultaneously transmit data synchronously between the physical interfaces of two adjacent chip arrays. Distribution refers to the data scheduling logic that hashes and routes multiple communication messages to different physical channels for concurrent transmission according to specific allocation rules.

[0098] For example, Figure 4 This is a schematic diagram of an eight-chip array structure of a multi-chip integrated chip provided in an embodiment of this application. Figure 4 As shown, the multi-chip integrated circuit includes two identical chip arrays, each containing four compute and storage chips. The reserved boundary extension port on the right edge of the left chip array is physically connected to the open boundary extension port on the left edge of the right chip array, allowing the two chip arrays to be horizontally joined to form a structure containing the first to eighth compute and storage chips. Figure 4An eight-piece array with two rows and four columns (labeled C0 to C7).

[0099] Or see Figure 5 , Figure 5 This is a schematic diagram of a sixteen-chip array structure of a multi-chip integrated circuit provided in an embodiment of this application. Figure 5 As shown, the multi-chip integrated circuit comprises four identical chip arrays. These four chip arrays are arranged in a square matrix in a two-dimensional packaging plane, interconnected using open boundary extension ports. This allows the multiple chip arrays to be spliced ​​together horizontally and vertically to form a chip containing sixteen computing and storage chips. Figure 5 A 16-piece array in four rows and four columns (labeled C0 to C15).

[0100] When operating with an eight- or sixteen-chip array, the distributed router receives communication packets and extracts their target physical coordinates. If the target physical coordinates are within the current chip array, the distributed router prioritizes forwarding the communication packet using the nearest neighbor port within the group. If the target physical coordinates are within another chip array, the distributed router reads the status registers of multiple pairs of parallel boundary extension ports connected to the corresponding array to obtain the congestion status of the boundary extension ports. After obtaining the congestion status, the distributed router distributes the cross-array data traffic to be sent across the multiple pairs of parallel boundary extension ports based on the congestion status.

[0101] In some embodiments, the process of a distributed router acquiring congestion status and distributing communication packets among multiple pairs of parallel boundary extension ports can be implemented by combining dynamic allocation of link credits and dynamic load balancing mechanisms. The distributed router monitors the hardware link credit values ​​and transmission queue depths corresponding to the multiple pairs of parallel boundary extension ports in real time. The distributed router calculates the instantaneous communication impedance of each physical extension channel based on the link credit value and transmission queue depth. When the difference in instantaneous communication impedance between different physical extension channels is lower than a preset congestion deviation threshold, the distributed router distributes communication packets evenly to the multiple pairs of parallel boundary extension ports using a hash round-robin algorithm. When the instantaneous communication impedance of one of the physical extension channels is detected to be higher than a preset congestion alarm threshold, the distributed router reduces the data routing distribution ratio for that high-impedance interface and dynamically transfers subsequent communication packets entering the buffer bus to idle boundary extension ports with lower instantaneous communication impedances for concurrent forwarding.

[0102] For example, in such Figure 4In the eight-chip array shown, the first computing storage chip (C1) and the second computing storage chip (C5) are connected to an adjacent chip array to the right. A first pair of boundary extension ports is established between C1 and C2, and a second pair of boundary extension ports is established between C5 and C6. When the left chip array generates 10 communication packets, each 512 bytes long, that need to be sent to the right array, the distributed router in C1 reads that the available link credits of the first pair of boundary extension ports are 80, and the available link credits of the second pair of boundary extension ports are 20. The distributed router calculates the distribution ratio based on a 4:1 ratio of available link credits. The distributed router directly sends 8 of the communication packets from C1 to C2 concurrently via the first pair of boundary extension ports, and then forwards the remaining 2 communication packets to C5 via the nearest neighbor port within the group, and finally to C6 via the second pair of boundary extension ports.

[0103] This application's embodiments involve horizontally splicing multiple core arrays to form an eight-panel array or horizontally and vertically splicing to form a sixteen-panel array. A distributed router prioritizes forwarding communication packets through neighboring ports within the group and obtains the congestion status of boundary extension ports to dynamically distribute communication packets among multiple pairs of parallel boundary extension ports. This scheme smoothly expands the basic array into a large-capacity physical matrix through modular splicing, and uses two-level port splitting to achieve physical isolation between high-frequency local memory access within the group and long-distance traffic across arrays. Simultaneously, it utilizes underlying congestion status awareness to dynamically distribute and schedule traffic across multiple parallel cross-array boundary links, improving the utilization efficiency of the total cross-array bandwidth. This solves the technical problems of communication aggregation and congestion, as well as extremely unbalanced loads, that easily occur on cross-array boundary links during large-scale multi-core topology splicing in related technologies.

[0104] In some embodiments, the distributed router is configured to: obtain the target coordinates of the target computational storage chip pointed to by the communication message and the local coordinates of the computational storage chip itself; compare the lateral dimension coordinates of the target coordinates with the lateral dimension coordinates of the local coordinates to obtain a lateral comparison result; when the lateral comparison result indicates that the lateral dimension coordinates are inconsistent, determine the next-hop port in the chip interconnection port in the lateral interconnection direction, and forward the communication message through the next-hop port to reduce the lateral coordinate difference; compare the longitudinal dimension coordinates of the target coordinates with the longitudinal dimension coordinates of the local coordinates to obtain a longitudinal comparison result; when the longitudinal comparison result indicates that the longitudinal dimension coordinates are inconsistent, determine the next-hop port in the chip interconnection port in the longitudinal interconnection direction, and forward the communication message through the next-hop port.

[0105] Here, target coordinates refer to data parameters indicating the exact location of the final receiving node in a two-dimensional physical array. Local coordinates refer to data parameters indicating the absolute location of the computing and storage core where the distributed router is currently located in the two-dimensional physical array. Horizontal dimension coordinates refer to the physical coordinate indices set along the row direction of the physical node in the two-dimensional matrix. Vertical dimension coordinates refer to the physical coordinate indices set along the column direction of the physical node in the two-dimensional matrix. Horizontal comparison result refers to the logical judgment flag generated by the distributed router after comparing two horizontal position indices. Vertical comparison result refers to the logical judgment flag generated by the distributed router after comparing two vertical position indices. Horizontal coordinate difference refers to the mathematical difference between the horizontal position index of the target node and the horizontal position index of the current node.

[0106] In some embodiments, the distributed router acquires a communication message to be sent. After acquiring the communication message, the distributed router extracts the target coordinates of the target compute storage die to which the communication message points. After extracting the target coordinates, the distributed router reads its built-in hardware registers to obtain the local coordinates of the compute storage die itself. After obtaining the local coordinates, the distributed router extracts the horizontal dimension coordinates of the target coordinates and the horizontal dimension coordinates of the local coordinates. The distributed router compares the horizontal dimension coordinates of the target coordinates with the horizontal dimension coordinates of the local coordinates to obtain a horizontal comparison result. When the horizontal comparison result indicates that the horizontal dimension coordinates are inconsistent, the distributed router determines the next-hop port in the interconnection ports between dies in the horizontal interconnection direction. After determining the next-hop port, the distributed router forwards the communication message through the next-hop port. The communication message moves to adjacent compute storage dies to reduce the difference in horizontal coordinates. When the communication message is transmitted to a node with consistent horizontal dimension coordinates, the distributed router extracts the vertical dimension coordinates of the target coordinates and the vertical dimension coordinates of the local coordinates. The distributed router compares the vertical dimension coordinates of the target coordinates with the vertical dimension coordinates of the local coordinates to obtain a vertical comparison result. When the vertical alignment results represent inconsistent vertical dimension coordinates, the distributed router determines the next-hop port in the inter-core interconnection port in the vertical interconnection direction and forwards communication packets through that next-hop port.

[0107] In some embodiments, for the dimension-order routing forwarding process that proceeds first laterally and then vertically, a virtual channel isolation mechanism can be combined to improve transmission reliability. The distributed router obtains the lateral comparison results. When the lateral comparison results indicate inconsistencies in the lateral dimension coordinates, the distributed router allocates the communication packet to a first virtual channel dedicated to lateral transmission. After allocation to the first virtual channel, a forwarding operation to reduce the lateral coordinate difference is performed using the first virtual channel. When the lateral coordinate difference is reduced to zero and the vertical comparison results indicate inconsistencies in the vertical dimension coordinates, the distributed router removes the identifier from the first virtual channel. After removing the identifier, the distributed router switches the communication packet to a second virtual channel dedicated to vertical transmission. After allocation to the second virtual channel, the final vertical data delivery is completed through the second virtual channel.

[0108] For example, the multi-chip integrated circuit is arranged in a 4x4 grid. The distributed router obtains the target coordinates as horizontal coordinate 4 and vertical coordinate 1. The distributed router reads the local coordinates as horizontal coordinate 2 and vertical coordinate 3. The distributed router compares horizontal coordinate 4 with horizontal coordinate 2, obtaining a horizontal comparison result indicating a mismatch in the horizontal coordinates. The distributed router determines that it needs to move horizontally in the positive direction and determines the next-hop port in the inter-chip interconnection port in the eastward interconnection direction. The communication packet is forwarded eastward for 2 consecutive hops, reducing the horizontal coordinate difference to 0, reaching the relay node with horizontal coordinate 4 and vertical coordinate 3. The distributed router at the relay node finds that the horizontal coordinates are consistent, and then compares vertical coordinate 1 with vertical coordinate 3. After comparison, a vertical comparison result indicating a mismatch in the vertical coordinates is obtained. The distributed router determines that it needs to move vertically in the negative direction and determines the next-hop port in the northward inter-chip interconnection port. The communication packet is forwarded northward for 2 hops to reach the final receiving node.

[0109] This application embodiment obtains the target coordinates and local coordinates through a distributed router, and compares the horizontal and vertical coordinates sequentially. When the horizontal coordinates are inconsistent, forwarding is prioritized through the inter-core interconnection port in the horizontal interconnection direction to reduce the difference in horizontal coordinates. When the vertical coordinates are inconsistent, forwarding is prioritized through the inter-core interconnection port in the vertical interconnection direction. The above scheme, through a hardware-implemented two-dimensional dimensional routing logic, enforces a deterministic data transmission path that prioritizes horizontal routing over vertical routing. By utilizing a deterministic routing path, the conditions for physical transmission loops are blocked, reducing the storage resource requirements of the large-scale routing table within the distributed router, improving the hardware processing efficiency of the underlying packet forwarding, and solving the technical problems of packet circular deadlock caused by non-deterministic routing and excessive hardware overhead due to large routing tables in multi-core networks in related technologies.

[0110] In some embodiments, the distributed router is used to allocate mutually isolated independent virtual channels for request messages and response messages included in the communication messages.

[0111] Here, a request message refers to a data packet initiated by the source node, requesting the target node to perform a specific memory access operation or provide status feedback. A response message refers to a data packet returned by the target node to the source node after processing the initial instruction, carrying the operation result or confirmation status. An independent virtual channel refers to a logical transmission channel on the same physical link, divided using hardware multiplexing technology, with independent buffer queues and independent flow control mechanisms. Mutual isolation means that the data buffering and scheduling states of different logical transmission channels do not interfere with each other; even if one channel becomes congested, it will not affect the normal data transmission and reception process of other channels.

[0112] In some embodiments, the distributed router receives communication messages to be processed from the internal system bus. Upon receiving a communication message, the distributed router parses the header parameters of the message. Based on the instruction opcode attribute in the header parameters, the distributed router identifies whether the communication message is a request message or a response message. After identifying the message type, the distributed router controls the hardware allocation logic. When the communication message is determined to be a request message, the distributed router allocates the request message to the first independent virtual channel. When the communication message is determined to be a response message, the distributed router allocates the response message to the second independent virtual channel. The first and second independent virtual channels are multiplexed on the physical transmission line, but internally, the distributed router uses isolated queues for buffering and flow control.

[0113] In some embodiments, the isolation control process of allocating independent virtual channels for request and response messages can be combined with a two-layer credit scheduling mechanism to achieve protocol-level deadlock prevention. The distributed router initializes two sets of independent hardware credit counters at the inter-core interconnect port. The first set of hardware credit counters specifically records the buffer space capacity of the next-hop node for receiving request messages. The second set of hardware credit counters specifically records the buffer space capacity of the next-hop node for receiving response messages. When the distributed router allocates a request message to the first independent virtual channel, it only checks and consumes the data from the first set of hardware credit counters. When the distributed router allocates a response message to the second independent virtual channel, it only checks and consumes the data from the second set of hardware credit counters. Using two sets of independent hardware credit control, when the first independent virtual channel for a request message stops forwarding due to network end-point congestion, the response message can still use the unoccupied second independent virtual channel to return and release system resources.

[0114] For example, the distributed router receives two communication packets. Parsing the header parameters reveals that the first packet carries a read opcode for reading hexadecimal address 0x12340000, identifying it as a request packet. The second packet carries an acknowledgment opcode confirming successful write, identifying it as a response packet. The distributed router pushes the request packet for reading hexadecimal address 0x12340000 into the independent virtual channel buffer queue numbered channel 0. The distributed router pushes the acknowledgment response packet confirming successful write into the independent virtual channel buffer queue numbered channel 1. At this time, the buffer queue for channel 0 is in a stopped sending state due to network congestion, but channel 1 has 5 available credit units. The distributed router continues to send acknowledgment response packets confirming successful write in channel 1 via the physical link.

[0115] This application embodiment allocates isolated virtual channels to request and response messages within the communication packets using distributed routers. This scheme, by virtualizing logical transmission paths with independent flow control buffers on the same physical link, blocks circular resource dependencies between instruction requests and result feedback at the hardware scheduling level. By utilizing the independence and non-interference mechanism of different types of message transmission queues, it ensures smooth routing of feedback results during localized network congestion, solving the technical problem of severe protocol-level deadlocks caused by request and response sharing the same physical queue in multi-core networks during congestion in related technologies.

[0116] In some embodiments, the communication message includes a priority field, which indicates the priority level of the communication message; when the communication message carries key-value cache or gradient synchronization data, the priority field is configured as the first priority; the distributed router is used to parse the priority field and, in the nearest neighbor port or boundary extension port within the group, prioritize forwarding communication messages with the first priority field; the distributed router is configured with multiple alternative forwarding paths to the target computing storage core, and the multiple alternative forwarding paths correspond to different inter-core interconnection ports; the distributed router is also used to obtain the link credit or sending queue depth of the inter-core interconnection ports corresponding to the multiple alternative forwarding paths, and, based on the link credit or sending queue depth, determine the next-hop port among the inter-core interconnection ports corresponding to the multiple alternative forwarding paths.

[0117] Here, the priority field refers to the binary identifier data in the communication message used to declare the importance and urgency of data packets. Priority level refers to the quantified level by which the hardware scheduling logic sorts and ranks the forwarding order of data packets. Key-value cache refers to the historical context feature matrix data temporarily stored during the generative inference process of artificial intelligence models to accelerate autoregressive computation. Gradient synchronization data refers to the backpropagation parameters of errors that need to be frequently exchanged and reduced between various computing nodes during the distributed training process of artificial intelligence models. First priority refers to the processing level that enjoys the highest preemption right and the first forwarding right in a multi-level scheduling strategy. Multiple alternative forwarding paths refer to the set of various different but reachable intermediate forwarding routes connecting the source node and the target computing and storage core in the physical topology. Link credit is a numerical indicator representing the current remaining available buffer space capacity of the downstream receiving node. Sending queue depth refers to the number of data packets waiting to be sent that are backed up in the current port hardware buffer queue.

[0118] In some embodiments, the computational die generates communication packets. When generating the communication packets, the data processing module extracts the payload type carried by the packets. When the payload type is determined to be key-value cache or gradient synchronization data, the data processing module configures the priority field of the communication packet to the first priority. After configuration, the communication packet enters the distributed router. The distributed router parses the priority field of the communication packet. Among the nearest ports or boundary extension ports within the group, the distributed router identifies communication packets with a priority field of the first priority and forcibly interrupts the scheduling of low-priority packets currently being processed, directly prioritizing the forwarding of these communication packets. During this process, the distributed router queries its internal hardware routing table to read multiple alternative forwarding paths to the target computational storage die. These alternative forwarding paths point to interconnection ports between dies in different physical directions. The distributed router obtains the current link credit or sending queue depth of the interconnection ports corresponding to these alternative forwarding paths. After obtaining this information, the distributed router compares the link credit or sending queue depth. Based on the comparison results, the distributed router determines the optimal channel as the next-hop port among the interconnection ports corresponding to the multiple alternative forwarding paths.

[0119] In some embodiments, the process of determining the next-hop port based on link credit or sending queue depth can be implemented by combining adaptive shortest path and congestion detour mechanisms. The distributed router obtains two alternative forwarding paths to the target computing and storage core: a horizontal priority path and a vertical priority path. The distributed router reads the link credit of the horizontal interconnection port and the sending queue depth of the vertical interconnection port. The distributed router calculates the congestion weight values ​​of the horizontal and vertical interconnection ports. When the congestion weight value of the horizontal interconnection port is less than a set threshold, the distributed router determines the horizontal interconnection port as the next-hop port, causing communication packets to be forwarded along the shortest horizontal path. When the congestion weight value of the horizontal interconnection port is greater than the set threshold and the congestion weight value of the vertical interconnection port is extremely low, the distributed router triggers congestion detour logic. After triggering the congestion detour logic, the distributed router deviates from the default horizontal shortest path and determines the vertical interconnection port as the next-hop port.

[0120] For example, a distributed router receives two communication packets. The first packet carries ordinary weight loading data with a priority field set to second priority. The second packet carries gradient synchronization data with a priority field set to first priority. After resolving that the second packet has the first priority, the distributed router pushes it to the head of the sending queue. At this point, there are two alternative forwarding paths to the target computation-storage core: eastbound and southbound. The distributed router obtains that the link credit of the eastbound interconnect port is 0 and the sending queue depth is 20, and that the link credit of the southbound interconnect port is 15 and the sending queue depth is 2. Based on the greater link credit and shallower sending queue depth of the southbound interconnect port, the distributed router determines the southbound interconnect port as the next-hop port from the two alternative forwarding paths and prioritizes sending the communication packet carrying the gradient synchronization data.

[0121] This application embodiment indicates the priority level by configuring the priority field of communication messages. Messages carrying key-value caches or gradient synchronization data are assigned the first priority and forwarded preferentially using distributed routers. Simultaneously, the link credit or sending queue depth corresponding to multiple alternative forwarding paths is obtained, and the next-hop port is determined from the inter-core interconnect ports corresponding to the alternative forwarding paths. The above scheme, through business semantic-aware hardware priority scheduling, ensures that critical contexts and synchronization parameters that are extremely sensitive to latency in AI computing are not blocked by ordinary data flows, thus accelerating the release of computing power. Adaptive multi-path routing scheduling using link state awareness effectively avoids local network hotspots and solves the technical problems in related technologies where uniform, indiscriminate forwarding leads to a surge in critical communication latency, and static routing cannot dynamically alleviate data traffic in the face of local congestion.

[0122] In some embodiments, the computing die further includes an on-chip buffer memory for temporarily storing computing data; the computing die is used to perform a data write operation to at least one dynamic random access memory or invalidate the computing data in the on-chip buffer memory in response to a refresh instruction, an invalidation instruction, or a read-write barrier instruction; the computing die is used to synchronize the data visibility status of computing data with other computing storage dies through the data write operation or invalidation.

[0123] Here, on-chip buffer memory refers to a collection of high-speed static random access memory devices integrated within the computing die, physically close to the computing engine, and with extremely low read / write latency. Computational data refers to intermediate temporary variables or final output results generated by the computing engine during the execution of computational instructions. A refresh instruction is a hardware control instruction issued by software or a control unit that forces the modified data in the on-chip buffer memory to be written back to main memory. An invalidation instruction is a control instruction used to mark a copy of data in the on-chip buffer memory as expired and invalid, requiring subsequent reads of the latest data from main memory. A read / write barrier instruction is a synchronization control instruction used in multi-threaded or multi-node concurrent execution to enforce the instruction execution order, ensuring that subsequent operations can only be executed after previous memory operations are fully visible. A data write operation refers to the process of transferring and storing charge signals from high-speed static random access memory devices to the physical capacitor array of dynamic random access memory. Invalidation processing refers to the logical reset process of clearing the status bits of the corresponding data in the on-chip buffer memory hardware catalog list or marking it as unusable. Data visibility status refers to a system-wide consistency indicator that shows whether data modifications at a specific physical address can be correctly observed and read by other independent computing and storage cores in the network.

[0124] In some embodiments, the computing die temporarily stores computational data generated by the computing engine using on-chip buffer memory. After storing the computational data, the computing engine continues to execute subsequent instruction streams. When a refresh instruction, invalidation instruction, or read-write barrier instruction appears in the computing engine's instruction stream, the computing engine sends a response signal to the control bus. In response to the refresh instruction or read-write barrier instruction, the computing die retrieves the modified computational data that has not yet been written back from the on-chip buffer memory. After retrieval, the computing die performs a data write operation to at least one dynamic random access memory (DRAM) on this portion of the computational data, fixing the latest value into physical main memory. In response to the invalidation instruction, the computing die retrieves a cache line at a specified address in the on-chip buffer memory. After retrieving the corresponding cache line, the computing die invalidates the computational data in the on-chip buffer memory. After performing the data write operation or invalidation, the computing die updates its internal status register, synchronizing the data visibility status of the computational data with other computing storage dies in the multi-die network through the aforementioned explicit hardware operations.

[0125] In some embodiments, the synchronization mechanism for data visibility status can be implemented by combining lightweight short message broadcasting with a status event register. Upon receiving a read-write barrier instruction, the compute die suspends subsequent instruction fetching operations of the current compute engine. After suspending, the compute die initiates a forced data write operation to flush the compute data in the on-chip buffer memory to at least one dynamic random access memory. After the data flush is complete, the compute die generates an event control short message carrying a synchronization completion flag. The compute die sends this event control short message to other compute storage granules in the multi-granule array via a distributed router. Upon receiving the event control short message, the other compute storage granules update their internal status event registers to confirm that the data visibility status of the compute data has been achieved. After confirmation, the other compute storage granules release the blocking state of their local compute engines waiting for the compute data.

[0126] For example, the compute die temporarily stores 16 kilobytes of tensor multiplication and accumulation calculation data using an internal 2-megabyte on-chip buffer. To allow this data to be processed by an adjacent die, the compute die responds to a refresh instruction inserted in the program code, performing a data write operation to the vertically stacked dynamic random access memory (DRAM) below the 16 kilobytes of computation data. The data write operation takes 50 nanoseconds. After the write is complete, the compute die responds to a read-write barrier instruction inserted in the program code, sending a one-byte barrier arrival notification to the adjacent compute memory die. Upon receiving the notification, the adjacent compute memory die confirms that the visibility status of the 16 kilobytes of data has been updated and then initiates a read operation to the DRAM.

[0127] This application embodiment calculates the die response refresh instruction, invalidation instruction, or read-write barrier instruction to perform data writing to dynamic random access memory or invalidation processing on the computational data in the on-chip buffer memory, thereby synchronizing the data visibility status with other computing storage dies. The above scheme replaces the cumbersome full hardware cache consistency protocol with a lightweight synchronization mechanism triggered by explicit software instructions. While retaining the performance advantages of on-chip high-speed caching, it entrusts the precise control of cross-die data synchronization timing to software. By utilizing read-write barriers and fixed-point refresh, the global consistency boundary of data is effectively standardized, solving the technical problem in related technologies where maintaining multi-die global forced hardware cache consistency leads to massive invalid directory broadcasts, severely consuming interconnect communication bandwidth.

[0128] In some embodiments, at least one dynamic random access memory is used to store tensor data that has been physically fragmented; when the computing die executes cross-die set communication instructions, the communication message includes intra-group stage communication messages and cross-group stage communication messages; a distributed router is used to forward intra-group stage communication messages carrying tensor data through intra-group nearest neighbor ports; a distributed router is used to forward cross-group stage communication messages carrying tensor data through boundary extension ports.

[0129] Here, physical sharding refers to the process of physically dividing a large, complete computational matrix into multiple smaller data blocks according to network hierarchy or feature dimension during the training or inference of a large-scale artificial intelligence model. Tensor data refers to a multi-dimensional set of matrix values ​​representing weights, activation values, or gradient parameters in deep learning algorithms. Cross-core cluster communication commands refer to the underlying hardware control commands used in a multi-core architecture to coordinate synchronous aggregation operations such as global data reduction, full collection, or full distribution across all computing nodes. Intra-group phase communication messages refer to network data packets specifically used to transmit locally integrated data between nodes within the basic core array during the first half of the cluster communication process. Cross-group phase communication messages refer to network data packets specifically used to transmit globally aggregated data across physical boundaries between different core arrays during the second half of the cluster communication process.

[0130] In some embodiments, a computing die acquires a large artificial intelligence model. After acquiring the model, the compiler divides the model parameters into multiple smaller data blocks according to the memory capacity of the computing nodes, forming physically fragmented tensor data. The computing die discretizes and stores this physically fragmented tensor data in at least one dynamic random access memory within each computing storage die. After storage, when the computing die executes cross-die set communication instructions such as global gradient aggregation, the distributed router decomposes the instruction into two levels of communication logic. In the first level of communication logic, the computing die generates intra-group stage communication messages carrying tensor data. The distributed router forwards and locally accumulates the intra-group stage communication messages among the four computing storage dies belonging to the same physical array through intra-group nearest neighbor ports. After the local data accumulation is completed, the second level of communication logic is entered, where the computing die generates cross-group stage communication messages carrying locally accumulated tensor data. The distributed router forwards the cross-group stage communication messages to another adjacent die array through a reserved boundary extension port, and completes the final data synchronization globally.

[0131] In some embodiments, for the hierarchical execution process of cross-chip set communication instructions in two-level ports, a ring reduction mechanism can be combined to achieve optimal utilization of communication bandwidth. When responding to a cross-chip set communication instruction of the All-Reduce type, the four computation dies within the basic chip array logically constitute a first-layer local communication loop. In this first-layer local communication loop, the distributed router performs the first round of ring data delivery through the nearest neighbor port within the group, carrying intra-group stage communication messages containing tensor data, completing the tensor partial sum computation within the array. After completing the tensor partial sum computation, a boundary computation die is selected in the basic chip array as the master node. The master nodes of multiple adjacent arrays logically constitute a second-layer global communication loop. In this second-layer global communication loop, the distributed router of the master node performs the second round of ring data delivery through the boundary extension port, carrying cross-group stage communication messages containing local parts and results, completing the final tensor sum computation between arrays, and finally distributing the final result back to each computation storage chip within the array through the nearest neighbor port within the group.

[0132] For example, a multi-core integrated chip comprises two interconnected core arrays with a total of eight compute-storage cores. An 8-megabyte tensor is physically fragmented, with each compute-storage core storing 1 megabyte of fragmented data in its dynamic random access memory. When a cross-core set communication command is executed, the four compute-storage cores in the left array generate their respective intra-group phase communication messages. The distributed router circulates these messages among the four left cores via their intra-group nearest neighbor ports, calculating the 1-megabyte result of the local accumulation in the left array. Subsequently, the boundary core of the left array generates a cross-group phase communication message, and the distributed router forwards this message carrying the local accumulation result to the right array via its boundary extension port for final summation and synchronization.

[0133] This application's embodiments store physically fragmented tensor data in dynamic random access memory. When executing cross-core-set communication instructions, the communication message is divided into two stages: intra-group and cross-group. Distributed routers are used for hierarchical forwarding via intra-group nearest neighbor ports and boundary extension ports, respectively. This scheme, through deep integration of a two-level hardware communication routing architecture and AI distributed computing communication primitives, converges high-frequency, massive local data reduction traffic onto high-bandwidth intra-group links, transmitting only a very small amount of aggregated results via cross-group links. By perfectly matching the physical topology with the algorithmic communication mode, the communication bottleneck pressure across core-set boundaries is greatly reduced, solving the technical problems of severe network congestion and extreme waste of interconnect bandwidth caused by the explosion of global communication volume during large-scale matrix parallel computation in related technologies.

[0134] In some embodiments, this application provides a computing storage die, which includes a computing die and at least one dynamic random access memory stacked perpendicularly to the computing die. The computing die includes a distributed router and a plurality of inter-die interconnect ports disposed around the computing die. The computing storage die is used to send probe signals through the plurality of inter-die interconnect ports during initialization and generate an effective port bitmap based on the feedback results of the probe signals. The computing storage die is used to dynamically divide the plurality of inter-die interconnect ports into intra-group neighbor ports and boundary extension ports based on the effective port bitmap. Intra-group neighbor ports are used to communicate with other computing storage dies belonging to the same die array, and boundary extension ports are used to connect computing storage dies outside the die array across the array. The distributed router is used to determine the next-hop port among the intra-group neighbor ports and boundary extension ports, and forward communication messages through the next-hop port.

[0135] In some embodiments, Figure 6 This is a flowchart illustrating the distributed routing method provided in an embodiment of this application. The distributed routing method provided in this application is applied to a multi-chip integrated circuit as described above. The multi-chip integrated circuit includes at least one chip array, and the chip array includes multiple computing and storage chips. Each computing and storage chip includes a distributed router and multiple inter-chip interconnect ports disposed around the computing die of the computing and storage chip. See also... Figure 6 The distributed routing method provided in this application embodiment may include the following steps 601 to 605.

[0136] In step 601, a probe signal is sent via multiple inter-core interconnect ports by calculating the storage core.

[0137] In step 602, a valid port bitmap is generated based on the feedback results of the probe signal by calculating the storage core.

[0138] In step 603, by calculating the storage core, the interconnect ports between multiple cores are divided into intra-group nearest neighbor ports for connecting the inside of the core array and boundary extension ports for connecting the outside of the core array based on the effective port bitmap.

[0139] In step 604, the next-hop port is determined from the nearest neighbor ports within the group and the border extension ports using a distributed router.

[0140] In step 605, communication messages are forwarded via the next-hop port through a distributed router.

[0141] In some embodiments, determining the next-hop port among the nearest neighbor ports and boundary extension ports within a group using a distributed router can be achieved in the following ways: obtaining the target coordinates of the target computing storage chip pointed to by the communication message, and the local coordinates of the computing storage chip where the distributed router is located; comparing the lateral dimension coordinates of the target coordinates with the lateral dimension coordinates of the local coordinates to obtain a lateral comparison result; if the lateral comparison result indicates that the lateral dimension coordinates are inconsistent, determining the next-hop port among the nearest neighbor ports or boundary extension ports in the lateral interconnection direction to reduce the lateral coordinate difference; if the lateral comparison result indicates that the lateral dimension coordinates are consistent but the vertical dimension coordinates are inconsistent, determining the next-hop port among the nearest neighbor ports or boundary extension ports in the vertical interconnection direction.

[0142] In some embodiments, determining the next-hop port among the nearest neighbor ports and the boundary extension ports within a group using a distributed router can be achieved in the following way: obtaining multiple alternative forwarding paths configured in the distributed router to reach the target computing and storage core, each of the multiple alternative forwarding paths corresponding to different inter-core interconnect ports; obtaining the link credit or sending queue depth of the inter-core interconnect ports corresponding to the multiple alternative forwarding paths; and determining the next-hop port among the inter-core interconnect ports corresponding to the multiple alternative forwarding paths based on the congestion state represented by the link credit or sending queue depth.

[0143] In some embodiments, the communication message includes a priority field. The communication message is forwarded via the next-hop port through a distributed router, which can be achieved by: parsing the priority field in the communication message to determine the priority level of the communication message; allocating an independent virtual channel for the communication message when the priority level is first priority; wherein, the communication message carrying key-value cache data or gradient synchronization data is configured as first priority; and forwarding the communication message via the next-hop port with priority through the distributed router based on the independent virtual channel.

[0144] In some embodiments, after dividing multiple inter-core interconnect ports into intra-group neighbor ports for connecting the inside of the core array and boundary extension ports for connecting the outside of the core array based on the effective port bitmap, the distributed routing method provided in this application embodiment can also be implemented in the following way: performing termination detection on the boundary extension ports in the unconnected state; when the termination detection fails, shutting down the driving circuit of the boundary extension ports in the unconnected state to perform a shielding operation.

[0145] In some embodiments, this application provides a non-uniform memory access method applied to a multi-chip integrated circuit. The multi-chip integrated circuit has a unified global physical address space and includes multiple compute-memory chips. Each compute-memory chip includes vertically stacked compute dies and at least one layer of dynamic random access memory (DRAM). The method can be implemented as follows: a memory access request is generated by a compute engine in the compute die, the memory access request containing a target physical address in the global physical address space; the target physical address is parsed by a local address decoder to obtain a target chip identifier, and the target chip identifier is compared with a local chip identifier; if the target chip identifier matches the local chip identifier, the memory access request is distributed to a local memory controller, and the DRAM is accessed via a vertical signal channel through the local memory controller; if the target chip identifier does not match the local chip identifier, the memory access request is distributed to a remote memory access agent, the remote memory access agent encapsulates the memory access request into a communication message, and sends the communication message to a distributed router for cross-chip forwarding.

[0146] In some embodiments, this application provides a data communication method applied to a multi-chip integrated circuit. The multi-chip integrated circuit includes multiple chip arrays. The computing and storage chips within each chip array are interconnected via intra-group nearest neighbor ports, and adjacent chip arrays are interconnected via boundary extension ports. The method can be implemented as follows: in response to a cross-chip set communication instruction, determine the physically fragmented tensor data to be processed; decompose the cross-chip set communication instruction into an intra-group communication stage and an inter-group communication stage; in the intra-group communication stage, through a distributed router of the computing and storage chips, forward and reduce the intra-group stage communication message carrying the tensor data within the chip array via the intra-group nearest neighbor ports; in the inter-group communication stage, through a distributed router, forward and synchronize the inter-group stage communication message carrying the tensor data between adjacent chip arrays via the boundary extension ports.

[0147] This application provides an electronic device, which includes a processor, wherein the processor includes any of the multi-chip integrated chips or computing storage chips provided in the above embodiments.

[0148] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0149] This application relates to the technical fields of artificial intelligence (AI) acceleration chips, heterogeneous chip integration, three-dimensional stacked storage, package-level interconnects, and distributed storage access, and particularly to an AI acceleration chip system composed of four or more computing storage chips (corresponding to the computing storage chips in the above embodiments). In this multi-chip integrated architecture, each computing storage chip vertically stacks and integrates the AI ​​computing die (corresponding to the computing die in the above embodiments) with a local three-dimensional dynamic random access memory (3D DRAM), and utilizes internal intra-group communication links and external reserved boundary expansion links to jointly construct a decentralized, distributed interconnect network without a central switching die (central I / O Die). Through this architecture design, it is possible to support a smooth expansion from a basic four-chip array unit to a large-scale computing network containing eight, sixteen, or even more computing storage chips through modular splicing, either chip by chip or group by group.

[0150] During the training and inference of large-scale artificial intelligence models (such as large language models), the access volume to data such as model weights, activation function values, key-value caches (KV caches), and parameter gradients has exploded. Traditional single large-area computing dies are severely constrained by wafer manufacturing yield, photomask physical limits, package edge length, and the number of memory interface pins, making it difficult to linearly improve their computing power and memory bandwidth simply by increasing the silicon wafer area.

[0151] Common 2.5D integration solutions typically place a single large-size compute die alongside multiple stacks of high-bandwidth memory (HBM) on the same substrate, with lateral electrical connections between them requiring silicon interposers or local silicon bridge structures. However, this lateral transfer scheme faces significant physical bottlenecks in bandwidth density, data transport energy consumption, and shoreline interface width around the compute die. To overcome this bottleneck, vertically stacking dynamic random access memory directly on top of the compute die can significantly improve the vertical memory access bandwidth of local nodes. However, simply achieving "vertical stacking of 3D DRAM and a single compute die" does not fundamentally solve the interconnection challenges faced by complex arrays containing multiple compute memory chips, such as smooth topology expansion, global memory addressing, and cross-node remote memory access.

[0152] Furthermore, in existing multi-chip interconnect designs, some architectures heavily rely on central switching dies, central switching components, or fully interconnected topologies. In a fully interconnected topology, each compute / storage chip must establish direct point-to-point physical connections (i.e., direct pairwise connections) with all other chips in the network. As the number of compute / storage chips increases, the central switching component quickly encounters new bottlenecks in port count, shoreline cabling width, total internal switching bandwidth, and chip power consumption. The number of physical links required by a fully interconnected topology increases exponentially with the number of nodes, which not only significantly increases the difficulty of underlying physical packaging and substrate routing but also severely hinders modular design and model reuse of chip products.

[0153] Furthermore, regarding storage access mechanisms, explicitly partitioning the independent local storage resources within multiple compute storage granules using pure software significantly increases the programming difficulty of upper-layer application software and the development complexity of compilers. Conversely, if a full hardware cache coherence protocol across granules is enforced at the underlying hardware level to simplify programming, a massive state directory must be maintained in the hardware, introducing a large amount of invalid coherence broadcast traffic into the communication link and severely consuming valuable data transmission bandwidth. Therefore, multi-granule architectures urgently need a non-uniform memory access (NUMA) scheme that can achieve a good balance between unified address space addressing, local memory access performance optimization, and underlying hardware implementation complexity.

[0154] To overcome the aforementioned technical challenges, this application provides a multi-chip integrated chip, which is essentially a scalable AI computing and storage chip matrix array based on a 3D DRAM vertical stacking integration and distributed two-level interconnect architecture. It aims to achieve efficient data computing and physical expansion through a decentralized hardware architecture and hardware-software collaborative storage scheduling.

[0155] This application aims to provide a highly scalable multi-chip integrated architecture. In terms of hardware structure innovation, each compute-storage chip adopts an integrated design, specifically including a compute die, vertically stacked dynamic random access memory (DRAM), a vertical storage interface, a local storage controller, a remote memory access agent, a distributed router, and interconnect ports between chips in a physical direction. Based on this integrated structure, four compute-storage chips can be combined to form a basic chip array arranged in two rows and two columns. Without adding any central switching die, the logical responsibilities of neighboring ports and boundary extension ports within the group are distinguished, and traffic scheduling is uniformly handled by the built-in distributed router. Furthermore, network control functions such as global address decoding, target compute-storage chip selection, local and remote memory access determination, request packet routing, and response packet return are all distributed and integrated into the underlying hardware of each compute-storage chip. This allows multi-chip integrated chips to smoothly scale from a basic four-chip array along one or two package plane dimensions, either chip by chip or group by group, to array sizes of eight, sixteen, or even more chips.

[0156] At the physical packaging and architecture expansion level, the above design brings significant engineering implementation advantages. Firstly, it offers superior packaging feasibility and physical scalability. Because the number of interconnect ports between individual compute memory chips is fixed (e.g., only interconnect pins in the four physical directions of east, west, south, and north are uniformly set, and are fixedly divided into inner-group neighbor ports and outer reserved boundary expansion ports), the total number of communication ports of a single compute memory chip is completely decoupled from the overall number of chips in a multi-chip integrated circuit. This decoupling design of port numbers fundamentally avoids the secondary growth physical routing challenges faced when using a fully interconnected topology. Secondly, it provides efficient design reuse capabilities. Individual compute memory chips of the same specifications and four basic chip arrays with the same structure can be combined and assembled like building blocks into various product units (Stock Keeping Units, SKUs) with different specifications, configurations, such as 4, 8, or 16 chips, simply by dynamically enabling reserved boundary expansion ports at different locations during hardware initialization. Furthermore, when the chip powers on, it generates a routing table by autonomously reading the valid port bitmap and physical coordinates from the probe feedback. The distributed router can flexibly select different shortest forwarding paths and dynamically perform rerouting operations when a specific physical link failure is detected, thereby improving the maintainability and fault tolerance of the underlying hardware topology.

[0157] At the storage access mechanism level, this application constructs a Non-Uniform Memory Access (NUMA) model that balances unified addressing with high local bandwidth performance. On the one hand, by exposing a unified and globally reachable global physical address space to the top-level software, the necessity of explicit data copying between multiple nodes is effectively reduced, significantly improving the friendliness of cross-compute node programming. On the other hand, this NUMA model retains the differences in local data storage capacity, physical transmission distance, and access bandwidth at the hardware level. When processing memory access instructions, any compute storage chip can preferentially access the locally vertically stacked dynamic random access memory using the high bandwidth, low latency, and low power consumption provided by fine-pitch hybrid bonded vertical channels; only when decoding determines a mismatch will cross-node remote data access traffic occupy the two-dimensional plane chip interconnect ports. This mechanism realizes the physical layering of high-frequency local data traffic and cross-group remote data traffic, ensuring the high-efficiency communication bandwidth and excellent energy efficiency of the overall multi-chip architecture.

[0158] Based on the aforementioned innovative hardware interconnect topology and non-uniform memory access mechanism, the embodiments of this application are particularly suitable for the efficient execution of large-scale artificial intelligence computing tasks. In actual deployment of model training and inference tasks, the compiler or underlying runtime software can physically partition the massive artificial intelligence computing data according to different computing strategies in the artificial intelligence algorithm, such as model parallelism, data parallelism, tensor parallelism, or pipeline parallelism. By establishing a precise mapping relationship between the partitioned weight data, activation function operation values, key-value cache data, backpropagation gradient data, and ensemble communication feature data, and the two-dimensional arrangement coordinates of the physical computing storage chips and the specific storage area of ​​the local dynamic random access memory, deep localization of various types of computing data is achieved. This hardware-software collaborative data partitioning and physical mapping execution method maximizes the local throughput advantage of vertically stacked storage media, ensuring that the actual effective computing power of multi-chip integrated circuits can achieve near-linear growth when the node array scale is expanded.

[0159] In some embodiments, see Figure 2The specific structure of a single compute-storage chip (corresponding to compute-storage chip 100 in the above embodiment) is described in detail below. The bottom of the compute-storage chip 100 is fixed above the packaging substrate 160. Each chip includes at least one AI compute die (corresponding to compute die 130 in the above embodiment) and one or more DRAM (Dynamic Random Access Memory) dies (corresponding to DRAM 140 in the above embodiment). The DRAM dies and AI compute dies are stacked vertically, forming high-density vertical signal channels 150 mainly through copper-copper (Cu-Cu) hybrid bonding; optionally, TSV (Through-Silicon Via), microbumps, or a combination of both can also be used to transmit power, ground, clock, and data. In addition, intra-group proximity ports 121 and boundary extension ports 122 are respectively led out on both sides of the AI ​​compute die.

[0160] Specifically, the AI ​​computing die includes a matrix / tensor computing engine (corresponding to the computing engine in the above embodiments), a vector or scalar processing unit, on-chip buffer storage (corresponding to the on-chip buffer memory in the above embodiments), a local DRAM controller (corresponding to the local storage controller in the above embodiments), a remote memory access agent, a D2D (Die-to-Die) link layer (corresponding to the link layer in the above embodiments), and a distributed router.

[0161] During operation, the local DRAM controller translates memory requests (corresponding to storage access requests in the above embodiments) into DRAM channels (corresponding to target storage channels in the above embodiments), banks (corresponding to target memory banks in the above embodiments), and row and column operations (corresponding to operations on target row addresses and target column addresses in the above embodiments). The remote memory access agent adds the opcode, source chipplet ID (corresponding to local chip identifier in the above embodiments), target chipplet ID (corresponding to target chip identifier in the above embodiments), global address (corresponding to target physical address in the above embodiments), length, priority (corresponding to priority field in the above embodiments), and transaction identifier to the request, and restores the original request completion state (corresponding to transaction completion state in the above embodiments) when the response (corresponding to response message in the above embodiments) is returned.

[0162] In some embodiments, see Figure 3 Four identical or compatible chiplets are arranged in a 2×2 configuration to form a basic group (corresponding to the chiplet array 20 in the above embodiment). Figure 3In this diagram, the four chiplets are labeled C0, C1, C2, and C3, respectively. Each chiplet has one or more D2D ports (corresponding to the chiplet interconnection port 120 in the above embodiment) in the north, south, east, and west directions. The D2D ports facing the interior of the basic group are considered as neighboring ports within the group (e.g., ...). Figure 3 The interfaces corresponding to the solid double-headed arrows are shown in the diagram. The nearest neighbor ports within the group primarily transmit remote memory access within the group (corresponding to communication messages in the above embodiments), tensor data, and synchronization messages; the D2D ports facing outwards from the basic group serve as boundary extension ports (such as...). Figure 3 (As shown by the double-headed arrow in the middle dashed line), the boundary extension port is used to connect a new chiplet or another basic group. Both types of ports can share the physical layer specification, but are managed hierarchically in terms of startup configuration, routing priority, and traffic category.

[0163] In some embodiments, the expansion process of a multi-chip integrated circuit is described in conjunction with the expansion topology and startup configuration. For example... Figure 4 and Figure 5 As shown, the basic set can be expanded horizontally into a 2×K array (e.g., Figure 4 (e.g., a 2×4 eight-piece array composed of C0 to C7), or extended into an R×C array along both horizontal and vertical plane dimensions (e.g.) Figure 5 (A 4×4 sixteen-piece array consisting of C0 to C15). Figure 4 and Figure 5 In the diagram, solid double-headed arrows represent connections established within a basic group, while dashed double-headed arrows represent cross-group connections established through the boundary extension ports of adjacent basic groups. For a number N that is not an integer multiple of four, chiplets can be added sequentially from the boundary ports, and unconnected ports are masked after termination detection.

[0164] When the multi-chip integrated chip starts up, each chiplet obtains its own coordinates, chiplet ID (corresponding to the local chiplet identifier in the above embodiments) and valid port bitmap through hardwire, fuse, register or neighbor probe. The local router (corresponding to the distributed router in the above embodiments) (located inside the AI ​​computing die) generates or loads the routing table.

[0165] In some embodiments, the access paths under a unified address space are described by combining global addressing and NUMA (Non-Uniform Memory Access) mechanisms. See also Figure 7 , Figure 7 This is a schematic diagram of local and remote DRAM access paths under a unified address space provided in the embodiments of this application.

[0166] In the local and remote DRAM access paths under a unified address space, the addressing formula is as follows: Figure 7 As shown at the bottom, the following condition must be met: Global Address = ChipletID | Local DRAM Address | Offset. Here, the high-order field of the global address (corresponding to the target physical address in the above embodiment) is the ChipletID (corresponding to the chip identifier field in the above embodiment), the middle field is the local DRAM address inside the Chiplet (corresponding to the chip's internal storage address in the above embodiment), and the low-order field is the specific byte offset; alternatively, a programmable region table can be used to map a continuous range of global addresses to the local DRAM of a specific Chiplet.

[0167] Taking the source Chiplet C0 as an example, the source Chiplet C0 internally houses a computing engine and an address decoder / proxy (corresponding to the local address decoder and remote memory access proxy in the above embodiments). When a global address issued by the computing engine in the source Chiplet C0 enters the address decoder / proxy, the source Chiplet C0 logically compares the target identifier with the local identifier. When the target identifier matches the local identifier, a local hit is triggered, and the C0 local 3D DRAM stacked above is accessed directly via the vertical link; when the target identifier does not match the local identifier, the address decoder / proxy in the source Chiplet C0 encapsulates it into a remote request (corresponding to the communication message in the above embodiments) and sends it to the C0 router (corresponding to the distributed router in the above embodiments).

[0168] The C0 router is responsible for performing XY / adaptive routing. One specific implementation is the XY dimension-order shortest route: first, the X coordinates of the source and destination are compared and the difference is reduced via the east / west ports; then, the Y coordinates are compared and forwarded via the north / south ports. To avoid congestion or failures, the C0 router can also select YX or other shortest / non-shortest alternative paths based on credit (corresponding to link credit in the above embodiment), queue depth (corresponding to the sending queue depth in the above embodiment), and the effective port bitmap.

[0169] During remote forwarding, the sent request is forwarded hop-by-hop through intermediate chipplets and intermediate chipplet routers until it reaches the target chipplet Ck. Inside the target chipplet Ck, its built-in remote memory access agent receives the request and directly accesses its local DRAM. After completing the operation, the target chipplet Ck returns the response along the original path to the intermediate chipplet router and finally back to the C0 router. To ensure efficient and secure network transmission, the requests and responses in the above round trip are isolated using independent virtual channels or class fields to avoid protocol deadlock.

[0170] In some embodiments, the synchronization, sequencing, and data consistency boundary mechanisms are described in conjunction with a multi-chip integrated chip architecture. This application does not specify cross-chip hardware cache consistency as a necessary feature. In practice, each chipplet can manage its local buffer (corresponding to the on-chip buffer memory in the above embodiments) through software-triggered refresh / invalidation instructions (corresponding to refresh instructions and invalidation instructions in the above embodiments), read-write barriers (corresponding to read-write barrier instructions in the above embodiments), events, semaphores, and set communication primitives to establish and synchronously calculate data visibility (corresponding to the data visibility state in the above embodiments).

[0171] In addition, some embodiments may add directory or lease management in the internal remote memory access agent to help maintain consistency, but this does not affect the main topology of the two-level interconnect and extension ports (corresponding to the boundary extension ports in the above embodiments) in this architecture.

[0172] In some embodiments, the deployment and execution process of upper-layer artificial intelligence computing tasks is described in conjunction with the underlying hardware architecture of multi-chip integrated circuits. See also Figure 8 , Figure 8 This is a schematic diagram of the data sharding and execution process for AI training and inference provided in the embodiments of this application.

[0173] When performing data sharding and execution processes for AI (Artificial Intelligence) training and inference, the compiler or runtime environment first performs model / tensor analysis.

[0174] After analysis, the process proceeds to the weighting, activation, KV cache, or gradient sharding stage. During this stage, the compiler or runtime environment flexibly selects data parallelism, tensor parallelism, pipelined parallelism, or hybrid parallelism strategies based on the model layer, tensor size, the local DRAM (dynamic random access memory) capacity of each chipplet (corresponding to the computation storage chip in the above embodiments), and topological distance. The KV cache corresponds to the key-value cache in the above embodiments.

[0175] Subsequently, the topology and capacity are mapped to local DRAM. Specifically, during model inference, weights and KV caches are placed in the local DRAM of the chiplets executing the corresponding attention heads or model layers as much as possible; during model training, model parameters, optimizer states, activation function values, and gradients are physically partitioned and distributed according to the above parallel strategy.

[0176] After the data is mapped and placed, each Chiplet performs GEMM / attention / reduction (where GEMM stands for General Matrix Multiplication).

[0177] During distributed computing, cross-shard aggregation communication and synchronization barriers will be triggered. Cross-shard aggregation communication (such as distributed communication primitives like All-Reduce, All-Gather, Reduce-Scatter, or All-to-All) can be decomposed into intra-group phases and cross-group phases. The intra-group phase corresponds to the intra-group phase communication messages in the above embodiments, and the cross-group phase corresponds to the cross-group phase communication messages in the above embodiments. These two phases utilize two levels of links (i.e., the aforementioned intra-group nearest neighbor ports and boundary extension ports) for hierarchical data transmission and protocol. Finally, after computation and synchronization, the output is sent to the next layer.

[0178] The entire task mapping and execution process described above strictly follows the locality optimization principle: on the one hand, it requires that frequently used weights, activations, KV caches or gradients be placed in the local DRAM of the Chiplet where they are executed in order to maximize the use of the extremely high bandwidth of the vertical signal channel; on the other hand, remote data is accessed through a two-level D2D network only when capacity, parallel sharding or aggregate communication is required.

[0179] In some embodiments, for a four-chiplet base group topology, the four chiplets C0, C1, C2, and C3 are arranged in a 2×2 configuration. C0 is adjacent to C1 and C2, and C3 is adjacent to C1 and C2. Intra-group ports facing each other (corresponding to intra-group nearest neighbor ports in the above embodiments) are connected in pairs. Each chiplet reserves an enableable boundary extension port at at least two outer physical edges. After device startup, these four chiplets obtain two-dimensional coordinates (0,0), (1,0), (0,1), and (1,1), respectively. During addressing, the corresponding local DRAM is selected based on the target field in the global address (corresponding to the target chip identifier in the target physical address in the above embodiments). Under this structure, any source node and target node can reach each other with at most two intra-group forwardings.

[0180] In some embodiments, for an eight-chip extended topology, two four-chip basic groups are horizontally spliced ​​together, forming a 2×4 array by connecting the east boundary port of the left basic group to the west boundary port of the right basic group. During data routing, requests within the same basic group preferentially use intra-group links; when the destination is located in another basic group, the request is routed to the boundary chiplet, enters the other basic group via extended links (corresponding to multiple pairs of parallel boundary extended ports in the above embodiments), and continues to execute the shortest route. If there are multiple pairs of parallel extended links between two groups, data flow can be distributed and scheduled according to the destination row coordinates, traffic hashing mechanism, or the current congestion state.

[0181] In some embodiments, for array structures of sixteen or more chips, the four basic groups are arranged into a 2×2 array, forming a total 4×4 chipplet array. In this large-scale array, each chipplet remains directly connected only to its four adjacent nodes, and its number of physical ports does not increase with the total number of nodes. For embodiments with N>16 nodes, basic groups can be added to the periphery or chipplets can be added piece by piece from the outer edge. Furthermore, since the routing table is generated by calculating logical coordinate ranges rather than enumerating every pair of source and destination nodes in the network, the overhead of hardware routing states increases only linearly or sublinearly with the number of nodes.

[0182] In some embodiments, a complete remote read / write transaction execution process is as follows: First, the computing engine generates a load or store request with a global address. Next, the local address decoder parses the request to obtain the target Chiplet ID; if it is determined to be a local access, the request directly enters the local DRAM controller. If it is determined to be a remote access, the memory access agent assigns a transaction identifier to the request, saves the order attribute, and encapsulates it into a network packet. Subsequently, the distributed router selects the next-hop port based on the target coordinates, port bitmap, and credit score, performing multi-hop forwarding until it reaches the target Chiplet. At the target Chiplet, its internal target memory access agent receives and accesses its local DRAM, then returns the read data or write operation completion status through the response virtual channel. Finally, the source Chiplet matches the received response according to the transaction identifier and, if necessary, updates the visibility status of the computed data based on synchronization barriers or event mechanisms.

[0183] In some embodiments, alternative implementations for packaging and heat dissipation are described. In the main structural embodiment, a fine-pitch Cu-Cu hybrid bonding process is used between the DRAM Die and the AI ​​computing Die. To accommodate different heat dissipation requirements, the AI ​​computing Die can be flexibly placed above or below the DRAM stack, and a thermal diffusion sheet, a back power supply network, or a dedicated thermal channel can be provided above the AI ​​computing Die. Furthermore, the D2D links responsible for inter-die interconnects can be implemented not only by the same hybrid bonding layer, but also by local silicon bridges, passive / active interposers, or organic packaging substrates. Similarly, alternative physical connection methods such as microbumps, TSVs, and interposers can be flexibly combined without changing the underlying logic topology described in the embodiments of this application.

[0184] In summary, through the embodiments of this application, on the one hand, by vertically stacking and integrating computing and storage, and through distributed two-level port routing, the number of chip interconnect ports and the total number of nodes are decoupled, enabling flexible and smooth expansion of the array size without the need for a central switching node, and reducing the complexity of the underlying packaging and wiring. On the other hand, by utilizing a unified physical address space and a non-consistent memory access mechanism, high-frequency local traffic and cross-group remote traffic are physically isolated, avoiding global hardware cache consistency overhead while improving the friendliness of upper-layer software programming. In addition, by combining hardware and software collaborative data sharding and localized mapping schemes, it is highly adapted to the communication characteristics of parallel computing in artificial intelligence, effectively improving the overall effective computing power and storage bandwidth utilization of the multi-chip architecture.

[0185] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A multi-chip integrated circuit, characterized in that, It includes at least one chip array, the chip array including a plurality of compute storage chips, each compute storage chip including a compute die and at least one dynamic random access memory stacked perpendicularly to the compute die; The computing die includes a distributed router and multiple inter-die interconnect ports disposed around the computing die; The multiple interconnect ports between the cores include intra-group nearest neighbor ports and boundary extension ports. The intra-group nearest neighbor ports are used to connect computing and storage cores inside the core array, and the boundary extension ports are used to connect newly added computing and storage cores outside the core array. The distributed router connects to the nearest neighbor port within the group and the boundary extension port respectively. The distributed router is used to determine the next-hop port among the nearest neighbor port within the group and the boundary extension port, and to forward communication packets through the next-hop port.

2. The multi-chip integrated chip according to claim 1, characterized in that, The chip array includes four computing storage chips arranged in two rows and two columns.

3. The multi-chip integrated chip according to claim 1, characterized in that, The multi-chip integrated circuit is configured with a unified global physical address space, and the computing die also includes a computing engine, a local address decoder, a local storage controller, and a remote memory access agent. The computing engine is used to generate storage access requests, wherein the storage access requests include a target physical address in the global physical address space; The local address decoder is used to parse the target physical address in the storage access request, obtain the target die identifier, and compare the target die identifier with the local die identifier of the computing die itself to obtain a comparison result; The local address decoder is also used to distribute the storage access request to the local storage controller or the remote storage access agent based on the comparison result.

4. The multi-chip integrated chip according to claim 3, characterized in that, The local address decoder is used to extract the kernel identifier field from the target physical address to obtain the target kernel identifier; or, The local address decoder is used to query the programmable region table to obtain the target chip identifier mapped by the target physical address, wherein the programmable region table is used to map a continuous segment of the global physical address space to the dynamic random access memory of the computing storage chip.

5. The multi-chip integrated chip according to claim 3, characterized in that, The local storage controller is configured to receive the storage access request when the comparison result indicates that the target kernel identifier matches the local kernel identifier; The local storage controller is used to extract the core internal storage address from the target physical address contained in the received storage access request; The local storage controller is used to parse the internal storage address of the chip to obtain the target storage channel, target storage body, target row address and target column address; The local storage controller is used to perform data read operations or data write operations in the at least one dynamic random access memory via a vertical storage interface at physical locations corresponding to the target storage channel, the target storage volume, the target row address, and the target column address.

6. The multi-chip integrated chip according to claim 3, characterized in that, The remote memory access agent is used to receive the storage access request when the comparison result indicates that the target kernel identifier does not match the local kernel identifier; The remote memory access agent is used to encapsulate the storage access request into the communication message; The remote memory access agent is used to send the communication message to the distributed router.

7. The multi-chip integrated chip according to claim 6, characterized in that, The computing die also includes a link layer; The remote memory access agent is used to assign a transaction identifier to the received storage access request and inject the transaction identifier into the communication message; The link layer is used to encode the communication messages output by the distributed router and send the encoded communication messages to the computing and storage core corresponding to the target core identifier in a serial or parallel manner. The remote memory access agent is used to receive a response message including the transaction identifier, and update the transaction completion status corresponding to the transaction identifier based on the response message.

8. The multi-chip integrated chip according to claim 1, characterized in that, The at least one dynamic random access memory and the computing die are hybrid-bonded to form a vertical signal channel; Electrical signals are transmitted between the computing die and the at least one dynamic random access memory via through-silicon vias or microbumps. The computing die is stacked on top of or below the at least one dynamic random access memory; The computing die has a heat dissipation sheet or a back power supply network disposed on the surface opposite to the dynamic random access memory.

9. The multi-chip integrated chip according to claim 1, characterized in that, When the compute storage die includes an unconnected boundary extension port, the compute storage die is used to perform termination detection on the unconnected boundary extension port; The computing storage chip is used to shut down the drive circuit of the unconnected boundary extension port to perform a shielding operation when the termination detection fails.

10. The multi-chip integrated chip according to claim 1, characterized in that, The computing storage chip is used to send a probe signal during the initialization of the multi-chip integrated circuit. The computational storage core is used to obtain the effective port bitmap of the probe signal feedback; The distributed router is used to generate a routing table based on the effective port bitmap, and to determine the next-hop port from the nearest neighbor ports in the group and the boundary extension ports based on the routing table.

11. The multi-chip integrated chip according to claim 1, characterized in that, The multi-chip integrated chip includes multiple chip arrays, which are spliced ​​together in the horizontal and vertical directions to form a sixteen-chip array, or multiple chip arrays are spliced ​​together in the horizontal direction to form an eight-chip array. In multiple of the aforementioned chip arrays, the distributed router is used to preferentially use the nearest neighbor ports within the group to forward the communication packets; The distributed router is used to obtain the congestion status of the border extension port and distribute the communication packets in multiple pairs of parallel border extension ports based on the congestion status.

12. The multi-chip integrated chip according to claim 1, characterized in that, The distributed router is used for: Obtain the target coordinates of the target computing storage core pointed to by the communication message and the local coordinates of the computing storage core itself; The horizontal dimension coordinates of the target coordinates are compared with the horizontal dimension coordinates of the local coordinates to obtain the horizontal comparison result; When the lateral comparison result indicates that the lateral dimension coordinates are inconsistent, the next-hop port is determined in the inter-core interconnection port in the lateral interconnection direction, and the communication message is forwarded through the next-hop port to reduce the lateral coordinate difference. The vertical dimension coordinates of the target coordinates are compared with the vertical dimension coordinates of the local coordinates to obtain the vertical comparison result; When the longitudinal comparison result indicates that the longitudinal dimension coordinates are inconsistent, the next-hop port is determined in the inter-core interconnect port in the longitudinal interconnection direction, and the communication message is forwarded through the next-hop port.

13. The multi-chip integrated chip according to claim 1, characterized in that, The distributed router is used to allocate mutually isolated independent virtual channels for the request messages and response messages included in the communication messages.

14. The multi-chip integrated chip according to claim 1, characterized in that, The communication message includes a priority field, which is used to indicate the priority level of the communication message; When the communication message carries key-value cache or gradient synchronization data, the priority field is configured as the first priority; The distributed router is used to parse the priority field and, in the nearest port within the group or the boundary extension port, prioritize forwarding the communication packets whose priority field is the first priority. The distributed router is configured with multiple alternative forwarding paths to the target computing and storage chip, and each of the multiple alternative forwarding paths corresponds to a different interconnection port between chips. The distributed router is also used to obtain the link credit or transmission queue depth of the inter-core interconnect ports corresponding to the multiple alternative forwarding paths, and to determine the next-hop port among the inter-core interconnect ports corresponding to the multiple alternative forwarding paths based on the link credit or the transmission queue depth.

15. The multi-chip integrated chip according to claim 1, characterized in that, The computing die also includes an on-chip buffer memory, which is used to temporarily store computing data; The computing die is used to respond to refresh instructions, invalidation instructions, or read / write barrier instructions to perform a data write operation to the at least one dynamic random access memory on the computing data in the on-chip buffer memory, or to perform invalidation processing on the computing data in the on-chip buffer memory. The computing die is used to synchronize the data visibility status of the computing data with other computing storage dies through the data write operation or the invalidation process.

16. The multi-chip integrated chip according to claim 1, characterized in that, The at least one dynamic random access memory is used to store tensor data that has been physically fragmented; When the computing die executes cross-die set communication instructions, the communication message includes intra-group phase communication messages and cross-group phase communication messages; The distributed router is used to forward the intra-group phase communication message carrying the tensor data through the intra-group nearest neighbor port. The distributed router is used to forward the cross-group phase communication message carrying the tensor data through the boundary extension port.

17. A computing storage chip, characterized in that, The computing storage die includes a computing die and at least one dynamic random access memory stacked perpendicularly to the computing die; The computing die includes a distributed router and multiple inter-die interconnect ports disposed around the computing die; The computing storage core is used to send a probe signal through the interconnect ports between the multiple cores during initialization, and to generate a valid port bitmap based on the feedback result of the probe signal; The compute storage kernel is used to dynamically divide the interconnect ports between the multiple kernels into intra-group neighbor ports and boundary extension ports based on the effective port bitmap. The intra-group neighbor ports are used to communicate with other compute storage kernels belonging to the same kernel array, and the boundary extension ports are used to connect compute storage kernels outside the kernel array across the array. The distributed router is used to determine the next-hop port among the nearest ports within the group and the boundary extension ports, and to forward communication packets through the next-hop port.

18. A distributed routing method, characterized in that, The method is applied to a multi-chip integrated circuit, the multi-chip integrated circuit including at least one chip array, the chip array including a plurality of compute storage chips, each compute storage chip including a distributed router and a plurality of inter-chip interconnect ports disposed around the compute die of the compute storage chip, the method comprising: The detection signal is transmitted via the computing storage core through the interconnect ports between the multiple cores; The effective port bitmap is generated based on the feedback result of the probe signal through the computational storage core. The computational storage core is used to divide the interconnect ports between the multiple cores into intra-group neighbor ports for connecting the core array and boundary extension ports for connecting the core array to the outside of the core array based on the effective port bitmap. The next-hop port is determined from the nearest neighbor ports within the group and the boundary extension ports using the distributed router. The distributed router forwards communication messages via the next-hop port.

19. An electronic device, characterized in that, The electronic device includes a processor, wherein the processor includes: a multi-chip integrated circuit as described in any one of claims 1 to 16, or a computing storage chip as described in claim 17.

Images