Three-dimensional stacked chip system, memory access method, and electronic device

Abstract

Embodiments of the present application provide a three-dimensional stacked chip system, a memory access method and an electronic device. The chip system comprises: a substrate; one or more computing dies arranged on the substrate; one or more memory dies, each of which is arranged above a corresponding computing die by a through-silicon via stack, and the computing dies and the memory dies are packaged as a first chip; and one or more input / output dies arranged on the substrate on a side of the computing dies and connected to each of the computing dies through substrate wiring, the input / output dies are packaged as a second chip, and there is no direct communication path between the input / output dies and the one or more memory dies. The above technical solution can reduce the transmission delay between the computing dies and the memory dies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuits, and in particular, to a three-dimensionally stacked chip system, a memory access method and an electronic device. BACKGROUND

[0002] With the development of integrated circuit manufacturing processes, the production cost of chips is becoming higher and higher. Under the same process, the more functions a chip needs to implement, the higher the design complexity will be, which will lead to a lower chip yield. Moreover, in order to support the functions to be implemented, the area of the chip will become larger. Under the constraints of chip yield and chip area, the manufacturing cost of the chip is high.

[0003] In order to reduce the manufacturing cost of the chip, chiplet technology has emerged. Among them, chiplet technology is a design method that splits a complex chip into multiple small, independent and reusable modules. These modules can be processor cores, memory chips, sensors or other types of integrated circuits, connected through high-speed interfaces or advanced packaging technologies to form a complete system chip. This method effectively addresses the challenges of chip design complexity and manufacturing cost.

[0004] However, the existing chiplet technology still needs to be improved.

[0005] The above information disclosed in the background art is only used to strengthen the understanding of the background of the present application, and therefore it can contain information that is not prior art known to those skilled in the art. SUMMARY

[0006] The embodiments of the present application provide a three-dimensionally stacked chip system, a memory access method and an electronic device, which can reduce the transmission delay between the computing chiplets and the memory chiplets.

[0007] According to a first aspect of the embodiments of the present application, a three-dimensionally stacked chip system is provided, comprising:

[0008] a substrate;

[0009] one or more computing chiplets disposed on the substrate;

[0010] one or more memory chiplets, each of the memory chiplets being disposed above a corresponding computing chiplet by a through-silicon via stack, and the computing chiplets and the memory chiplets being packaged as a first chip;

[0011] one or more input / output chiplets disposed on the substrate on a side of the computing chiplets and connected to each of the computing chiplets through substrate routing, the input / output chiplets being packaged as a second chip and having no direct communication path with the one or more memory chiplets.

[0012] Optionally, the compute die comprises one or more of: a central processing unit, a graphics processing unit, and a neural processing unit.

[0013] The memory die comprises: a high bandwidth memory die and a double data rate memory die.

[0014] Optionally, the chip system further comprises: solder balls between the compute die and the substrate, and between the input / output die and the substrate.

[0015] The solder balls are used to connect the compute die and the substrate, and the input / output die and the substrate, and the substrate routing passes through the solder balls to connect the input / output die and the compute die.

[0016] Optionally, the compute die further transmits a first semantic tag carried by a memory access request through a multiplexed through-silicon via channel, the first semantic tag is parsed and executed by a memory controller, the memory controller is located in the first chip and connected with the one or more memory dies.

[0017] Wherein, the first semantic tag is generated when a destination address of the memory access request points to a first storage space, and the first storage space is a location of the one or more memory dies.

[0018] Optionally, the input / output die comprises: a routing control unit configured to perform the following operations in response to a second semantic tag carried by a memory access request of the compute die: in a write merge buffer mode, aggregate a plurality of write requests and send them to the memory controller; in a prefetch buffer mode, based on an access pattern prompt, prefetch data to a cache;

[0019] Wherein, the compute die sends a memory access request carrying a second semantic tag to the routing control unit through a substrate semantic channel, and the second semantic tag is generated when a destination address of the memory access request points to a second storage space, which can be different from the first storage space.

[0020] Optionally, the second semantic tag comprises: an operation mode identifier and parameter domain information.

[0021] The routing control unit comprises:

[0022] A semantic decoding module configured to receive a memory access request from the compute die and parse the second semantic tag carried thereby, output a mode identifier and parameter domain, the mode identifier comprising: a write merge mode and a prefetch mode.

[0023] An address matching module coupled with the semantic decoding module is configured to determine whether the target address of the memory access request falls into an accessible storage space according to the target address of the memory access request and a configurable storage space range in a programmable register group, and output a hit signal;

[0024] A feature analysis module coupled with the semantic decoding module is configured to generate a suggestion signal based on a data packet size, a request type, and a historical access trajectory;

[0025] An arbitration module coupled with the semantic decoding module, the address matching module, and the feature analysis module is configured to, in response to the mode identifier being valid, adopt a processing mode of the memory access request corresponding to the mode identifier according to a first priority, in response to a first storage space hit signal output by the address matching module, adopt the pre-fetch mode according to a second priority, in response to a second storage space hit signal output by the address matching module, adopt the write-merge mode according to a third priority, and in response to the suggestion signal, adopt a suggestion mode of the suggestion signal according to a fourth priority.

[0026] Optionally, the operation of the write-merge buffer mode includes collecting unaligned write requests within a time window and merging the write requests into a burst write operation after aligning with a DRAM page boundary.

[0027] The operation of the pre-fetch buffer mode includes calculating a pre-fetch step according to an access mode prompt and storing a subsequent data block in the cache after continuous reading by the step.

[0028] Optionally, the chip system further includes a programmable interconnection layer arranged between the computing chiplets and the memory chiplets having a corresponding relationship, and the programmable interconnection layer includes a dynamic memory pooling module configured to manage heterogeneous memory resources, and the heterogeneous memory resources are composed of a plurality of the memory chiplets.

[0029] The dynamic memory pooling module is configured to generate a uniform resource vector based on physical topology location attributes, bandwidth, delay, durability and energy consumption characteristics of the heterogeneous memory resources; allocate, release and address map the uniform resource vector with a basic unit of cache behavior; perform topology-aware allocation according to the relative position relationship between the computing core and the memory core, and preferentially allocate frequently accessed data to a physical memory region with the lowest access delay; divide the physical memory region into time slices and time-multiplex between different tasks, and the time-multiplexing process is coordinated with the hardware context switching operation; monitor the energy consumption state, and migrate cold data to a low-power memory region or place idle memory cores into a low-power state according to a predefined energy efficiency strategy; perform data striping and check computation inside the memory pooling layer, perform online data recovery using check information and redundant data in response to detection of a memory core failure, and mark and isolate the failed memory core.

[0030] According to a second aspect of the embodiments of the present application, a memory scheduling method is provided, applied to the three-dimensionally stacked chip system of any one of the preceding aspects, and the memory scheduling method comprises:

[0031] In response to a memory access request sent by the computing core, a parsing operation is performed;

[0032] In response to a parsing result of the parsing operation, the following operations are performed: in a write-merge buffer mode, a plurality of write requests are aggregated and sent to a memory controller, and the memory controller is located in the first chip; and in a prefetch buffer mode, data is prefetched to a cache based on an access mode.

[0033] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising the three-dimensionally stacked chip system of any one of the preceding aspects.

[0034] The embodiments of the present application have the following technical effects due to the use of the above technical solutions:

[0035] One or more computing cores, one or more memory cores and one or more input / output cores are arranged on a substrate, each memory core is arranged above the corresponding computing core by a through-silicon via and is packaged as a first chip, and the input / output core is packaged as a second chip and has no direct communication path with the one or more memory cores. In this way, compared with the existing scheme in which the input / output core and the memory core are packaged together, the access path between the computing core and the memory core in the present application is shortened, thereby reducing the transmission delay between the computing core and the memory core. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0037] Figure 1 is a schematic diagram of a packaging structure of a compute chiplet and an input / output chiplet;

[0038] Figure 2 is a schematic diagram of a three-dimensional stacked chip system in an embodiment of the application;

[0039] Figure 3 is a schematic diagram of a routing control unit in an embodiment of the application. DETAILED DESCRIPTION

[0040] In order to make the technical solutions and advantages of the embodiments of the application clearer, the following further describes the exemplary embodiments of the application with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the application, and are not exhaustive of all embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0041] As in the background art, chiplet technology effectively addresses the challenges of chip design complexity and manufacturing cost. In the current scheme, chiplets are divided into compute chiplets and input / output chiplets according to different functions.

[0042] Among them, the compute chiplet includes various computing units such as central processing units (CPUs), graphics processing units (GPUs), and neural network processing units (NPUs).

[0043] The input / output chiplet includes peripheral component interconnect express (PCIE), universal serial bus (USB), double data rate (DDR), high bandwidth memory (HBM), etc.

[0044] The compute chiplet and the input / output chiplet are manufactured under different processes, and are packaged together after production. This can greatly reduce manufacturing costs and iteration time.

[0045] Referring to Figure 1 is a schematic diagram of a packaging structure of a compute chiplet and an input / output chiplet, as shown in Figure 1As shown, after the computing chiplet 12 and the input / output chiplet 10 are manufactured respectively, the communication between the computing chiplet 12 and the input / output chiplet 10 is realized through the interconnection line. In this way, the computing chiplet 12 can access the memory chiplet 14, such as DDR and HBM, through the input / output chiplet 10.

[0046] However, because of the protocol conversion, additional delay will be caused in the cross-chiplet transmission, and the bandwidth of the interconnection between the chips is significantly lower than the bandwidth within a single chip. For those computing units sensitive to data delay, the impact of the bandwidth and delay cannot be ignored.

[0047] And the memory chiplet 14 is packaged together with the input / output chiplet 10, so that the memory chiplet 14 has a long access path with the computing chiplet 12.

[0048] To solve the above technical problems, the embodiment of the present application provides a chiplet design scheme based on three-dimensional (3D) stacking, which stacks a memory chiplet on a computing chiplet sensitive to delay. By stacking the memory chiplet, the computing unit in the computing chiplet can directly access the memory chiplet within the packaging structure of the computing chiplet, and the access path between the computing chiplet and the memory chiplet is shortened.

[0049] And because the protocol conversion of the memory chiplet is not involved, the delay of the computing unit in the computing chiplet accessing the memory can be effectively reduced, and thus the transmission delay between the computing chiplet and the memory chiplet can be reduced.

[0050] In addition, the present application can realize a bus width of more than 1024 bits through the through-silicon via (TSV) technology. Compared with the universal chiplet interconnect express (UCIE), the bandwidth can be greatly improved.

[0051] In other words, the embodiment of the present application simultaneously realizes the reduction of the transmission delay of memory access and the improvement of the bandwidth.

[0052] In order for those skilled in the art to better understand and implement the present application, the specific schemes, principles, advantages and effects of the present application will be described in detail below with reference to the accompanying drawings.

[0053] Referring to Figure 2 As shown in the structure schematic diagram of the three-dimensional stacked chip system in the embodiment of the present application, the three-dimensional stacked chip system 200 can include: Figure 2 As shown, the three-dimensional stacked chip system 200 can include:

[0054] a substrate 202;

[0055] one or more computing chiplets 204 disposed on the substrate 202;

[0056] one or more memory dies 206, each memory die 206 is stacked above a corresponding compute die 204 via through-silicon via 208, and the compute die 204 and the memory die 206 are packaged as a first chip 210;

[0057] one or more input / output dies 212, disposed on the substrate 202 at a side of the compute dies 204 and connected with each compute die 204 via substrate routing, the input / output dies 212 are packaged as a second chip 220, and have no direct communication path with the one or more memory dies.

[0058] In some embodiments, the substrate 202 provides a process platform for the formation of the chip system 200.

[0059] Specifically, the substrate 202 can serve as a bridge between the dies and the circuit board, made of organic material, ceramic or silicon, etc., with a wiring layer on it to provide electrical connection, mechanical support and heat dissipation channels.

[0060] Exemplarily, in advanced packaging schemes, the substrate 202 is used to connect the chips (or dies) and the external system.

[0061] The compute dies 204 serve to be responsible for processing and operation. Similar to the CPU or GPU core in traditional schemes.

[0062] The memory dies 206 provide storage and cache for storing data. Similar to on-chip memory or externally expanded memory.

[0063] The input / output dies 212 are used to process external interfaces and signal transmission.

[0064] In short, the compute dies 204 are the “brain”, the memory dies 206 are the “memory”, and the input / output dies 212 are the “neural network interface”. The three work together to realize the data processing process.

[0065] In some embodiments, the memory dies 206 are stacked above the corresponding compute dies 204 via through-silicon via 208. In this way, the compute dies 204 can directly access the memory dies 206, reducing transmission latency.

[0066] And, by packaging the compute dies 204 and the memory dies 206 as the first chip 210, the interaction between the compute dies 204 and the memory dies 206 is in the same package, further reducing transmission latency.

[0067] In some embodiments, the through-silicon via interconnection is a vertical straight connection, with a very short path and significantly reduced signal delay.

[0068] In some embodiments, the input / output core 212 is connected to the computing core 204 through the substrate wiring, is packaged as a second chip 220 alone, and has no direct communication path with the one or more memory cores 206, so that the computing core 204 can directly access the memory core 206 without the input / output core 212, thereby reducing the transmission delay.

[0069] In other words, the present application binds the computing core 204 and the memory core 206 together through 3D stacking, thereby effectively reducing the transmission delay of the computing unit in the computing core 204 when accessing the memory.

[0070] It should be noted that, first, Figure 2 The packaging structure of the schematic chip system 200 is only an example for illustrating that the computing core 204 and the memory core 206 are packaged together, and is an abstract expression, and cannot be understood as a limitation on the present application; second, Figure 2 The number and position of the schematic computing core 204, the memory core 206, the through-silicon via 208, and the input / output core 212 are only examples for illustrating the components of the chip system 200; third, the focus of the present application is on the structural change, and how to form the computing core, the memory core, the through-silicon via, and the input / output core is not mentioned, which can be referred to the existing examples.

[0071] For example, the through-silicon via can be formed in the following manner: an interlayer dielectric layer is formed on the computing core; a via is formed in the interlayer dielectric layer, and a metal interconnection structure is formed in the via.

[0072] In some embodiments, the computing core can be used as a processing core to control different processes.

[0073] For example, the computing core can include one or more of a central processing unit, a graphics processing unit, and a neural network processing unit.

[0074] The use of the stacked computing core and the memory core in the widely used processing can reduce the delay and meet most of the use scenarios.

[0075] In some embodiments, the memory core can include a high-bandwidth memory core and a double data rate memory core. The high-bandwidth memory prioritizes bandwidth and is suitable for compute-intensive applications, and can reduce the signal delay in AI model training scenarios; the double data rate memory prioritizes capacity and cost, and is suitable for large-scale general-purpose computing.

[0076] For example, the double data rate memory can include different types of memory such as DDR4, DDR5, etc.

[0077] In some embodiments, referring next to Figure 2To realize the interconnection between different corelets, the three-dimensionally stacked chip system 200 further comprises: solder balls 230 between the computing corelet 204 and the substrate 202, and between the input / output corelet 212 and the substrate 202.

[0078] The solder balls 230 are used to connect the computing corelet 204 and the substrate 202, and the input / output corelet 212 and the substrate 202, and the substrate wiring connects the input / output corelet 212 and the computing corelet 204 through the solder balls 230.

[0079] In other words, the interconnection between the internal components of the three-dimensionally stacked chip system 200 is realized through the solder balls 230.

[0080] In addition, the solder balls can provide a stable conductive path to ensure the transmission of high-speed signals, clocks, and power. Compared with traditional wire bonding, the solder ball connection path is shorter, and the signal integrity is better.

[0081] It should be noted that the solder ball connection is only one example. In other embodiments, one or more of through-silicon vias, hybrid bonding, and pin / socket connections (for example, upper and lower structure interconnection through ball grid array) can be used to realize the connection between different structures.

[0082] In some embodiments, a memory corelet is bonded above a latency-sensitive computing corelet, and the computing corelet directly accesses the memory corelet through a through-silicon via.

[0083] As the complexity of the system increases, it is necessary to integrate multiple types of memory and intelligently manage memory access (such as write merging and prefetching) to further improve efficiency.

[0084] However, the current direct access path solution cannot realize such intelligent management, especially when other computing corelets access memory corelets through input / output corelets.

[0085] To further overcome the limitations of different access mechanisms, the present application further limits a solution that can support heterogeneous memory access and intelligent memory management, i.e., by introducing a dual-path memory access architecture to achieve the above purpose.

[0086] The dual-path memory access path includes:

[0087] The first access path, i.e., for access requests for stacked memory (corresponding to the first storage space), the computing corelet directly sends to the memory controller through a through-silicon via channel, ensuring ultra-low latency access of core computing to high-speed memory.

[0088] The second access path, i.e. the access request for the on-board memory (corresponding to the second storage space), is sent by the computing core to the input-output core by the computing core. Through the routing control unit built in the input-output core, the efficiency and bandwidth utilization of accessing the on-board memory are improved by analyzing the semantic label in the request.

[0089] In short, in the scheme in which the computing core and the memory core are packaged together, the memory core is directly accessed; and in the scheme in which the memory core and the input-output core are packaged together, the memory core is accessed through the input-output core.

[0090] Specifically, there are two different forms of memory cores in the present application. In order to adapt to this change, the input-output core and the working logic of the computing core can also be configured so that different access strategies are executed in response to the memory access request of the computing core.

[0091] Through this differentiated setting mode, different configuration options are provided to meet the needs of different users and cover a wide range of use scenarios.

[0092] Exemplarily, the computing core is configured to generate different semantic labels for memory access requests to different destination addresses and transmit them through the substrate semantic channel or the through-silicon via channel respectively.

[0093] In some embodiments, the computing core also transmits the first semantic label carried by the memory access request through the multiplexed through-silicon via channel, and the first semantic label is parsed and executed by a memory controller located in the first chip and connected with one or more memory cores.

[0094] Wherein, when the destination address of the memory access request points to the first storage space, the first semantic label is generated, and the first storage space is the location of one or more memory cores.

[0095] In other words, when it is determined to access the memory core inside itself, the data processing operation is directly performed in the traditional manner of memory controller-memory core.

[0096] The multiplexed through-silicon via channel refers to a communication path based on through-silicon via, which is used to transmit data between cores or between cores and memory controllers.

[0097] Wherein, the first semantic label is transmitted to the memory controller through the multiplexed through-silicon via channel, and the corresponding operation (such as triggering prefetch, write merging or other optimization) is parsed and executed by the memory controller.

[0098] Adaptively, the input-output corelet comprises a routing control unit configured to, in response to a second semantic tag carried by a memory access request of the compute corelet, perform the following operation: in a write-combining buffer mode, aggregate multiple write requests and send to a memory controller; in a prefetch buffer mode, based on an access pattern, prompt prefetching of data to a cache; wherein the compute corelet sends the memory access request carrying the second semantic tag to the routing control unit through a substrate semantic channel.

[0099] In some embodiments, in a memory access request initiated by the compute corelet, indication information of the memory to be accessed can be carried. That is, through the second semantic tag, the memory corelet to be accessed by the compute corelet is indicated. For example: the type, address, and other information of the memory corelet, etc.

[0100] In this case, the routing control unit can analyze the second semantic tag, and based on the final analysis result, perform different access paths.

[0101] Exemplarily, in one case, in the write-combining buffer mode, multiple write requests are aggregated and sent to the memory controller.

[0102] Among them, in the computer system, write combining refers to that when the memory needs to write data, instead of sending a single write request to the memory immediately each time, multiple small write requests are temporarily stored in a buffer, and then sent after being combined into one or several larger requests.

[0103] Compared with the direct write mode, the write-combining mode reduces the number of memory access times and reduces the bandwidth pressure by combining requests.

[0104] In some embodiments, the buffer is a temporary storage area for collecting and organizing multiple write requests. Among them, the physical location of the buffer can be the memory corelet, the inside of the compute corelet, etc.

[0105] The memory controller is a hardware component responsible for managing data transmission between the memory corelet and the compute corelet. It handles read and write requests, controls the timing, refresh, and data transmission of the memory. The memory controller can be located inside the first chip. In this way, in the write-combining buffer mode, after multiple write requests are aggregated, the buffer sends the combined data as a whole to the memory controller. The memory controller then writes these data to the memory corelet.

[0106] Specifically, the operation of the write-combining buffer mode includes: collecting unaligned write requests within a time window, and combining them into burst write operations after aligning with the DRAM page boundary.

[0107] The time window refers to a specific time range during which the write coalescing buffer collects write requests from the processor, rather than sending each request to the memory controller immediately.

[0108] Misalignment refers to the target address of a write request not fully meeting certain boundary requirements (e.g., cache line boundary or DRAM page boundary) of the memory.

[0109] For example, a typical cache line size is 64 bytes. If the target address of a write request is not on the 64-byte aligned boundary, it is called misaligned.

[0110] In DRAM, misalignment can refer to the address of a write request not being aligned with the page boundary of the DRAM.

[0111] Collecting misaligned write requests means that the buffer temporarily stores these misaligned requests, rather than sending them directly to the memory controller.

[0112] Further, the DRAM page boundary is a contiguous memory unit, usually 4KB, 8KB or larger in size. Page boundary alignment refers to ensuring that the starting address of a write request is aligned with the boundary of a DRAM page.

[0113] In the buffer, the collected misaligned write requests are reorganized so that their addresses are aligned with the DRAM page boundary. This alignment can involve: address remapping, i.e., adjusting the address of the request so that the data is written to an aligned DRAM page; data reorganization, i.e., rearranging multiple misaligned requests to fill into the same DRAM page.

[0114] A burst write refers to an operation in which multiple data blocks are transmitted continuously in one memory transaction, thus efficiently utilizing the bandwidth to transmit multiple data blocks at once and reducing control overhead on the memory bus.

[0115] In another case, in the prefetch buffer mode, data is prefetched to the cache based on access pattern hints.

[0116] The cache is a fast storage unit (such as L1, L2, L3 cache) located between the processing core and the memory core, used to store frequently accessed data.

[0117] The write coalescing buffer mode refers to a specific working state in which the write coalescing mechanism is enabled. In this mechanism, a special prefetch mechanism is enabled to determine the memory addresses that may be needed based on access pattern hints. According to these hints, data is prefetched from the main memory and loaded into the cache (or prefetch buffer) so that the computing chip can access it quickly, reducing memory access latency.

[0118] Specifically, the operation of the prefetch buffer mode includes: calculating a prefetch step according to the access pattern hints, and continuously reading and storing subsequent data blocks in the cache according to the step.

[0119] By way of example, the access pattern hints are used to guide the prefetch mechanism to predict the memory addresses that the processor is likely to access in the future. These hints can come from: analyzing the past memory access behavior of the computing chip to identify regular patterns, such as sequential access, stride access, and pointer tracking; or the compiler or metadata explicitly specifying the addresses that need to be prefetched. These hints help the prefetch mechanism to more accurately predict future memory access addresses.

[0120] The prefetch step refers to the address interval used by the prefetch mechanism when predicting subsequent data addresses. The step reflects the regularity of memory access, such as: sequential access: the step is 1 cache line (usually 64 bytes), i.e. continuously loading subsequent memory blocks; stride access: the step is a fixed size, such as loading a data block every 256 bytes (commonly used in matrix or multi-dimensional array access); dynamic step: the step can be dynamically adjusted according to the runtime access pattern.

[0121] Once the prefetch step is determined, the prefetch mechanism will predict the subsequent memory addresses according to this step, and continuously load multiple data blocks according to the step. This takes advantage of the burst transfer characteristics of DRAM, efficiently reading data from the main memory, and implementing the storage of the read data blocks to the cache or prefetch buffer, so that the processor directly hits the cache when requesting in the future, avoiding accessing the slower DRAM.

[0122] The computing core sends a memory access request carrying a first semantic tag to the routing control unit through the substrate semantic channel, which means that the computing core generates a memory request carrying a semantic tag, sends it to the routing control unit through the semantic channel on the substrate, and the routing control unit optimizes the routing according to the tag and the address.

[0123] This improves the intelligence and efficiency of traditional memory access by introducing semantic information, reduces latency and power consumption. It is particularly suitable for high-performance computing and AI scenarios, and is closely integrated with historical memory optimization modes such as write merging and prefetching to form a complete memory management system.

[0124] In some embodiments, when the destination address of the memory access request points to a second storage space, a second semantic tag is generated, and the second storage space can be different from the first storage space.

[0125] In other words, the computing core generates a memory access request and selectively attaches a first semantic tag or a second semantic tag according to the destination address, i.e. if the destination address points to the first storage space, the second semantic tag is generated; if the destination address points to the second storage space, the first semantic tag is generated.

[0126] It should be noted that the reason why the second storage space is able to be different from the first storage space is that some unexpected situations are taken into account, for example, the access request signal sent by another computing core carries information indicating the first storage space, and the first storage space is the address of the memory core specific to another computing core. Thus, when the second storage space is able to be the same as the first storage space, inter-core interaction is realized.

[0127] In this way, by generating different semantic tags according to the destination address, the system realizes adaptive optimization based on the storage space, enhances flexibility and efficiency, and adapts to the needs of heterogeneous storage in a multi-core system.

[0128] In short, for the structure of multi-core heterogeneous storage, for a memory access request sent to the second storage space, the computing core sends a memory access request carrying a second semantic tag to the routing control unit through the substrate semantic channel. For a memory access request sent to the first storage space, the computing core sends a memory access request carrying a first semantic tag to the memory controller through the multiplexed through-silicon via channel.

[0129] In some embodiments, the second semantic tag includes an operation mode identifier and parameter field information. The operation mode identifier is used to indicate the processing mode of the memory access request, and the parameter field information is used to indicate the behavior of the processing mode.

[0130] Specifically, the operation mode identifier is a field in the semantic tag, which is used to indicate which working mode the dynamic routing control unit should use to process the current request.

[0131] For example: 000 represents a normal pass-through mode, 001 represents a write-merge mode, 010 represents a prefetch mode, and 011 represents a cache direct-write mode. It should be noted that more modes can be extended according to actual needs, such as error correction mode, delay shielding mode, etc., and the above characters are only examples.

[0132] Thus, when it is detected that the field is valid, the corresponding mode is directly used as the highest priority input.

[0133] The parameter field is a control parameter in the semantic tag matched with the mode identifier, which is used to refine the behavior of the execution mode, and at least includes: a time window size used to determine the aggregation range of write-merge; a prefetch step (4-256 bytes) used to determine the granularity of continuous data reading; a buffer depth limit, and a trigger condition, such as an access frequency threshold.

[0134] Correspondingly, referring to the structure diagram of a routing control unit in an embodiment of the present application shown in Figure 3 As shown in the structure diagram of a routing control unit in an embodiment of the present application shown in Figure 3 The routing control unit 300 can include:

[0135] The semantic decoding module 302 is configured to receive the memory access request from the computing core 204, and parse the second semantic tag carried thereby to output a mode identifier and a parameter field, the mode identifier including a write merge mode and a prefetch mode.

[0136] The address matching module 304 is coupled to the semantic decoding module 302 and is configured to determine whether the target address of the memory access request falls within the accessible storage space according to the target address of the memory access request and the configurable storage space range in the programmable register set, and output a hit signal.

[0137] The feature analysis module 306 is coupled to the semantic decoding module 302 and is configured to generate a suggestion signal based on the packet size, the request type, and the historical access trajectory.

[0138] The arbitration module 308 is coupled to the semantic decoding module 302, the address matching module 304, and the feature analysis module 306, respectively, and is configured to, in response to the mode identifier being valid, adopt the processing mode of the memory access request corresponding to the mode identifier according to a first priority, in response to the first storage space hit signal output by the address matching module, adopt the prefetch mode according to a second priority, in response to the second storage space hit signal output thereby, adopt the write merge mode according to a third priority, and in response to the suggestion signal, adopt the suggested mode of the suggestion signal according to a fourth priority.

[0139] Specifically, the second semantic tag carried in the memory access request output by the computing core 204 has a large amount of useful information, and by performing a parsing operation, the data used by the address matching module 304 can be obtained.

[0140] The target address of the memory access request can be understood as the memory core to be accessed.

[0141] The programmable register set is written by the initialization software / firmware or the chip system configuration logic at startup. In design, the programmable register is part of the dynamic routing control unit and can be configured by external software through register mapping (MMIO).

[0142] The programmable register can include the start address and end address of the first storage space, the start address and end address of the second storage space, and possibly the address range of other memories or peripherals.

[0143] The first storage space can be different from the second storage space. For example, the first storage space can be a memory core in the first chip, and the second storage space can be a memory core in the second chip or other memory core.

[0144] Thus, the address matching module 304 can determine the address range of the memory to be read according to the target address of the memory access request and the storage space range, and output a hit signal.

[0145] It can be understood that a part of the first storage space and the second storage space can be the same, that is, there is an overlapping space between the two.

[0146] In some embodiments, the request type can be obtained from the memory access request sent by the computing core. Each memory access request contains its type information, which is usually generated by the memory access control logic (such as load, store, write, or read, etc.) and embedded in the request.

[0147] In the routing control unit 300, the semantic decoding module 302 parses the second semantic tag, extracts and identifies the request type.

[0148] The packet size is usually defined in the memory access request. Each memory request not only carries an address, but also specifies the size of the data block requested. The size of the data packet can also be dynamically determined by the computing core or the memory controller according to the needs of the memory operation.

[0149] The historical access trajectory can be obtained from the access history buffer of the computing core or the memory controller. The dynamic routing control unit can query these historical records for analyzing access patterns and optimizing decisions.

[0150] For example, if a certain address is frequently accessed, it can be considered as a strong locality access pattern, resulting in a prefetch request or enabling a write merge strategy.

[0151] In this case, the feature analysis module 306 is no longer simply responding to each request, but can actively analyze traffic features, predict future behavior, and make reasonable recommendations.

[0152] For example, the computing core sends a write request to write 128 bytes of data to a non-aligned address. The feature analysis module 306 analyzes the packet size and finds that it is 128 bytes, which is a relatively small write operation and has the value of merging.

[0153] Since it is a write request, it is suitable for write merge mode, and by analyzing the historical access trajectory, it is determined that there is no other write request near this address and its vicinity recently, and a merge queue cannot be formed, so the urgency of this merge is not the highest. In this way, after comprehensive judgment, a suggestion signal is generated, the content of which can be: "suggest using write merge mode for this request".

[0154] When the above information all reaches the arbitration module 308, the arbitration module 308 can point to the following operations based on the output results:

[0155] In response to the mode identifier being valid, a processing mode corresponding to the memory access request of the mode identifier is adopted according to a first priority.

[0156] Specifically, the first priority is the highest priority, and when it is determined that the mode identifier is valid, the processing is performed based on the processing mode indicated by the valid mode identifier.

[0157] In response to a first storage space hit signal output by the address matching module, a pre-fetch mode is adopted according to a second priority.

[0158] Specifically, the first storage space hit signal is used to indicate that the memory core particle in the first chip is accessed. Although the above example has mentioned that the direct access of the memory core particle through the through silicon via is the most optimal solution, it only provides a more efficient default path, but at the same time, the possibility of passing through the input-output core particle for the purpose of some advanced optimization is reserved.

[0159] For example, in an application scenario, the data being processed by the computing core particle A will soon be needed by the computing core particle B. These data are stored in the HBM of the computing core particle A. If the direct path is used, the computing core particle B cannot directly access the HBM of the computing core particle A.

[0160] However, by configuration, the HBM address space of the computing core particle A is mapped as a "first storage space" that can be managed by the input-output core particle, so that the computing core particle B can send a request carrying a pre-fetch prompt to the input-output core particle.

[0161] In this way, the routing control unit of the input-output core particle can analyze the request, and then actively pre-fetch the data from the HBM of the computing core particle A through the substrate and put them into a shared cache or directly send them to the computing core particle B.

[0162] In other words, in the best embodiment, the first storage space is mapped as a high-speed memory (such as HBM) directly accessed through a through silicon via (TSV) to obtain the lowest access delay.

[0163] However, this application also supports a more flexible configuration. In some application scenarios, the access request of the first storage space (such as HBM) can also be routed to the input-output core particle for intelligent management, and this mode is of great significance for realizing cross-computing core particle data coordination and system-level pre-fetching.

[0164] Among them, the priority of the second priority is lower than that of the first priority.

[0165] In response to the output second storage space hit signal, the write-merge mode is adopted according to a third priority.

[0166] Specifically, the second storage space points to DDR. In this case, the write-merge mode is adopted.

[0167] In response to the suggestion signal, the suggestion mode of the suggestion signal is adopted according to a fourth priority. That is, the priority of the suggestion mode of the suggestion signal is the lowest.

[0168] In this way, the above-mentioned logical control is adopted to provide a multi-input, hierarchical priority arbitration scheme, and a robust, efficient and highly intelligent decision system is realized. By clearly defining the decision logic, conflicts and uncertainties are avoided.

[0169] Specifically, the present application adopts the mode of instruction priority, configuration second, and optimization suggestion lowest enablement, which ensures that the arbitration module can make a unique and determined decision, and avoids the contradictions caused by multiple signals being active at the same time. That is, the priority levels of the first priority to the fourth priority decrease in turn.

[0170] In some embodiments, the above-mentioned scheme can be further optimized.

[0171] Exemplarily, the programmable interconnection layer provided in the computing core and the memory core with a corresponding relationship includes a dynamic memory pooling module configured to manage heterogeneous memory resources, and the heterogeneous memory resources are composed of a plurality of memory cores.

[0172] Specifically, the dynamic memory pooling module is configured to generate a unified resource vector (i.e., to realize resource abstraction) based on the physical topology location attributes, bandwidth, delay, durability and energy consumption characteristics of the heterogeneous memory resources; to allocate, release and address map the unified resource vector in a cache line as a basic unit; to perform topology-aware allocation according to the relative position relationship between the computing core and the memory core, and to preferentially allocate frequently accessed data to the physical memory area with the lowest access delay; to divide the physical memory area into time slices and perform time-sharing multiplexing between different tasks, and the time-sharing multiplexing process is coordinated with the hardware context switching operation; to monitor the energy consumption state, and to migrate cold data to a low-power memory area or put idle memory cores into a low-power state according to a predefined energy efficiency strategy; to perform data striping and check computation inside the memory pooling layer, to perform online data recovery using check information and redundant data in response to detecting a memory core failure, and to mark and isolate the failed memory core.

[0173] Where, by identifying which memory resources (HBM, DDR, etc.) are in the system, and evaluating the static attributes (such as physical location, type) and dynamic attributes (such as current bandwidth, latency) of each resource in detail. It abstracts all this information into a unified software-readable format (uniform resource vector), which is the basis for intelligent management.

[0174] Then, in the basic unit of cache-line, allocation, release and address mapping are carried out, so that the management granularity is very fine, not the traditional large block memory. This makes it extremely fine to arrange the location of data, avoid waste, and achieve extreme optimization.

[0175] Then, according to the relative position relationship, the topology-aware allocation is performed, and the frequently accessed data is preferentially allocated to the physical memory area with the lowest access delay. Specifically, this module knows that the computing core accesses the HBM much faster than accessing the DDR. Therefore, it will intelligently place hot data (i.e. frequently accessed data) on the HBM closest to the computing unit, and cold data on the DDR, thereby maximizing overall performance.

[0176] And by dividing the physical memory area into time slices, time-sharing multiplexing is carried out between different tasks, in coordination with the hardware context switching operation. When switching from one task to another (i.e. hardware context switching), the memory pooling module can quickly allocate HBM space to the new task for use, enabling multiple tasks to share the same high-speed memory, greatly improving resource utilization.

[0177] In addition, by continuously monitoring the data access frequency (hot data / cold data), cold data that is not frequently accessed is automatically migrated to more energy-efficient memory (such as some low-power DDR mode). Even completely idle memory cores can be placed in a sleep state, thereby dynamically reducing the overall power consumption of the system.

[0178] To further enhance reliability and improve, when a memory chip fails, the system can reconstruct the lost data in real time through the check code and the data on other chips, and repair the upper-layer application without affecting the service, ensuring uninterrupted service. And mark the bad chip as unusable, and no longer allocate data to it, thereby preventing error propagation.

[0179] In other words, by setting up a dynamic memory pooling module, the computing initiative and memory passive paradigm can be changed, and the memory system can be transformed from a simple storage warehouse into an intelligent, active, and schedulable resource service grid.

[0180] Its role can be summarized in the following three core aspects:

[0181] First, by Cache-Line granularity allocation and topology-aware allocation, the memory access latency is significantly reduced, and the bandwidth utilization is greatly improved, which is beneficial to breaking through the memory wall.

[0182] Second, by time slicing, the physical memory resources are cut in the time dimension and allocated to multiple users or tasks in time, which can maximize resource utilization and economic benefits.

[0183] Third, by energy-aware strategy, data is intelligently placed in the memory with the optimal energy efficiency ratio, and the memory core enters low-power state when idle, significantly reducing system energy consumption.

[0184] In this embodiment, "context" refers to the complete environment state on which a task (process, thread) depends when executing. And "hardware context" specifically refers to the key state in this environment that is directly managed and used by the computing core hardware.

[0185] Among them, the hardware context is usually stored in specific registers inside or very close to the computing core, mainly including: program counter, the current instruction address being executed; stack pointer, the memory address of the current stack; general register, storing temporary calculation data; state register, recording the result of the last operation (such as whether it is 0, whether it is overflow, etc.); memory management unit register, such as page table base address register, which defines how the "virtual memory space" seen by the current task is mapped to "physical memory".

[0186] By switching the hardware context, the memory mapping or strategy is switched, ensuring that the new task can correctly access its allocated memory time slice, thereby realizing efficient and transparent memory sharing.

[0187] The physical memory resource is the sum of all available physical memory hardware in the system, and the physical memory region refers to a part of the physical memory resource divided according to specific attributes or functions.

[0188] In some embodiments, the programmable interconnection layer can also include a switch matrix network configured to dynamically reconfigure the connection path of the computing core and the memory core.

[0189] Among them, the generation of channel scheduling instructions in the switch matrix network depends on the resource vector and topology information provided by the dynamic memory pooling module, and the topology-aware allocation of the dynamic memory pooling module depends on the channel state information provided by the switch matrix network.

[0190] Exemplarily, the switch matrix network is configured to receive a memory access request issued by a compute core; based on a predetermined monitoring period, collect real-time state data of each silicon channel, the real-time state data including channel utilization, queue depth and access latency; in combination with historical channel state data and real-time state data, generate a plurality of logical candidate paths through a path calculation algorithm, wherein each logical candidate path corresponds to a virtualization mapping and scheduling scheme of a silicon channel; through a conflict arbitration mechanism, determine a target logical path group from the plurality of logical candidate paths according to the priority and bandwidth requirement of the memory access request; generate channel scheduling instructions according to the target logical path group, and dynamically allocate data flow to the plurality of silicon channels in a logical manner, thereby achieving multi-channel load balancing and cross-period bandwidth allocation.

[0191] The application also provides a memory scheduling method, which can be applied to the three-dimensionally stacked chip system in the foregoing examples.

[0192] The memory scheduling method can include: in response to a memory access request sent by a compute core, performing a parsing operation (i.e., step 1); in response to a parsing result of the parsing operation, performing the following operations: in a write-merge buffer mode, aggregating a plurality of write requests and sending them to a memory controller, the memory controller being located in a first chip; in a prefetch buffer mode, based on an access pattern, prompting to prefetch data to a cache (i.e., step 2).

[0193] In an example, the memory scheduling method can include: in response to a target address of a memory access request generated by a compute core pointing to a first storage space, generating a first semantic tag, the first semantic tag being transmitted to a memory controller by multiplexing a silicon channel; in response to the target address of the memory access request generated by the compute core pointing to a second storage space, generating a second semantic tag, the second semantic tag being transmitted to a routing control unit through a substrate semantic channel.

[0194] The step 2 is for the execution scheme of the second storage space.

[0195] The application also provides an electronic device, which can include the three-dimensionally stacked chip system in the foregoing examples.

[0196] The electronic device can include a mobile device such as a mobile phone or a tablet, and can also include a computer.

[0197] It should be noted that although several modules or units of a device for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. In fact, according to the embodiments of the application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units for embodiment.

[0198] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to encompass within their scope all possible alternatives and modifications to the preferred embodiments.

[0199] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended to include all such changes and modifications in the scope of the application recited in the claims.

Claims

1. A three-dimensional stacked chip system, characterized by, The application relates to a chip system, comprising: a substrate; one or more computing cores arranged on the substrate; one or more memory cores, each of which is arranged above a corresponding computing core through a through-silicon via stack, and the computing core and the memory core are packaged as a first chip; one or more input / output cores arranged on the substrate on one side of the computing core and connected with each of the computing core through substrate wiring, the input / output core is packaged as a second chip, and there is no direct communication path with one or more memory cores; the computing core also transmits a first semantic tag carried by a memory access request through a multiplexed through-silicon via channel, the first semantic tag is parsed and executed by a memory controller, the memory controller is located in the first chip, and is connected with the one or more memory cores; wherein the first semantic tag is generated when a destination address of the memory access request points to a first storage space, and the first storage space is a location of the one or more memory cores; the input / output core comprises a routing control unit configured to execute the following operation in response to a second semantic tag carried by a memory access request of the computing core: in a write merge buffer mode, a plurality of write requests are aggregated and sent to the memory controller; in a prefetch buffer mode, data is prompted to be prefetched to a cache based on an access mode; wherein the computing core sends a memory access request carrying a second semantic tag to the routing control unit through a substrate semantic channel, and the second semantic tag is generated when a destination address of the memory access request points to a second storage space, and the second storage space can be different from the first storage space.

2. The chip system according to claim 1, characterized by the computing core comprises one or more of a central processing unit, a graphics processing unit and a neural network processing unit; the memory core comprises a high-bandwidth memory core and a double data rate memory core.

3. The chip system according to claim 1, characterized by further comprising: solder balls between the computing core and the substrate, and between the input / output core and the substrate; the solder balls are used to connect the computing core and the substrate, and the input / output core and the substrate, and the substrate wiring connects the input / output core and the computing core through the solder balls.

4. The chip system according to claim 3, characterized by the second semantic tag comprises an operation mode identifier and parameter domain information; the routing control unit comprises: a semantic decoding module configured to receive a memory access request from the computing core and parse a second semantic tag carried thereby, and output a mode identifier and parameter domain, the mode identifier comprising a write merge mode and a prefetch mode; an address matching module coupled with the semantic decoding module and configured to determine whether a target address of the memory access request falls within an accessible storage space according to the target address of the memory access request and a configurable storage space range in a programmable register group, and output a hit signal; a feature analysis module coupled with the semantic decoding module and configured to generate a suggestion signal based on a data packet size, a request type and a historical access trajectory. The arbitration module is coupled with the semantic decoding module, the address matching module and the feature analysis module, and is configured to, in response to the mode identifier being valid, adopt a processing mode of a memory access request corresponding to the mode identifier according to a first priority; in response to a first storage space hit signal output by the address matching module, adopt the prefetch mode according to a second priority, and in response to a second storage space hit signal output, adopt the write merge mode according to a third priority; and in response to the suggestion signal, adopt a suggestion mode of the suggestion signal according to a fourth priority.

5. The chip system according to claim 4, characterized by The operation of the write merge buffer mode includes: collecting unaligned write requests within a time window, and merging into a burst write operation after alignment according to a DRAM page boundary; The operation of the prefetch buffer mode includes: calculating a prefetch step according to an access mode prompt, and storing a subsequent data block in the cache according to the step.

6. The chip system according to claim 1, characterized by Further comprising: A programmable interconnection layer arranged between the computing core and the memory core having a corresponding relationship, the programmable interconnection layer comprising: a dynamic memory pooling module configured to manage heterogeneous memory resources, the heterogeneous memory resources being composed of a plurality of memory cores; The dynamic memory pooling module is configured to generate a unified resource vector based on the physical topology location attributes, bandwidth, delay, durability and energy consumption characteristics of the heterogeneous memory resources; allocate, release and address map the unified resource vector in a basic unit of cache behavior; perform topology-aware allocation according to the relative position relationship between the computing core and the memory core, and preferentially allocate frequently accessed data to the physical memory region with the lowest access delay; divide the physical memory region into time slices and perform time multiplexing between different tasks, the time multiplexing process being coordinated with the hardware context switching operation; monitor the energy consumption state, and according to a predefined energy efficiency strategy, migrate cold data to a low-power memory region or place idle memory cores into a low-power state; perform data striping and check computation within the memory pooling layer, and in response to detecting a memory core failure, perform online data recovery using check information and redundant data, and mark and isolate the failed memory core.

7. A memory scheduling method, comprising: The memory scheduling method is applied to the three-dimensional stacked chip system of any one of claims 1 to 6, and the memory scheduling method comprises: In response to a memory access request sent by the computing core, performing a parsing operation; In response to a parsing result of the parsing operation, performing the following operations: in a write merge buffer mode, aggregating a plurality of write requests and sending them to a memory controller, the memory controller being located in a first chip; in a prefetch buffer mode, prefetching data to a cache based on an access mode prompt.

8. An electronic device, comprising: The three-dimensional stacked chip system of any one of claims 1 to 6. The three-dimensional stacked chip system of any one of claims 1 to 6.

Images