A DLRM training accelerator and acceleration method based on heterogeneous multi-core particles

Abstract

This application belongs to the field of computer architecture and artificial intelligence accelerators, and discloses a DLRM training accelerator and acceleration method based on heterogeneous multi-chip architecture. The DLRM training accelerator includes input / output chips, a high-bandwidth memory (HBM) chipset, and a high-density flash memory chipset. The HBM chipset and the high-density flash memory chipset are connected to the input / output chipset through inter-chip interconnect links. The HBM chipset integrates a memory computing module, and the high-density flash memory chipset integrates a flash memory computing module. This application realizes near-data processing for embedded layer operations and gradient updates, effectively eliminating the bottleneck caused by large-scale data transfer between storage and computing chips. While balancing massive storage capacity and high-bandwidth access, it significantly improves the throughput of DLRM training and greatly reduces system energy efficiency and deployment costs.

Description

Technical Field

[0001] This application belongs to the field of computer architecture and artificial intelligence accelerators, specifically relating to a DLRM training accelerator and acceleration method based on heterogeneous multi-core particles. Background Technology

[0002] With the explosive growth of internet data, deep learning recommendation models (DLRM) have become the core of personalized services. Their performance is highly dependent on the size of the embedding tables, which have evolved to terabyte (TB) or even petabyte (PB) scales to capture more complex feature interactions. However, there is a serious contradiction between this enormous storage demand and hardware storage performance: traditional DRAM-based architectures, due to their low storage density, are difficult to economically scale to TB-level capacity; while frequent feature extraction and backpropagation updates generate extremely sparse and irregular memory access requests, leading to frequent data transfer between the processor and memory, causing severe memory and power wall bottlenecks, which greatly limit the training efficiency of ultra-large-scale recommendation models.

[0003] To alleviate memory access bottlenecks, existing technologies primarily explore two approaches: First, DRAM-based in-memory computation (NDP / PIM) schemes, such as TensorDIMM or RecNMP. While these schemes offer high bandwidth, the high cost and limited capacity of DRAM prevent them from supporting complete, ultra-large-scale embedding tables. Furthermore, as the model size increases, the communication overhead between multiple nodes offsets the performance gains. Second, SSD or flash memory-based near-memory computation schemes aim to leverage their low cost and high capacity. However, existing flash memory schemes mostly perform controller-side computation, requiring data to still be transferred from the flash memory chip to the controller via internal channels with limited bandwidth. More critically, existing schemes ignore the severe write amplification phenomenon caused by high-frequency, random parameter updates during DLRM training, which significantly reduces training throughput and severely shortens the lifespan of flash memory chips. Additionally, existing systems often employ uniform storage, failing to effectively utilize the heat distribution characteristics of embedding access. Summary of the Invention

[0004] To address the technical challenges faced by existing technologies in training TB-level embedded tables, such as memory access bottlenecks, excessive power consumption during data transfer, and write amplification and shortened lifespan when using flash memory as a storage medium, this application provides a DLRM training accelerator and acceleration method based on heterogeneous multi-core chips. Through a near-data processing architecture designed in conjunction with hardware and software, it significantly improves training throughput and reduces energy consumption while ensuring the storage capacity of ultra-large-scale models.

[0005] To achieve the above objectives, this application employs the following technical solution:

[0006] This application discloses a heterogeneous multi-chip DLRM training accelerator, comprising an input / output die (I / O die), a high-bandwidth memory (HBM) chipset, and a high-density flash memory (HDF) chipset. The HBM chipset and the HDF chipset are connected to the I / O dies via die-to-die links.

[0007] The input / output die integrates a scheduling mapping management unit (SMMU), a flash memory controller, a protocol converter, and a cross-media interconnect interface, which includes a high-bandwidth memory interface and a high-density flash memory interface.

[0008] A further improvement of this application is that the high bandwidth (HBM) memory chipset adopts a three-dimensional stacked structure, including a base die and several memory computing dies (IMP dies), and the several memory computing dies (IMPDies) are stacked on top of the base die through vertical through silicon vias (TSVs).

[0009] Each of the memory computing die (IMP Die) is physically and logically divided into 32 independent banks. Each IMP Die integrates 16 vector processing units (VPUs) in situ, and the VPUs are located adjacent to the banks.

[0010] A further improvement of this application is that: the base die integrates a computing subsystem, a buffer subsystem, and an on-chip network (NoC), wherein the on-chip network (NoC) connects the computing subsystem and the buffer subsystem, wherein:

[0011] The computing subsystem integrates a dedicated data register, a dedicated processing engine, and a multilayer perceptron engine (MLP engine) to perform unified representation conversion of heterogeneous inputs, parallel vector accumulation, and fully connected layer computation.

[0012] The buffer subsystem includes a data buffer for caching operands and hot embeddings, and a global buffer for managing cross-grain data exchange.

[0013] The Network on-Chip (NoC) is equipped with a Near Memory Router (NOP Router), a Memory Controller (DRAMCTRL), and a Unified Controller (CTRL) to enable high-speed routing and coordinated scheduling of internal instructions and data.

[0014] A further improvement of this application is that the vector processing unit (VPU) includes a computing component, a storage component, and a control and switching component, wherein:

[0015] The computing components include a cascaded hardware adder tree and an arithmetic logic unit (ALU).

[0016] The storage components include a local vector buffer, a gradient buffer, and a partial sum buffer.

[0017] The control and switching components include a hardware switch, which contains an instruction register set for specifying the target bank and operand range according to the received instructions.

[0018] A further improvement of this application is that the high-density flash memory chipset includes: Each flash data physical channel is connected to multiple flash chips. Each flash chip contains multiple flash dies. Each flash die integrates a flash computing unit (PU) and a low-level controller.

[0019] A further improvement of this application is that the flash computing unit (PU) includes:

[0020] Local computing components: including high-speed multiplexers (MUX) and dedicated local computing units (Compute Units).

[0021] The buffer system includes a flash-side read buffer, a flash-side gradient buffer, and a write buffer with physical write traffic shaping and integration functions. It is used to implement data temporary storage, gradient alignment, logical merging of updated data, and batch write-back.

[0022] The underlying controller includes a first-in-first-out hardware queue (FIFO), an instruction register (Instr Register), an address register (Addr Register), and a dedicated control state machine. The dedicated control state machine includes independent compute control, write control, and read control.

[0023] This application also provides a method for accelerating DLRM training based on a heterogeneous multi-core DLRM training accelerator, which specifically includes the following steps:

[0024] Step 1, Data Mapping and Adaptive Hotspot Partitioning Initialization: The input / output die (I / O Die) receives input layer data consisting of embedded tables distributed by the host, and the scheduling mapping management unit (SMMU) configured inside the input / output die (I / O Die) completes the hierarchical storage initialization of the target hotspot vector and long tail vector;

[0025] Step 2, Request Routing and Multi-Level Near-Memory Instruction Generation: The input / output die (I / O Die) determines the physical location of the target embedding vector by concurrently searching the global residency table based on the received sparse training index request using the internal scheduling mapping management unit (SMMU). The protocol converter configured adjacent to the scheduling mapping management unit (SMMU) parses the original sparse training index request into multi-level near-memory instructions with a regular hardware field format. The multi-level near-memory instructions are distributed in parallel to the corresponding hardware paths of the high-bandwidth memory (HBM) chipset or the high-density flash memory (HDF) chipset via die-to-die links, so that subsequent steps can perform corresponding hierarchical near-data processing based on the value of the path selection factor field (PathSel).

[0026] Step 3: Hierarchical near-data processing in parallel execution: The high-bandwidth memory (HBM) chipset or the high-density flash memory (HDF) chipset initiates distributed differential feature extraction and parallel reduction at the near end of the medium based on the path selection factor field (PathSel) in the received multi-level near-memory instructions, to obtain all local parts and results of the memory computing (IMP) module and the flash computing (IFP) module;

[0027] Step 4, Global Aggregation and Perceptron Forward Computation: The Base Die collects all local parts and results from the In-Memory Computing (IMP) module and the Flash Computing (IFP) module returned in Step 3. After performing physical dimension alignment, feature concatenation and unified representation transformation on the local parts and results, a complete global embedded vector data entity is obtained. The global embedded vector data entity is sent to the Multilayer Perceptron (MLP) engine to obtain the activation output prediction results corresponding to each neural network layer. Finally, the entire forward propagation result containing the activation output prediction results is output.

[0028] Step 5, Backward Gradient Backpropagation and Software-Hardware Collaborative Asynchronous Weight Update: Automatically triggered after forward propagation and loss difference calculation, distributed differentiated in-situ overlay updates are performed near each storage medium through software and hardware collaboration.

[0029] A further improvement in this application is that step 1 specifically includes the following steps:

[0030] Step 1.1, Popularity Streaming Statistics and Calculation: Within a preset hardware observation window, the size of the window is dynamically determined by a preset multiple of the current training batch size. The input / output chip (I / ODie) receives input layer data composed of embedding tables. Its internal scheduling mapping management unit (SMMU) uses an on-chip hardware counter to stream-count the physical access count of each embedding table entry index in real time, and calculates the adaptive popularity value corresponding to each embedding table entry index within the current preset hardware observation window based on this streaming-counting.

[0031] ,

[0032] in, This represents the number of physical accesses accumulated and recorded by the on-chip hardware counter within the preset hardware observation window, and the counter is reset at the end of the preset hardware observation window. ;

[0033] Step 1.2, Real-time comparison of residency status: The dual-threshold hysteresis algorithm automatically compares the global physical residency level position of each embedded entry index with the built-in global residency table in real time: If the current embedded entry resides in the high bandwidth memory (HBM) chipset and its adaptive heat value is lower than the preset migration threshold, it is marked as a migration candidate; if the current embedded entry resides in the high density flash memory (HDF) chipset and its adaptive heat value is higher than the preset migration threshold, the cross-media migration process is immediately triggered at the hardware layer and the process flows to step 1.3.

[0034] Step 1.3, Cross-Media Space Detection and Dynamic Replacement / Evict: During the cross-media migration process, the Scheduling Mapping Management Unit (SMMU) first detects the residency status of the physical storage space within the High Bandwidth Memory (HBM) chipset used for caching hotspot vectors. If it is determined that there is still remaining space, the target vector is directly read from the High Density Flash Memory (HDF) chipset and written in parallel to the High Bandwidth Memory (HBM) chipset through the memory controller. If it is determined that the physical space is full, the embedded parameter with the lowest current adaptive heat value within the High Bandwidth Memory is forcibly retrieved as the proposed replacement sacrifice. If and only if satisfying The condition is that the adaptive heat value of the target hotspot vector that triggers the migration is higher than the sacrificial term to be replaced. When setting the adaptive heat value, the sacrifice item will be... The physical weights are asynchronously written back to the high-density flash (HDF) chipset via the flash controller. Then, the target hotspot vector that triggers the migration is written into the freed-up high-bandwidth memory physical space, and the global resident table that resides on the chip is refreshed in real time. The scheduling mapping management unit (SMMU) updates the global resident table according to the adaptive partitioning result to construct the physical mapping relationship and distribute it, thus completing the hierarchical storage initialization of the target hotspot vector and the long tail vector.

[0035] A further improvement in this application is that step 2 specifically includes the following steps:

[0036] Step 2.1, Concurrent Physical Residence Location Retrieval: During the bidirectional iterative training of the Deep Learning Recommendation Model (DLRM), the Input / Output Die (I / O Die) receives sparse training index requests distributed by the external host. The Scheduling Mapping Management Unit (SMMU) uses on-chip hardware counters and retrieval circuits to concurrently retrieve the real-time maintained on-chip global residence table for the sparse training index requests, so as to accurately determine the physical residence location results of each target embedding vector in the sparse training index request in the High Bandwidth Memory (HBM) chipset or High Density Flash Memory (HDF) chipset, and sends the physical residence location results to the adjacent Protocol Converter.

[0037] Step 2.2, Hard-core Decoding and Reassembly of Near-Memory Instructions: The Protocol Converter receives the physical residency location result issued in Step 2.1, and immediately performs hard-core decoding and reassembly parsing on the original sparse training index request to generate multi-level near-memory instructions with a regular hardware field format; the multi-level near-memory instructions completely and strictly include: opcode field (OpCode), table identifier field (TableId), physical embedding item index field (EmbIndex), path selection factor field (PathSel), and address field (Addr);

[0038] Step 2.3, Multi-field instruction bus distribution and submission: The generated multi-level near-memory instruction stream is distributed in parallel across the bus layer via a low-latency die-to-die link. Based on the hardware indication status of the path selection factor field (PathSel) in the multi-level near-memory instruction, the stream is routed and submitted to the corresponding high-bandwidth memory (HBM) chipset physical path or high-density flash memory (HDF) chipset physical path, thereby completing the request normalization routing before feature extraction.

[0039] A further improvement in this application is that, in step 3, the differential feature extraction specifically involves:

[0040] When the path selection factor field (PathSel) in the multi-level near memory instruction indicates a flow to a hotspot path, the memory computing (IMP) module within the high bandwidth memory (HBM) chipset uses the vector processing unit (VPU) to extract the hotspot vector partitioned in step 1 in-situ at the near end of the medium, and calls the internal cascaded hardware adder tree to perform partial sum (Psum) aggregation and reduction calculations in parallel.

[0041] When the path selection factor field (PathSel) in the multi-level near-memory instruction indicates a flow to the long-tail path of the flash memory, the flash computing (IFP) module within the high-density flash (HDF) chipset processes the long-tail vector divided in step 1 locally by the flash computing unit (PU) deeply embedded within each flash die. Before issuing a read request to the external physical flash memory array, it uses the table identifier field (TableId) and the physical embedded item index field (EmbIndex) of the current request to perform data matching and retrieval in its built-in on-chip write buffer. If a match is found, the latest logical copy data temporarily stored in the write buffer is directly called to participate in forward summation and reduction to skip the physical array read. If the retrieval fails, a non-volatile page read operation is initiated to the underlying physical flash memory array through the underlying first-in-first-out hardware queue (FIFO) and control state machine, and the original feature vector is streamed into the data register group. Then, the dedicated local computing unit (Compute) within the flash computing unit (PU) performs the data matching and retrieval. The Unit and the high-speed multiplexer (MUX) accurately extract the sparse subvectors required for the current batch, following the local partial sum aggregation reduction formula: Execute local partials and aggregations, where This represents the sparse subvector entity that has been precisely extracted. This represents the set of sparse subvectors within the current flash memory die that are activated by the current batch of instructions;

[0042] Pipeline parallel optimization: During the execution of the long tail path, the flash computing unit (PU) makes full use of the characteristics of 16 independent flash data physical channels and adopts an instruction-level pipeline overlapping scheduling strategy. While performing the vector aggregation calculation of the current batch, the instruction queue inside the input / output die (I / O Die) pre-initiates the flash page streaming pre-read of the long tail items of the subsequent batch.

[0043] A further improvement in this application is that: Step 4 result feedback: After the hot spot path and long tail path complete the local partial sum aggregation, only the obtained low-dimensional part and result are fed back. The data is streamed via on-chip network (NoC) or cross-die interconnect bus to the global buffer of the base die at the bottom of the high-bandwidth memory (HBM) chipset, following a global aggregation formula. The data is then concatenated and assembled into a complete global embedding vector to alleviate the bus bandwidth bottleneck in the feature extraction process at the chip architecture design level and compress the total amount of data transmitted to the memory chip. Specifically, this includes the following steps:

[0044] Step 4.1, Heterogeneous Sparse Feature Dimension Alignment and Representation Transformation: The Base Die collects all local parts and results from the In-Memory Computing (IMP) module and the Flash Computing (IFP) module returned in Step 3, and temporarily stores them in the on-chip Data Buffer. Then, the dedicated Process Engine is called to perform the final physical dimension alignment, feature splicing and unified representation transformation on these local parts and results, so as to assemble them into the complete global embedded vector data entity required for the current batch.

[0045] Step 4.2, Fully Connected Dense Feature Matrix Multiplication and Addition Fusion Calculation: The aligned global embedded vector data entities are streamed into the adjacent Multilayer Perceptron (MLP) engine; the MLP engine, through its internally configured hard-core systolic array, performs forward propagation (FP) calculation of fully connected matrix multiplication and addition fusion on the assembled global embedded vector data entities and the continuous dense feature stream synchronously input from the host; the MLP engine uses the hard-core systolic array to perform general multilayer neural network matrix multiplication and addition and activation function mapping, obtaining the activation values ​​corresponding to each neural network layer layer by layer, and finally obtaining and deriving the activation output prediction result of the current training batch;

[0046] Step 4.3: The on-chip control logic directly coordinates with the activation output prediction result and the current training batch size to perform a global summation according to the preset training loss function, directly calculating the global training loss value within the chip; finally, the entire forward propagation result containing the activation output prediction result and the global training loss value is output through the input / output die (I / O Die).

[0047] A further improvement in this application is as follows: Step 5.1, Distributed gradient flow cross-chip distribution scheduling: The multilayer perceptron (MLP) engine uses its internally configured hardware systolic array to perform general multilayer neural network inverse matrix multiplication and addition calculations, and initiates backpropagation (BP) calculations inside the chip. Based on the preset deep learning recommendation model (DLRM), it trains the cross-entropy loss function as the boundary condition, and calculates the sparse parameter gradients of the embedding table parameters of each layer corresponding to the current batch in real time and stores them in the data buffer. The gradient data is distributed across chips to the corresponding storage medium under the scheduling of the global resident table retrieved by the scheduling mapping management unit (SMMU) inside the input / output die (I / O Die).

[0048] Step 5.2, Local flushing of near-end parameters of memory hotspot path: When the gradient data is distributed to the memory hotspot path, the memory computing (IMP) module in the high bandwidth memory (HBM) chipset receives the gradient, uses the vector processing unit (VPU) in conjunction with the local vector buffer, directly calls its internally integrated arithmetic logic unit (ALU) in conjunction with the local inward buffer physical control, and performs gradient multiplication and addition in the memory chip according to the preset optimizer algorithm, completes the weight update of the hotspot vector divided in step 1 in the local area, and directly streams it back to the underlying connected bank.

[0049] Step 5.3, Decoupled Asynchronous Batch Programming Write of Long-Tail Flash Path: When the gradient data is distributed to the long-tail flash path, the flash computing (IFP) module in the high-density flash (HDF) chipset routes the gradient to the gradient buffer of the flash computing unit (PU). The write buffer is used to accumulate the gradient data with the temporarily stored old parameter data, thereby completing the local logical overwrite and merging of the long-tail vectors partitioned in Step 1. The system adopts a strategy of complete decoupling between logical updates and physical updates. The on-chip circuit determines that the amount of unsynchronized data temporarily stored in the write buffer satisfies the conditional expression: When the standard physical page full-page granularity threshold of the physical flash memory is reached, the control state machine centrally activates the flash memory controller and initiates a batch of one-time physical page programming write operations using multiple independent flash data physical channels. This converts discrete random writes inside the chip into spatially continuous and page-aligned batch sequential writes, centrally executing flash page physical programming operations to complete the asynchronous update of the underlying physical array. This fundamentally eliminates the flash write amplification effect and extends the physical lifespan of the heterogeneous accelerator.

[0050] The beneficial effects of this application are:

[0051] This application presents a hierarchical data flow based on a heterogeneous multi-core near-data processing architecture, which minimizes data transport latency and energy consumption in large-scale embedded tables, significantly improves training throughput, and optimizes operating power consumption.

[0052] This application provides a custom multi-level instruction set to maximize the hardware utilization of high-bandwidth memory computing and flash computing, thereby further enhancing the overall energy efficiency of the system.

[0053] This application uses an adaptive hierarchical organization of embedded tables based on memory access frequency to break the contradiction between capacity and bandwidth, and significantly reduce hardware deployment costs.

[0054] The overall architecture of this application, through heterogeneous multi-chip (Chiplet) integration, allows the control logic and different types of storage media (DRAM and NAND Flash) to be manufactured independently at the node most suitable for their process technology. It also achieves tight coupling between petabyte-level storage capacity and computing power through an interposer, effectively alleviating the storage wall bottleneck in DLRM training. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall architecture of the deep learning recommendation model training accelerator based on heterogeneous multi-core particles in this application.

[0056] Figure 2 This is a block diagram of the overall microarchitecture of the high-bandwidth memory chipset and stacked memory computing chips in this application.

[0057] Figure 3 This is a schematic diagram of the logical structure of the functional subsystem and hardware processing engine inside the substrate grain in this application.

[0058] Figure 4 This is a hierarchical microarchitecture diagram of the high-density flash memory chipset and embedded flash memory computing unit in this application.

[0059] Figure 5 This is a logical topology diagram of the input / output die internal modules and multi-level calculation instruction field format in this application.

[0060] Figure 6 This is a schematic diagram of the global forward and backward propagation hard core data flow of the training acceleration method in this application.

[0061] Figure 7 This is a comparison and evaluation chart of the end-to-end training throughput speedup between this application and traditional architectures under different recommendation model scales.

[0062] Figure 8 This is a comparison and evaluation chart of the normalized total operating energy consumption of this application and traditional architecture under different recommendation model scales. Detailed Implementation

[0063] The embodiments of this application will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit this application. That is, in some embodiments of this application, these practical details are not necessary. In addition, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0064] like Figure 1As shown, this application discloses a heterogeneous multi-chip DLRM training accelerator. The DLRM training accelerator includes an input / output die (I / O die), a high-bandwidth memory (HBM) chipset, and a high-density flash memory (HDF) chipset. The HBM chipset and the HDF chipset are connected to the I / O die via die-to-die links. The HBM and HDF are located on either side of the central I / O die, and they are physically connected and electrically interact with each other via the indicated die-to-die links.

[0065] Combination Figure 1 As shown in the enlarged hierarchical structure diagram on the left, the high-bandwidth memory chipset is physically constructed in a three-dimensional hierarchical stack, including a base die and four memory compute dies (IMP dies). The four IMP dies are stacked on top of the base die through vertical through silicon vias (TSVs). High-speed signal transmission between the dies is achieved through TSV technology. Eight channels are arranged on each physical layer inside these dies.

[0066] Combination Figure 1 As shown in the magnified channel topology diagram on the right, the high-density flash memory (HDF) chipset includes... Each has a separate physical flash memory data channel, labeled as Flash Channel 0, Flash Channel 1, and so on up to Flash Channel 2. In this embodiment, Each flash data physical channel is connected to four physical flash memory chips to provide high-concurrency data access capabilities.

[0067] like Figure 2 As shown, the high bandwidth memory (HBM) chipset includes a base die at the bottom and four memory computing dies (IMP Dies) stacked on top of the base die. The four memory computing dies (IMP Dies) are arranged vertically from bottom to top and are precisely positioned and labeled as memory computing die 0, memory computing die 1, memory computing die 2 and memory computing die 3.

[0068] To provide underlying concurrent lookup capabilities, each In-Memory Compute Die (IMP Die) is physically configured with 16 Bank Groups, each containing 2 Banks, thus forming 32 independent Banks within a single die to achieve high parallel access capabilities. To eliminate data transport overhead, each IMP Die integrates 16 Vector Processing Units (VPUs) in situ for performing local summation and gradient updates of hotspot embedded vectors. The specific microarchitecture of each VPU is shown in the enlarged view with the rightmost dashed box. Each VPU includes computing components, storage components, and control and switching components. The VPUs are located adjacent to the Banks and can directly obtain the raw vector data output by the Banks and perform in-situ reduction calculations. The computational components include a cascaded hardware adder tree and an arithmetic logic unit (ALU) for performing vector aggregation during the forward propagation phase and in-situ gradient application and embedding updates during the back propagation phase. The storage components include a local vector buffer, a gradient buffer, and a partial sum buffer for temporarily storing the embedding vectors to be processed, the corresponding gradient data, and the partial sum reduction results; these components enable a read-modify-write pipelined operation. The control and switching components include a hardware switch, which internally contains an instruction register set for specifying the target bank and operand range based on received instructions. The hardware switch allows the VPU to dynamically select the target bank based on control instructions.

[0069] like Figure 3 As shown, the high-bandwidth memory (HBM) chipset's control and computing hub, the base die, integrates a computing subsystem, a buffer subsystem, and an on-chip network (NoC) in situ through functional decoupling and modular design. The on-chip network (NoC) connects the computing subsystem and the buffer subsystem, wherein:

[0070] The computational subsystem internally comprises, from top to bottom, a dedicated data register, a dedicated processing engine, and a multilayer perceptron engine (MLP engine). These are used to perform unified representation transformation of heterogeneous inputs, parallel vector accumulation, and fully connected layer computation. The dedicated data register and processing engine are responsible for aligning and representing heterogeneous data from HBM and HDF. The multilayer perceptron engine performs subsequent forward computation and backward gradient generation for the fully connected layers.

[0071] The buffer subsystem includes a data buffer for caching operands and hot embeddings, and a global buffer for managing cross-die data exchange. The buffer subsystem includes one global buffer and one data buffer arranged in parallel from top to bottom, which are used as a relay station for cross-media data exchange and to temporarily store operands required for multilayer perceptron computation, respectively.

[0072] The Network on-Chip (NoC) is equipped with a Near-Memory Router (NOP Router), a Memory Controller (DRAMCTRL), and a Unified Controller (CTRL) to achieve high-speed routing and coordinated scheduling of internal instructions and data, and supports concurrent routing and coordinated scheduling of multiple instruction and data streams.

[0073] like Figure 3 As shown in the microarchitecture within the rightmost dashed box, the processing engine unit has a fully parallel pipelined arithmetic array cascaded internally. This array consists of an input register group (Input Reg), several horizontally parallel adder tree modules containing three hardware adders each, and an output register group (Output Reg) cascaded together to perform partial summation and regularization accumulation of feature vectors in situ.

[0074] like Figure 4 As shown, the high-density flash memory (HDF) chipset is physically configured with 16 completely independent flash data channels at the underlying level. Each physical channel has multiple high-density flash memory chips connected in parallel via a dedicated flash controller and interface, and each chip contains two flash dies. Each flash memory chip contains multiple flash dies, and each flash die integrates a flash computing unit (PU) and an underlying controller. The flash computing unit (PU) is used to perform in-situ extraction of long-tailed embedding vectors and aggregation of gradient update data.

[0075] This on-chip computing design enables the aggregation operation of long-tailed embedded vectors to be completed inside the flash memory, significantly reducing the power consumption pressure on the interface bus.

[0076] The internal microarchitecture of the flash computing unit (PU) is shown in the schematic diagram at the top right. It integrates a high-speed multiplexer (MUX), a dedicated local computing unit (Compute Unit), a flash-side read buffer, a flash-side gradient buffer, and a write buffer with physical write traffic shaping and integration functions. This is used to implement data temporary storage, gradient alignment, logical merging of updated data, and batch write-back. The high-speed multiplexer (MUX) is used to accurately locate the target embedded item sub-vector in the flash page according to the instructions. The Compute Unit performs cross-dimensional local summation and aggregation. The write buffer adopts an asynchronous update strategy, which logically merges multiple sparse reverse gradient update requests in the buffer, fundamentally suppressing random write amplification of flash memory.

[0077] In a preferred embodiment, the write buffer is preferably configured to have a capacity of 4MB. Meanwhile, the underlying controller of each flash memory die, as shown in the schematic diagram on the lower right, consists of a dedicated first-in-first-out (FIFO) hardware queue, an instruction register, an address register, and a dedicated control state machine interconnected. The dedicated control state machine internally includes independent compute control, write control, and read control units. The underlying controller is used to coordinate the scheduling of read, write, and erase states.

[0078] like Figure 5As shown, the input / output die, serving as the central management hub for the entire heterogeneous multi-chip system, integrates a First-In-First-Out (FIFO) hardware queue, a High-Bandwidth Memory Interface (HBMInterface), a Protocol Converter, a Scheduling Mapping Management Unit (SMMU), a Flash Controller, and a High-Density Flash Interface (HDF Interface) in situ through a highly cohesive modular design. The Flash Controller houses a Core Management Unit (CMU), which internally includes an Output Static Buffer (SRAM-OUT), an Input Static Buffer (SRAM-IN), and dedicated Address Management Unit (Addr), Garbage Collection Management Unit (GC), Flash Translation Layer Management Unit (FTL), and Error Correction Code Management Unit (ECC). Meanwhile, as shown at the top and bottom, the system's configured multi-level near-memory instructions have a highly regular hardware field format, which fully includes: opcode field (OpCode), table identifier field (TableId), physical embedding item index field (EmbIndex), path selection factor field (specifically configured as PathSel-HBM or PathSel-HDF), and address field (Addr).

[0079] The Deep Learning Recommendation Model (DLRM) training accelerator proposed in this application effectively accelerates the bidirectional operation of the DLRM model while ensuring computational accuracy, improving model inference and training speed, and significantly reducing data transmission energy consumption. Typically, the DLRM model consists of an embedding layer, a bottom fully connected layer, and a top fully connected layer. The embedding layer maps discrete sparse features such as user ID and product ID into low-dimensional dense vectors through table lookup operations; its massive embedding table, reaching terabytes in size, is the main bottleneck in terms of storage and power consumption. The bottom fully connected layer handles continuous dense features such as user age and price. The top fully connected layer receives the concatenated dense features and the sparse features output from the embedding layer to predict and output the final recommendation probability, such as the click-through rate. The acceleration challenge of this model lies in the fact that, facing a massive embedding layer, the lookup and lengthy transfer of discrete embedding vectors consumes the majority of the runtime, leading to increased latency in model inference and training. Therefore, multi-core collaborative scheduling is needed for near-memory acceleration.

[0080] This application also provides a DLRM training acceleration method based on a heterogeneous multi-chip DLRM training accelerator. Through hardware and software collaborative scheduling of the input / output die (I / O die) with the high-bandwidth memory (HBM) chipset and high-density flash memory (HDF) chipset, it achieves high-concurrency, low-latency near-memory computing. Specifically, it includes the following steps:

[0081] Step 1, Data Mapping and Adaptive Hotspot Partitioning Initialization: The input / output die (I / O Die) receives input layer data consisting of embedded tables distributed by the host, and the scheduling mapping management unit (SMMU) configured inside the I / O die completes the hierarchical storage initialization of the target hotspot vector and long-tail vector. Specifically:

[0082] Step 1.1, Popularity Streaming Statistics and Calculation: Within a preset hardware observation window, the size of the window is dynamically determined by a preset multiple of the current training batch size. The input / output chip (I / ODie) receives input layer data composed of embedding tables. Its internal scheduling mapping management unit (SMMU) uses an on-chip hardware counter to stream-count the physical access count of each embedding table entry index in real time, and calculates the adaptive popularity value corresponding to each embedding table entry index within the current preset hardware observation window based on this streaming-counting.

[0083] ,

[0084] in, This represents the number of physical accesses accumulated and recorded by the on-chip hardware counter within the preset hardware observation window, and the counter is reset at the end of the preset hardware observation window. ;

[0085] Step 1.2, Real-time comparison of residency status: The dual-threshold hysteresis algorithm automatically compares the global physical residency level position of each embedded entry index with the built-in global residency table in real time: If the current embedded entry resides in a high-bandwidth memory (HBM) chipset and its adaptive heat value is lower than the preset migration-out threshold (16-32 in this embodiment), it is marked as a migration-out candidate; if the current embedded entry resides in a high-density flash (HDF) chipset and its adaptive heat value is higher than the preset migration-in threshold (32-64 in this embodiment), the cross-media migration process is immediately triggered at the hardware layer and the process flows to step 1.3; wherein, the migration-in threshold and the migration-out threshold are scaled proportionally according to the capacity of the preset hardware observation window, and the migration-in threshold is always greater than the migration-out threshold during actual system operation to build a hysteresis mechanism at the hardware layer;

[0086] Step 1.3, Cross-Media Space Detection and Dynamic Replacement / Evict: During the cross-media migration process, the Scheduling Mapping Management Unit (SMMU) first detects the residency status of the physical storage space within the High Bandwidth Memory (HBM) chipset used for caching hotspot vectors. If it is determined that there is still remaining space, the target vector is directly read from the High Density Flash Memory (HDF) chipset and written in parallel to the High Bandwidth Memory (HBM) chipset through the memory controller. If it is determined that the physical space is full, the embedded parameter with the lowest current adaptive heat value within the High Bandwidth Memory is forcibly retrieved as the proposed replacement sacrifice. If and only if satisfying The condition is that the adaptive heat value of the target hotspot vector that triggers the migration is higher than the sacrificial term to be replaced. When setting the adaptive heat value, the sacrifice item will be... The physical weights are asynchronously written back to the high-density flash (HDF) chipset via the flash controller. Subsequently, the target hotspot vector triggering the migration is written into the freed-up high-bandwidth memory physical space, and the on-chip global residency table is updated in real-time via streaming. The time delay overhead model and energy consumption overhead model of the cross-media migration process satisfy the following: as well as The parameters correspond to the hardware latency and energy consumption of reading from high-density flash memory, writing to high-bandwidth memory, and updating metadata. The scheduling mapping management unit (SMMU) updates the global resident table according to the adaptive partitioning result to construct the physical mapping relationship and distribute it to complete the hierarchical storage initialization of the target hotspot vector and long tail vector.

[0087] Step 2, Request Routing and Multi-Level Near-Memory Instruction Generation: The input / output die (I / O Die) determines the physical location of the target embedding vector by concurrently retrieving the global residency table based on the received sparse training index request using its internal Scheduling Mapping Management Unit (SMMU). The Protocol Converter, configured adjacent to the SMMU, parses the original sparse training index request into multi-level near-memory instructions with a regularized hardware field format. These instructions are then distributed in parallel via die-to-die links to the corresponding hardware paths of the High Bandwidth Memory (HBM) chipset or the High Density Flash Memory (HDF) chipset. Subsequent steps then perform corresponding hierarchical near-data processing based on the value of the Path Selection Factor field (PathSel), thus completing the request regularization routing before feature extraction to improve bus utilization. Specifically, this includes the following steps:

[0088] Step 2.1, Concurrent Physical Residence Location Retrieval: During the bidirectional iterative training of the Deep Learning Recommendation Model (DLRM), the Input / Output Die (I / O Die) receives sparse training index requests distributed by the external host. The Scheduling Mapping Management Unit (SMMU) uses on-chip hardware counters and retrieval circuits to concurrently retrieve the real-time maintained on-chip global residence table for the sparse training index requests, so as to accurately determine the physical residence location results of each target embedding vector in the sparse training index request in the High Bandwidth Memory (HBM) chipset or High Density Flash Memory (HDF) chipset, and sends the physical residence location results to the adjacent Protocol Converter.

[0089] Step 2.2, Hard-core Decoding and Reassembly of Near-Memory Instructions: The Protocol Converter receives the physical residency location result issued in Step 2.1, and immediately performs hard-core decoding and reassembly parsing on the original sparse training index request to generate multi-level near-memory instructions with a regular hardware field format; the multi-level near-memory instructions completely and strictly include: opcode field (OpCode), table identifier field (TableId), physical embedding item index field (EmbIndex), path selection factor field (PathSel), and address field (Addr);

[0090] Step 2.3, Multi-field instruction bus distribution and submission: The generated multi-level near-memory instruction stream is distributed in parallel across the bus layer via a low-latency die-to-die link. Based on the hardware indication status of the path selection factor field (PathSel) in the multi-level near-memory instruction, the stream is routed and submitted to the corresponding high-bandwidth memory (HBM) chipset physical path or high-density flash memory (HDF) chipset physical path, thereby completing the request normalization routing before feature extraction.

[0091] Step 3: Hierarchical near-data processing in parallel execution: The high-bandwidth memory (HBM) chipset or the high-density flash memory (HDF) chipset initiates distributed differential feature extraction and parallel reduction at the near end of the medium based on the path selection factor field (PathSel) in the received multi-level near-memory instructions, obtaining all local parts and results of the in-memory computing (IMP) module and the flash memory computing (IFP) module; in this step, the differential feature extraction specifically includes:

[0092] When the path selection factor field (PathSel) in the multi-level near memory instruction indicates a flow to a hotspot path, the memory computing (IMP) module within the high bandwidth memory (HBM) chipset uses the vector processing unit (VPU) to extract the hotspot vector partitioned in step 1 in-situ at the near end of the medium, and calls the internal cascaded hardware adder tree to perform partial sum (Psum) aggregation and reduction calculations in parallel.

[0093] When the path selection factor field (PathSel) in the multi-level near-memory instruction indicates a flow to the long-tail path of the flash memory, the flash computing (IFP) module within the high-density flash (HDF) chipset processes the long-tail vector divided in step 1 locally by the flash computing unit (PU) deeply embedded within each flash die. Before issuing a read request to the external physical flash memory array, it uses the table identifier field (TableId) and the physical embedded item index field (EmbIndex) of the current request to perform data matching and retrieval in its built-in on-chip write buffer. If a match is found, the latest logical copy data temporarily stored in the write buffer is directly called to participate in forward summation and reduction to skip the physical array read. If the retrieval fails, a non-volatile page read operation is initiated to the underlying physical flash memory array through the underlying first-in-first-out hardware queue (FIFO) and control state machine, and the original feature vector is streamed into the data register group. Then, the dedicated local computing unit (Compute) within the flash computing unit (PU) performs the data matching and retrieval. The Unit and the high-speed multiplexer (MUX) accurately extract the sparse subvectors required for the current batch, following the local partial sum aggregation reduction formula: Execute local partials and aggregations, where This represents the sparse subvector entity that has been precisely extracted. This represents the set of sparse subvectors within the current flash memory die that are activated by the current batch of instructions;

[0094] Pipeline parallel optimization: During the execution of the long tail path, the flash computing unit (PU) makes full use of the characteristics of 16 independent flash data physical channels and adopts an instruction-level pipeline overlapping scheduling strategy. While performing the vector aggregation calculation of the current batch, the instruction queue inside the input / output die (I / O Die) pre-initiates the flash page streaming pre-read of the long tail items of the subsequent batch.

[0095] Step 4, Global Aggregation and Perceptron Forward Computation: The Base Die collects all local parts and results from the In-Memory Computing (IMP) module and the Flash Computing (IFP) module returned in Step 3. After performing physical dimension alignment, feature concatenation, and unified representation transformation on the local parts and results, a complete global embedded vector data entity is obtained. This global embedded vector data entity is then fed into the Multilayer Perceptron (MLP) engine to obtain the activation output prediction results for each neural network layer. Finally, the entire forward propagation result containing the activation output prediction results is output. In other words, after the hotspot path and long-tail path complete their local part aggregation, only the obtained low-dimensional parts and results are aggregated. The data is streamed via on-chip network (NoC) or cross-chip interconnect bus to the global buffer of the base die at the bottom of the high-bandwidth memory (HBM) chipset, following a global aggregation formula. The data is then concatenated and assembled into a complete global embedding vector to alleviate the bus bandwidth bottleneck in the feature extraction process at the chip architecture design level and compress the total amount of data transmitted to the memory chip. The process includes the following steps:

[0096] Step 4.1, Heterogeneous Sparse Feature Dimension Alignment and Representation Transformation: The Base Die collects all local parts and results from the In-Memory Computing (IMP) module and the Flash Computing (IFP) module returned in Step 3, and temporarily stores them in the on-chip Data Buffer. Then, the dedicated Process Engine is called to perform the final physical dimension alignment, feature splicing and unified representation transformation on these local parts and results, so as to assemble them into the complete global embedded vector data entity required for the current batch.

[0097] Step 4.2, Fully Connected Dense Feature Matrix Multiplication and Addition Fusion Calculation: The aligned global embedded vector data entities are streamed into the adjacent Multilayer Perceptron (MLP) engine; the MLP engine, through its internally configured hard-core systolic array, performs forward propagation (FP) calculation of fully connected matrix multiplication and addition fusion on the assembled global embedded vector data entities and the continuous dense feature stream synchronously input from the host; the MLP engine uses the hard-core systolic array to perform general multilayer neural network matrix multiplication and addition and activation function mapping, obtaining the activation values ​​corresponding to each neural network layer layer by layer, and finally obtaining and deriving the activation output prediction result of the current training batch;

[0098] Step 4.3: The on-chip control logic directly coordinates with the activation output prediction result and the current training batch size to perform a global summation according to the preset training loss function, directly calculating the global training loss value within the chip; finally, the entire forward propagation result containing the activation output prediction result and the global training loss value is output through the input / output die (I / O Die).

[0099] Step 5, Backward Gradient Propagation and Hardware-Software Coordinated Asynchronous Weight Update: Automatically triggered after forward propagation and loss difference calculation, this step involves distributed, differentiated in-situ overwrite updates performed near each storage medium through hardware-software coordination. Specifically, it includes the following steps:

[0100] Step 5.1, Distributed Gradient Flow Cross-Die Neck Distribution Scheduling: The Multilayer Perceptron (MLP) engine utilizes its internally configured hardware systolic array to perform general multilayer neural network inverse matrix multiplication and addition calculations, and initiates backpropagation (BP) calculations within the chip. Based on a preset general deep learning recommendation model (DLRM) in this field, it trains a cross-entropy loss function as boundary conditions, and calculates the sparse parameter gradients of the embedding table parameters of each layer corresponding to the current batch in real time and stores them in the data buffer. The gradient data is then distributed across dies to the corresponding storage media under the scheduling of the global resident table retrieved by the scheduling mapping management unit (SMMU) within the input / output die.

[0101] Step 5.2, Local flushing of near-end parameters of memory hotspot path: When the gradient data is distributed to the memory hotspot path, the memory computing (IMP) module in the high bandwidth memory (HBM) chipset receives the gradient, uses the vector processing unit (VPU) in conjunction with the local vector buffer, and directly calls its internally integrated arithmetic logic unit (ALU) in conjunction with the local inward buffer physical control to perform gradient multiplication and addition application locally in the memory chip according to the preset optimizer algorithm, and completes the weight update of the hotspot vector divided in step 1 locally, and directly streams it back to the underlying connected bank.

[0102] Step 5.3, Decoupled Asynchronous Batch Programming Write of Long-Tail Flash Path: When the gradient data is distributed to the long-tail flash path, the flash computing (IFP) module in the high-density flash (HDF) chipset routes the gradient to the gradient buffer of the flash computing unit (PU). The write buffer is used to accumulate the gradient data with the temporarily stored old parameter data, thereby completing the local logical overwrite and merging of the long-tail vectors partitioned in Step 1. The system adopts a strategy of complete decoupling between logical updates and physical updates. The on-chip circuit determines that the amount of unsynchronized data temporarily stored in the write buffer satisfies the conditional expression: When the standard physical page full-page granularity threshold of the physical flash memory is reached, the control state machine centrally activates the flash memory controller and initiates a batch of one-time physical page programming write operations using multiple independent flash data physical channels. This converts discrete random writes inside the chip into spatially continuous and page-aligned batch sequential writes, centrally executing flash page physical programming operations to complete the asynchronous update of the underlying physical array. This fundamentally eliminates the flash write amplification effect and extends the physical lifespan of the heterogeneous accelerator.

[0103] To implement and evaluate this application, dedicated hardware modules (including vector processing units, flash memory computing units, dedicated processing engines, and multilayer perceptron engines) were implemented using Verilog RTL. The synthesis, placement, and routing of these hardware modules were completed under the constraints of TSMC's 12nm CMOS process. A behavioral-level simulation platform for the heterogeneous multi-core accelerator was built based on the DRAMSim3 and SimpleSSD simulators. The key parameter configurations of the accelerator are shown in Table 1.

[0104] Table 1

[0105]

[0106] To further evaluate the performance of the training accelerator described in this application, this embodiment introduces three real-world large-scale deep learning recommendation models with different parameter sizes and feature vector dimensions as benchmark loads, labeled RM1, RM2, and RM3, for bidirectional iterative training testing. The total embedded storage capacity of the benchmark loads spans from 38.1GB to 343.3GB, and the parameter memory access request sequences of each load follow a high-skew, long-tail distribution pattern to quantitatively verify the bandwidth throughput and energy efficiency performance of the heterogeneous storage system in discrete sparse memory access scenarios. The specific topology model parameter configurations for each benchmark load are detailed in Table 2.

[0107] Table 2

[0108]

[0109] To execute the trained DLRM model and evaluate its performance, this application designed four comparative experiments: 1) Baseline (traditional heterogeneous computing benchmark), which uses an AMD Ryzen 7950X CPU, a high-performance NVIDIA GPU, and large-capacity DDR4 memory to build a memory-centralized heterogeneous computing node; 2) NDRec (near-memory solid-state drive solution), which uses a smart solid-state drive with integrated field-programmable gate array computing units based on the CXL protocol interconnect. Its solid-state drive portion is simulated based on the Samsung 983DCT 3.84TB specification and equipped with a dedicated DDR4-2400 memory cache; 3) COFAL-NWB (ablation control solution), which explicitly removes the on-chip write buffer inside the high-density flash memory chip in the system microarchitecture to isolate and evaluate the performance contribution of the write buffer mechanism; 4) COFAL, which is the solution in this application, fully integrates a memory computing module, a flash memory computing unit, and an on-chip write buffer component with write traffic logic merging function.

[0110] like Figure 6-7 As shown, under different recommendation model workloads, the proposed solution achieves throughput speedups of up to 3.15x and 1.46x compared to the fully distributed memory baseline solution and the near-memory solid-state drive solution (NDRec), respectively. Regarding energy efficiency...

[0111] like Figure 8 As shown, this application achieves a reduction in total system energy consumption of up to 79% compared to the fully distributed memory baseline scheme for large-scale long-tail recommendation model training scenarios.

[0112] The area and power consumption overhead brought about by the custom hardware module introduced in this application are within a controllable range: the near-end vector processing and routing control module only increases the total physical area of ​​the heterogeneous chipset by less than 1% to 2%, and the total power consumption overhead of the whole machine during bidirectional iterative operation increases by only about 2% to 2.4%, and its power consumption increase is completely offset by the reduction of cross-medium communication power consumption.

[0113] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A DLRM training accelerator based on heterogeneous multi-core, characterized in that: The DLRM training accelerator includes an input / output die, a high-bandwidth memory chipset, and a high-density flash memory chipset. The high-bandwidth memory chipset and the high-density flash memory chipset are connected to the input / output die via inter-die interconnect links. The input / output die integrates a scheduling mapping management unit, a flash memory controller, a protocol converter, and a cross-media interconnect interface, which includes a high-bandwidth memory interface and a high-density flash memory interface.

2. The DLRM training accelerator based on heterogeneous multi-core particles according to claim 1, characterized in that: The high-bandwidth memory chipset adopts a three-dimensional stacked structure, including a substrate die and several memory computing dies, which are stacked on top of the substrate die through vertical through-silicon vias; Each of the memory computing die is physically and logically divided into 32 independent repositories, and each of the memory computing die integrates 16 vector processing units in situ, which are located adjacent to the repositories.

3. The DLRM training accelerator based on heterogeneous multi-core particles according to claim 2, characterized in that: The substrate die integrates a computing subsystem, a buffer subsystem, and an on-chip network. The on-chip network connects the computing subsystem and the buffer subsystem, wherein: The computing subsystem integrates a dedicated data register, a dedicated processing engine, and a multilayer perceptron engine. The buffer subsystem includes a data buffer for caching operands and hot embeddings, and a global buffer for managing cross-grain data exchange; The on-chip network is equipped with a near-memory router, a memory controller, and a unified controller, enabling high-speed routing and coordinated scheduling of internal instructions and data.

4. The DLRM training accelerator based on heterogeneous multi-core particles according to claim 2, characterized in that: The vector processing unit includes a computing component, a storage component, and a control and switching component, wherein: The computing component includes a cascaded hardware addition tree and an arithmetic logic unit; The storage components include a local vector buffer, a gradient buffer, and a partial sum buffer; The control and switching component includes a hardware switching switch, which internally contains an instruction register set for specifying the target repository and operand range according to the received instructions.

5. The DLRM training accelerator based on heterogeneous multi-core particles according to claim 1, characterized in that: The high-density flash memory chipset includes Each flash data physical channel has an independent flash data physical channel, and each flash data physical channel is connected to multiple flash chips. Each flash chip contains multiple flash dies, and each flash die integrates a flash computing unit and a low-level controller.

6. The DLRM training accelerator based on heterogeneous multi-core according to claim 5, characterized in that: The flash memory computing unit includes: Local computing components: including high-speed multiplexers and dedicated local computing units; Buffer system: includes flash-side read buffer, flash-side gradient buffer, and write buffer with physical write traffic shaping and integration functions, used to realize data temporary storage, gradient alignment, logical merging of updated data, and batch write-back; The underlying controller includes a first-in-first-out hardware queue, an instruction register, an address register, and a dedicated control state machine. The dedicated control state machine includes independent calculation control, write control, and read control units.

7. A DLRM training acceleration method using the heterogeneous multi-core DLRM training accelerator as described in any one of claims 1-6, characterized in that: The acceleration method specifically includes the following steps: Step 1, Data Mapping and Adaptive Hotspot Partitioning Initialization: The input / output die receives input layer data consisting of embedded tables distributed by the host, and the scheduling mapping management unit configured inside the input / output die completes the hierarchical storage initialization of the target hotspot vector and long tail vector; Step 2, Request routing and multi-level near-memory instruction generation: The input / output dies, based on the received sparse training index request, concurrently retrieve the global residency table by the internal scheduling mapping management unit to determine the physical residency position of the target embedding vector; The protocol converter configured adjacent to the scheduling mapping management unit parses the original sparse training index request into a multi-level near memory instruction with a regular hardware field format. The multi-level near memory instruction is distributed in parallel to the corresponding hardware path of the high-bandwidth memory chipset or the high-density flash memory chipset via the inter-die interconnect link, so that subsequent steps can perform corresponding hierarchical near data processing according to the value of the path selection factor field. Step 3, Hierarchical near-data processing in parallel execution: The high-bandwidth memory chipset or the high-density flash memory chipset initiates distributed differential feature extraction and parallel reduction at the near end of the medium according to the path selection factor field in the received multi-level near-memory instructions, to obtain all local parts and results of the memory computing module and the flash memory computing module; Step 4, Global Aggregation and Perceptron Forward Computation: The substrate grain collects all local parts and results from the memory computing module and the flash computing module returned in Step 3. After performing physical dimension alignment, feature concatenation and unified representation transformation on the local parts and results, a complete global embedded vector data entity is obtained. The global embedded vector data entity is sent into the multilayer perceptron engine to obtain the activation output prediction results corresponding to each neural network layer. Finally, the entire forward propagation result containing the activation output prediction results is output. Step 5, Backward Gradient Backpropagation and Software-Hardware Coordinated Asynchronous Weight Update: Automatically triggered after forward propagation and loss difference calculation, distributed differentiated in-situ overlay update is performed near each storage medium through software and hardware coordination.

8. The DLRM training acceleration method based on heterogeneous multi-core particles according to claim 7, characterized in that: Step 2 specifically includes the following steps: Step 2.1, Concurrent Retrieval of Physical Dwelling Location: During the bidirectional iterative process of training the Deep Learning Recommendation Model (DLRM), the input / output die receives sparse training index requests distributed by the external host. The scheduling and mapping management unit uses on-chip hardware counters and retrieval circuits to concurrently retrieve the real-time maintained on-chip global residency table for sparse training index requests, so as to accurately determine the physical residency position of each target embedding vector in the sparse training index request in the high-bandwidth memory chipset or high-density flash memory chipset, and send the physical residency position result to the adjacent protocol converter. Step 2.2, Hard-core decoding and reassembly of near-memory instructions: The protocol converter receives the physical residency location result issued in Step 2.1, and immediately performs hard-core decoding and reassembly parsing on the original sparse training index request to generate multi-level near-memory instructions with a regular hardware field format; the multi-level near-memory instructions completely and strictly include: opcode field, table identifier field, physical embedding item index field, path selection factor field, and address field; Step 2.3, Multi-field instruction bus distribution and submission: The generated multi-level near-memory instruction stream is distributed in parallel across the bus layer via low-latency inter-die interconnect links, and is routed and submitted to the corresponding high-bandwidth memory chipset physical path or high-density flash memory chipset physical path according to the hardware indication status of the path selection factor field in the multi-level near-memory instruction, thereby completing the request normalization routing before feature extraction.

9. The DLRM training acceleration method based on heterogeneous multi-core particles according to claim 7, characterized in that: In step 3, the differential feature extraction specifically involves: When the path selection factor field in the multi-level near memory instruction indicates that the flow is directed to the hot spot path, the memory computing module in the high-bandwidth memory chipset uses the vector processing unit to extract the hot spot vector divided in step 1 in-situ at the near end of the medium, and calls the internal cascaded hardware addition tree parallel execution part and aggregation reduction calculation. When the path selection factor field in the multi-level near-memory instruction indicates a flow to the long-tail path of flash memory, the flash computing module within the high-density flash chipset processes the long-tail vector divided in step 1 locally by the flash computing units deeply embedded within each flash die. Before issuing a read request to the external physical flash memory array, it uses the table identifier field and the physical embedded item index field of the current request to perform data matching and retrieval on its built-in on-chip write buffer. If a match is found, the latest logical copy data temporarily stored in the write buffer is directly called to participate in forward summation and reduction to skip the physical array read. If the retrieval fails, a non-volatile page read operation is initiated to the underlying physical flash memory array through the underlying first-in-first-out hardware queue and control state machine. The original feature vector is streamed to the data register group, and then the dedicated local computing unit and high-speed multiplexer within the flash computing unit accurately extract the sparse sub-vectors required for the current batch, following the local partial sum aggregation reduction formula: Perform local partial and aggregate operations, where This represents the sparse subvector entity that has been precisely extracted. This represents the set of sparse subvectors within the current flash memory die that are activated by the current batch of instructions; Pipeline parallel optimization: During the execution of the long tail path, the flash computing unit makes full use of the characteristics of 16 independent flash data physical channels and adopts an instruction-level pipeline overlapping scheduling strategy. While performing the vector aggregation calculation of the current batch, the instruction queue inside the input / output die initiates the flash page streaming pre-read of the long tail items of the subsequent batch.

10. The DLRM training acceleration method based on heterogeneous multi-core particles according to claim 7, characterized in that: Step 5 specifically includes the following steps: Step 5.1, Distributed Gradient Flow Cross-Chip Distribution Scheduling: The multilayer perceptron engine utilizes its internally configured hardware systolic array to perform general multilayer neural network inverse matrix multiplication and addition calculations, and initiates backpropagation calculations within the chip. Based on the preset deep learning recommendation model, it trains a cross-entropy loss function as boundary conditions, and calculates the sparse parameter gradients of the embedding table parameters of each layer corresponding to the current batch in real time and stores them in the data buffer. The gradient data is distributed across chips to the corresponding storage media under the scheduling of the global resident table retrieved by the scheduling mapping management unit within the input / output chip. Step 5.2, Local flushing of near-end parameters of memory hotspot path: When the gradient data is distributed to the memory hotspot path, the memory computing module in the high-bandwidth memory chipset receives the gradient, uses the vector processing unit in conjunction with the local vector buffer, and directly calls its internally integrated arithmetic logic unit in conjunction with the local inward buffer physical control to perform gradient multiplication and addition application locally in the memory chip according to the preset optimizer algorithm, and completes the weight update of the hotspot vector divided in step 1 locally, and directly streams it back to the underlying connected storage repository; Step 5.3, Decoupled Asynchronous Batch Programming Write of Long-Tail Flash Path: When the gradient data is distributed to the long-tail flash path, the flash computing module in the high-density flash chipset routes the gradient to the gradient buffer of the flash computing unit. The gradient data is then accumulated with the temporarily stored old parameter data using the write buffer to complete the local logical overwrite and merging of the long-tail vector divided in Step 1. The system adopts a strategy of complete decoupling between logical updates and physical updates. The on-chip circuit determines that the amount of unsynchronized data temporarily stored in the write buffer satisfies the conditional expression: When the standard physical page full-page granularity threshold of the physical flash memory is reached, the control state machine centrally activates the flash memory controller and initiates a batch of one-time physical page programming write operations using multiple independent flash data physical channels. This converts discrete random writes inside the chip into spatially continuous and page-aligned batch sequential writes, centrally executing flash page physical programming operations to complete the asynchronous update of the underlying physical array. This fundamentally alleviates the flash write amplification effect and extends the physical lifespan of the heterogeneous accelerator.

Images