Methods and devices for accelerating reasoning in large language models

Abstract

This invention proposes a method to accelerate large language model inference, applied to a near-memory computing architecture. The method includes: rearranging the weights of the large language model based on the co-activation characteristics of neurons, placing the weights corresponding to neurons with high co-activation frequencies adjacent to each other in the storage unit; during inference runtime, dynamically adjusting the splitting ratio of computational tasks between the computing unit and the storage unit according to a real-time activation sparsity mask, thereby optimizing the load distribution between the computing unit and the storage unit. This method significantly improves the performance of the near-memory computing architecture in large language model inference scenarios with sparse activation, and enhances hardware utilization.

Description

Technical Field

[0001] This invention relates to the field of in-memory computing technology, and in particular to a method and apparatus for accelerating reasoning in large language models. Background Technology

[0002] In recent years, edge-based large language model inference architectures based on chiplets and near-memory computing (such as Cambricon-LLM) have effectively alleviated data transmission bandwidth bottlenecks and improved inference efficiency by embedding computing units inside flash memory. This type of architecture stores model weights in flash memory, and matrix-vector multiplication operations are performed collaboratively by the neural network processor (NPU) and flash memory, enabling efficient inference of large-parameter models on edge devices.

[0003] Studies have shown that LLM exhibits significant activation sparsity during inference, with most elements in the activation vector having minimal impact on the final output. These elements can be safely pruned, skipping the computation and transmission of corresponding weights, thereby reducing data transmission and computational overhead. However, activation sparsity leads to dynamic and discontinuous weight access patterns, severely limiting the practical performance improvements brought about by sparsification.

[0004] The existing near-memory computing architecture has the following problems in the activation sparse scenario: (1) Discontinuous weight access: Since activation sparsity is input adaptive, the position of the weight column skipped in each inference changes dynamically, which makes it impossible for Flash read operations to take advantage of continuous memory access, increasing memory access latency. (2) Low hardware resource utilization: The computation and data transmission in the sparse activation mode are unbalanced, and the existing static task allocation strategy cannot adapt to the dynamic sparse mode, resulting in a decrease in the cooperation efficiency between NPU and Flash. (3) Large amount of invalid data transmission: Although sparsification can theoretically reduce weight loading, due to the unoptimized weight storage layout, a large number of invalid weights corresponding to zero-value activation will still be loaded in the actual reading process, wasting transmission bandwidth. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a method and apparatus for accelerating large language model reasoning, which significantly improves the performance of near-memory computing architectures in large language model reasoning scenarios with sparse activation.

[0006] This invention provides a method for accelerating inference in large language models, applied to near-memory computing architectures, including:

[0007] The weights of a large language model are rearranged based on the co-activation characteristics of neurons, and the weights corresponding to neurons with high co-activation frequency are placed adjacent to each other in the storage unit.

[0008] During inference runtime, the splitting ratio of computing tasks between computing units and storage units is dynamically adjusted based on the real-time activated sparse mask, thereby optimizing the load distribution between computing units and storage units.

[0009] In one embodiment of the present invention, the step of rearranging the weights of a large language model based on the co-activation characteristics of neurons, and placing the weights corresponding to neurons with high co-activation frequencies adjacent to each other in the storage unit, includes:

[0010] The storage space of the storage unit is abstracted into a linear address space. With the goal of increasing the probability of obtaining effective weights in a single read operation, the weights corresponding to neurons whose co-activation frequency exceeds a preset frequency threshold are placed and stored in adjacent positions in the linear address space.

[0011] In one embodiment of the present invention, the weight rearrangement problem is formalized as the maximum weight Hamiltonian path finding problem in a complete graph, with the goal of maximizing the sum of path weights.

[0012] In one embodiment of the present invention, the maximum weight Hamiltonian path finding problem is solved using a heuristic greedy algorithm, including:

[0013] Construct a max-heap with co-activation frequencies as weights;

[0014] Based on the maximum heap, neurons with a co-activation frequency exceeding the preset frequency threshold are selected greedily in descending order of frequency.

[0015] Generate a chain-weighted rearranged sequence based on the selected neurons.

[0016] In one embodiment of the present invention, the step of dynamically adjusting the partitioning ratio of the computing task between the computing unit and the storage unit based on the real-time activated sparse mask includes:

[0017] A lightweight cost model is established, taking the real-time activation sparse mask and the candidate task block size as inputs, to predict the execution time under different task allocation schemes;

[0018] Based on the lightweight cost model, the optimal task block size is determined through constraint optimization, and the partitioning ratio of the computing task between the computing unit and the storage unit is adjusted according to the optimal task block size.

[0019] In one embodiment of the present invention, the lightweight cost model is expressed as:

[0020]

[0021] Where m represents the sparse mask of the current activation vector. and These are the execution times for read requests from the computing unit and read requests from the storage unit, respectively. Candidate block size, This represents the expected execution time of the output.

[0022] In one embodiment of the present invention, the optimal task block size is determined by solving the following constrained optimization problem, expressed as:

[0023]

[0024] in, To determine the optimal task block size. This represents the total number of computing cores in the computing unit. This is the page size of the storage unit.

[0025] In one embodiment of the present invention, the near-memory computing architecture includes the edge-end large language model inference architecture Cambricon-LLM architecture, the computing unit includes an NPU neural network processor, and the storage unit includes Flash memory.

[0026] In another aspect, the present invention also provides a large language model inference acceleration device, comprising:

[0027] The weight rearrangement module is used to rearrange the weights of a large language model based on the co-activation characteristics of neurons, placing the weights corresponding to neurons with high co-activation frequency adjacent to each other in the storage unit.

[0028] The dynamic partitioning and scheduling module is used to dynamically adjust the partitioning ratio of computing tasks between computing units and storage units based on the real-time activation sparse mask during inference runtime, thereby optimizing the load distribution between computing units and storage units.

[0029] In one embodiment of the present invention, the weight rearrangement module is run as a preprocessing step of model loading and stores the rearranged weights in the storage unit; the dynamic partitioning and scheduling module is integrated into the runtime driver of the near-memory computing architecture.

[0030] As can be seen from the above solutions, the advantages of the present invention are:

[0031] The present invention provides a method and apparatus for accelerating large language model inference. Based on the co-activation characteristics of neurons, the weights of the large language model are rearranged, placing the weights corresponding to neurons with high co-activation frequencies adjacent to each other in the storage unit. During inference, the computational task is dynamically adjusted between the computation unit and the storage unit according to a real-time activation sparsity mask, optimizing the load distribution between the computation unit and the storage unit. This invention significantly improves the performance of near-memory computing architecture in large language model inference scenarios with sparse activation: it improves the speedup ratio on typical large language models, significantly increasing inference speed; it effectively reduces the transmission of invalid weight data, saving memory bandwidth; dynamic partitioning significantly improves the utilization rate of storage unit channels, making the collaboration between the computation unit and the storage unit more efficient, and significantly improving hardware utilization. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a method for accelerating reasoning in large language models according to an embodiment of the present invention is shown.

[0033] Figure 2 It shows Figure 1 A detailed flowchart of step S2 is shown below;

[0034] Figure 3 This diagram illustrates a flowchart of a large language model inference acceleration device according to an embodiment of the present invention.

[0035] Figure 4 The end-to-end performance comparison of various language models under the Cambricon-LLM architecture in dense inference and sparse inference is shown.

[0036] Figure 5 The performance of various language models under the Cambricon-LLM architecture in dense and sparse inference under different inference modes is shown; among them Figure 5 Figure (a) shows the amount of data transmitted, and Figure (b) shows the bandwidth utilization.

[0037] The attached figures are labeled as follows:

[0038] 100: Calculation unit;

[0039] 200: Storage unit;

[0040] 300: Large Language Model Reasoning Accelerator;

[0041] 310: Weight rearrangement module;

[0042] 320: Dynamic splitting and scheduling module. Detailed Implementation

[0043] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0044] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0045] Performance analysis of near-memory computing architectures (such as the Cambricon-LLM architecture) in sparse activation scenarios reveals that although the activation sparsity rate can reach 40%-50%, the actual acceleration effect is far lower than the theoretical value due to the mismatch between weight storage layout and access patterns. Further research found a significant co-activation phenomenon among neurons, meaning that specific groups of neurons tend to be activated simultaneously. Therefore, this application considers rearranging weights during the storage phase based on neuron co-activation characteristics, placing weights corresponding to neurons with high co-activation probabilities adjacent to each other in physical storage. This can increase the proportion of effective weights in a single read during sparse activation. Simultaneously, by dynamically adjusting the allocation ratio of computing tasks between computing units (such as the NPU) and storage units (such as Flash) in conjunction with runtime sparse modes, a more balanced utilization of hardware resources can be achieved.

[0046] Specifically, refer to Figure 1 As shown, Figure 1 A flowchart illustrating a method for accelerating reasoning in large language models according to an embodiment of the present invention is shown.

[0047] A method for accelerating inference in large language models based on sparse activation, applied to near-memory computing architectures, includes the following steps:

[0048] Step S1: Based on the co-activation characteristics of neurons, rearrange the weights of the large language model and place the weights corresponding to neurons with high co-activation frequency adjacent to each other in the storage unit.

[0049] Step S2: During inference runtime, dynamically adjust the splitting ratio of computing tasks between computing units and storage units based on the real-time activated sparse mask, thereby optimizing the load distribution between computing units and storage units.

[0050] It should be noted that near-memory computing architecture is a general-purpose technical architecture in which computing units and storage units are tightly coupled. Its core feature is to alleviate data transmission bandwidth bottlenecks by shortening the physical distance between computing and storage. Examples include the Cambricon-LLM architecture, but it is not limited to this architecture. Other architectures such as the near-memory acceleration module of the IBM Telum processor, the near-memory extension architecture of the Google TPUv4, the Cubelet near-memory design of the Huawei Ascend 910B, and the in-memory computing architecture of the Graphcore IPU-M series are also applicable.

[0051] The computing unit is a general-purpose / dedicated computing unit with matrix operation capabilities required for large language model inference, such as an NPU neural network processor, but not limited to it; the storage unit is a short-range storage medium that supports block / page reading, such as Flash memory, but not limited to it.

[0052] In one embodiment, to optimize the weight layout in the storage unit during the model loading stage, in step S1, the weights of the large language model are rearranged based on the co-activation characteristics of neurons, placing the weights corresponding to neurons with high co-activation frequencies adjacent to each other in the storage unit. Specifically, by abstracting the storage space of the storage unit into a linear address space, with the goal of increasing the probability of obtaining effective weights in a single read operation, the weights corresponding to neurons with co-activation frequencies exceeding a preset frequency threshold are selected and placed in adjacent positions in the linear address space for storage, thereby increasing the probability of obtaining effective weights in a single flash memory read operation.

[0053] Specifically, the weight rearrangement problem is formalized as the problem of finding the maximum weight Hamiltonian path in a complete graph, namely, finding a Hamiltonian path that maximizes the sum of path weights. The complete graph is defined by the co-activation frequency matrix. Defined as follows, where N is the number of neurons. This represents the co-activation frequency between neurons i and j. Neurons i and j form a neuron pair, consisting of several Constructing the co-activation frequency matrix 1 ≤ i ≤ j ≤ N. Each neuron corresponds to a node in the graph, with a total of N nodes (N being the number of neurons); a neuron pair (i, j) corresponds to an undirected edge connecting node i and node j in the graph, representing a co-activation association between the two neurons; the weight of the edge (i, j) is the co-activation frequency. , which is the number of times neurons i and j are activated simultaneously during model inference. The higher the weight, the stronger the co-activation probability of the two neurons.

[0054] In one specific implementation, the maximum weight Hamiltonian path finding problem is solved using a heuristic greedy algorithm. The objective, maximizing the sum of path weights, is expressed as:

[0055]

[0056] in, Let it be the set of edges.

[0057] First, initialize the neighbor count and disjoint-set data structures, where the neighbor count data structure... ,in Disjoint-set data structure ,in and initialize the max heap. The disjoint-set data structure (Union-Find) is a classic data structure used for dynamic connectivity testing. sᵢ represents the connected component to which node i belongs. By using the union set, loops are avoided, ensuring the entire structure is a connected tree structure that can eventually be linked into a chain. This involves storing all neuron pairs (i,j) and their weights. Insert into the max heap In the middle, construct a max heap with co-activation frequency as the weight. , is represented as: Due to the frequency of co-activation If it is a non-negative integer, take its opposite. Edges with smaller values ​​(i.e., larger C[i,j]) have higher weights in a max-heap, so the top of the heap is the edge with the highest co-activation frequency. Taking the neuron pair with the highest co-activation frequency from the top of the heap is the core of the algorithm's "greedy" strategy: prioritizing the connection of the neuron pair with the highest co-activation frequency.

[0058] Then, based on the max-heap, neurons with co-activation frequencies exceeding the preset frequency threshold are greedily selected in descending order of frequency, while maintaining the degree constraint of the nodes; a chain-like weight rearrangement sequence is generated based on the selected neurons. The preset frequency threshold, for example, filters out edges with extremely low co-activation frequencies to reduce computation and avoid invalid connections. The threshold can be dynamically set according to model characteristics (such as the number of layers in an LLM, the number of neurons), or determined by statistically analyzing the quantiles of co-activation frequencies (e.g., selecting the top 30% of high-frequency edges). Specifically, the greedy construction of neuron connections begins with initializing the edge set. An empty set, i.e. From the max heap Select the neuron pair (i,j) with the largest weight, if degree constraint ,and Then merge the sets of i and j, that is Update neighbor count and And add edge (i,j) to the edge set. , Since the final result needs to generate a chain-like rearrangement sequence (corresponding to a Hamiltonian path in graph theory), the path structure requires that each node has at most two neighbors (two neighbors for nodes in the middle of the path, and one neighbor each for the start and end points). Therefore, the neighbor count does not exceed 2, ensuring that the final generated structure is a linear chain rather than a cycle or branching network. When sᵢ≠sⱼ, it means that nodes i and j belong to different connected components, and merging them will not form a cycle; the merging operation... This is a typical operation in disjoint-set data structures, merging two connected components into one to ensure the connectivity of the entire rearranged sequence. Choose any edge. Then, visit the unvisited neurons sequentially in a chain-like manner. That is, let... From the side Starting from any end (e.g., i), mark it as a visited node. Find another unvisited neighbor of node i (if it exists) as The neighboring nodes are visited recursively until all nodes have been visited, forming a chain structure; for the set of unvisited edges... Isolated nodes (typically neurons with extremely low co-activation frequency) are appended directly to the end of the sequence in their original order, ensuring that all neurons are included in the rearranged sequence, ultimately constructing a chain-like weighted rearranged sequence. This application employs a heuristic greedy algorithm that prioritizes connecting the neuron pairs with the highest co-activation frequency. It achieves high-quality solutions within a time complexity, compared to brute-force search. It has a significant advantage in terms of complexity.

[0059] In this embodiment, using the neuron co-activation frequency matrix as input, the weights corresponding to neurons with high co-activation frequencies are rearranged adjacently in the storage space. This method optimizes the physical layout of weights in the storage unit, enabling a single read operation to acquire more effective weights during sparse activation, reducing invalid data transmission, and improving access continuity.

[0060] In one embodiment, in step S2, during inference runtime, the partitioning ratio of computational tasks between the computation unit and the storage unit is dynamically adjusted based on the real-time activated sparse mask to optimize the load distribution between the computation unit and the storage unit, thereby optimizing hardware resource allocation. Specifically, refer to... Figure 2 As shown, Figure 2 A schematic diagram of the specific process for step S2 is shown.

[0061] The step of dynamically adjusting the partitioning ratio of the computing task between the computing unit and the storage unit based on the real-time activated sparse mask specifically includes the following steps:

[0062] Step S21: Establish a lightweight cost model, using the real-time activation sparse mask and candidate task block size as inputs, to predict the execution time under different task allocation schemes. The lightweight cost model is expressed as:

[0063]

[0064] Where m represents the sparse mask of the current activation vector. and These are the execution times for read requests from the computing unit and read requests from the storage unit, respectively. The execution time for read requests from the computing unit is... This includes the time it takes for the computation unit to read the activation mask and weight data from the storage unit, as well as the time it takes for the computation unit to perform computational operations such as matrix multiplication and activation functions. The execution time tr of the storage unit read request includes the time it takes for the storage unit to read the weight data and the time it takes to transfer the data to the computation unit via the bus. Candidate block size, The expected execution time is the output. This cost model uses candidate block sizes. The real-time sparse feature m is taken as input, and the output is the expected execution time.

[0065] It should be noted that the method for generating real-time activation sparse masks in this embodiment does not depend on a specific sparsification algorithm and can work in conjunction with various activation sparsification techniques (such as TEAL (Training-Free Activation Sparsity in Large Language Models)).

[0066] Step S22: Based on the lightweight cost model, determine the optimal task block size through constraint optimization, adjust the splitting ratio of computing tasks between the computing unit and the storage unit according to the optimal task block size, optimize the load distribution between the computing unit and the storage unit, and achieve load balancing.

[0067] The optimal task block size is determined by solving the following constrained optimization problem, expressed as:

[0068]

[0069] in, To determine the optimal task block size. This represents the total number of computing cores in the computing unit. This refers to the page size of a storage unit. Optimal block size. Dynamically guiding the load distribution between computing and storage units maximizes the utilization of hardware resources under sparse conditions. In this embodiment, adapting to the dynamic sparse mode maximizes the collaborative computing efficiency of computing and storage units, improving overall hardware resource utilization and inference speed.

[0070] In one embodiment, a large language model inference acceleration device is further provided. This embodiment is a device embodiment corresponding to the above embodiment, such as... Figure 3 As shown, Figure 3 A general structural block diagram of a large language model inference acceleration device according to an embodiment of the present invention is shown. This device embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments remain valid in this device embodiment, and will not be repeated here to avoid repetition.

[0071] A large language model inference acceleration device 300 includes:

[0072] The weight rearrangement module 310 is used to rearrange the weights of a large language model based on the co-activation characteristics of neurons, and to place the weights of neurons with high co-activation frequency adjacent to each other in the storage unit.

[0073] The dynamic partitioning and scheduling module 320 is used to dynamically adjust the partitioning ratio of computing tasks between computing units and storage units based on the real-time activation sparse mask during inference runtime, thereby optimizing the load distribution between computing units and storage units.

[0074] In one embodiment, the weight rearrangement module runs as a preprocessing step in model loading, storing the rearranged weights in the storage unit; the dynamic partitioning and scheduling module is integrated into the runtime driver of the near-memory computing architecture.

[0075] The technical effects of the present invention will be specifically verified below.

[0076] Experimental Setup: The Cambricon-LLM architecture was selected for near-memory computing. The weight rearrangement module was used as a preprocessing step during model loading, and the dynamic partitioning and scheduling module was integrated into the Cambricon-LLM runtime driver, forming a complete sparse inference acceleration system. Large language models selected were Llama2-7B / 13B, Llama3-8B, and Mistral-7B. The TEAL algorithm was used to generate activation masks with 40%-50% sparsity. The original Cambricon-LLM architecture was used to run both dense and sparse models as baselines. Figure 4 The end-to-end performance comparison of various language models under the Cambricon-LLM architecture in dense inference and sparse inference is shown. Figure 5The performance of various language models under the Cambricon-LLM architecture in dense and sparse inference under different inference modes is shown. Figure 5 (a) shows the amount of data transmitted between dense and sparse reasoning in various language models; Figure 5 Table (b) shows the bandwidth utilization of various language models in dense and sparse inference. Figures 4-5 As can be seen from experiments, on the sparse model, Cambricon-LLM-S / M / L achieves a speedup of 1.19 to 2.43 times compared to its dense mode, with an average speedup of 1.27 to 1.54 times. The rearrangement and partitioning strategies reduce data transmission volume by 0.2 to 5.0 GB; the larger the number of model parameters, the more significant the savings. The sparsity-aware partitioning strategy significantly improves channel bandwidth utilization.

[0077] Therefore, in summary, the large language model inference acceleration method and apparatus provided by this invention rearranges the weights of the large language model based on the co-activation characteristics of neurons, placing the weights corresponding to neurons with high co-activation frequencies adjacent to each other in the storage unit; during inference runtime, the splitting ratio of the computation task between the computation unit and the storage unit is dynamically adjusted according to the real-time activation sparse mask, optimizing the load distribution between the computation unit and the storage unit. This invention significantly improves the performance of the near-memory computing architecture in large language model inference scenarios with sparse activation: experimental verification shows a speedup of 1.27 to 1.54 times on typical large language models, significantly improving inference speed; effectively reducing the transmission of invalid weight data, saving memory bandwidth; the dynamic splitting strategy significantly improves the utilization rate of storage unit channels, making the collaboration between the computation unit and the storage unit more efficient, and significantly improving hardware utilization.

[0078] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for accelerating reasoning in large language models, applied to near-memory computing architectures, characterized in that, include: The weights of a large language model are rearranged based on the co-activation characteristics of neurons, and the weights corresponding to neurons with high co-activation frequency are placed adjacent to each other in the storage unit. During inference runtime, the splitting ratio of computing tasks between computing units and storage units is dynamically adjusted based on the real-time activated sparse mask, thereby optimizing the load distribution between computing units and storage units.

2. The method according to claim 1, characterized in that, The step of rearranging the weights of a large language model based on the co-activation characteristics of neurons, and placing the weights corresponding to neurons with high co-activation frequency adjacent to each other in the storage unit, includes: The storage space of the storage unit is abstracted into a linear address space. With the goal of increasing the probability of obtaining effective weights in a single read operation, the weights corresponding to neurons whose co-activation frequency exceeds a preset frequency threshold are placed and stored in adjacent positions in the linear address space.

3. The method according to claim 2, characterized in that, The weight rearrangement problem is formalized as the maximum weight Hamiltonian path finding problem in a complete graph, with the goal of maximizing the sum of path weights.

4. The method according to claim 3, characterized in that, The maximum weight Hamiltonian path finding problem is solved using a heuristic greedy algorithm, including: Construct a max-heap with co-activation frequencies as weights; Based on the maximum heap, neurons with a co-activation frequency exceeding the preset frequency threshold are selected greedily in descending order of frequency. Generate a chain-weighted rearranged sequence based on the selected neurons.

5. The method according to claim 1, characterized in that, The step of dynamically adjusting the partitioning ratio of the computing task between the computing unit and the storage unit based on the real-time activated sparse mask includes: A lightweight cost model is established, taking the real-time activation sparse mask and the candidate task block size as inputs, to predict the execution time under different task allocation schemes; Based on the lightweight cost model, the optimal task block size is determined through constraint optimization, and the partitioning ratio of the computing task between the computing unit and the storage unit is adjusted according to the optimal task block size.

6. The method according to claim 5, characterized in that, The lightweight cost model is expressed as follows: Where m represents the sparse mask of the current activation vector. and These are the execution times for read requests from the computing unit and read requests from the storage unit, respectively. Candidate block size, This represents the expected execution time of the output.

7. The method according to claim 6, characterized in that, The optimal task block size is determined by solving the following constrained optimization problem, expressed as: in, To determine the optimal task block size. This represents the total number of computing cores in the computing unit. This is the page size of the storage unit.

8. The method according to claim 1, characterized in that, The near-memory computing architecture includes the Cambricon-LLM architecture, a large language model inference architecture at the edge. The computing unit includes an NPU neural network processor, and the storage unit includes Flash memory.

9. A large language model reasoning acceleration device, characterized in that, include: The weight rearrangement module is used to rearrange the weights of a large language model based on the co-activation characteristics of neurons, placing the weights corresponding to neurons with high co-activation frequency adjacent to each other in the storage unit. The dynamic partitioning and scheduling module is used to dynamically adjust the partitioning ratio of computing tasks between computing units and storage units based on the real-time activation sparse mask during inference runtime, thereby optimizing the load distribution between computing units and storage units.

10. The apparatus according to claim 9, characterized in that, The weight rearrangement module runs as a preprocessing step for model loading, storing the rearranged weights in the storage unit; the dynamic partitioning and scheduling module is integrated into the runtime driver of the near-memory computing architecture.

Images