In-memory computing device, end-side ai inference method, medium, and terminal

Abstract

The application provides a storage-computation integrated device, an end-side AI inference method, a medium and a terminal. Through direct interconnection of a storage group and an AI computation module, register transfer and parallel data carrying mechanism, system performance and energy efficiency are improved, storage delay and bandwidth are optimized, serial-parallel conversion overhead is saved, cache management overhead is avoided, power consumption and area of prefetch logic are saved, complexity of the storage system itself and data carrying is reduced, and storage carrying is lightweighted. In the process of executing a distributed AI inference operation, data carrying and shaping can be synchronously performed, data synchronization parallelism is improved, data supply rate is matched with computation rate, idle waiting of an AI computation unit is reduced, and system throughput and energy efficiency ratio are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an in-memory computing device, an edge AI inference method, a medium, and a terminal. Background Technology

[0002] In recent years, edge AI inference has become an important development direction in the AI ​​field due to its advantages in real-time performance, privacy protection, and energy efficiency. Transformer architecture models, represented by large language models (such as Llama2-7B), primarily consist of attention mechanisms and multilayer perceptrons (MLPs) as their core computational units. However, as the length of sequences supported by these models continues to grow, edge inference systems face increasingly severe storage performance bottlenecks.

[0003] During inference, the storage system faces three key challenges due to the small computational cost of a single token and the difficulty in optimizing it through batch processing: First, long-sequence processing requires extremely high storage bandwidth to meet the frequent access needs of parameters and intermediate results; second, the lightweight nature of single-token computation makes storage access latency the main constraint on performance; and finally, the data synchronization and consistency maintenance operations introduced by multi-level storage architectures (such as L1 / L2 / LLC caches) significantly increase system complexity and energy consumption.

[0004] Specifically, taking the Llama2-7B model as an example, performance analysis of its core operators under typical hardware configurations shows that data transfer time far exceeds actual computation time, becoming the main source of inference latency. When memory access latency reaches 30 clock cycles, the system has to mask the latency by increasing parallel transmission requests. This not only leads to a significant increase in register resources and control logic area and power consumption, but also fails to completely eliminate head blocking and tail blocking effects caused by latency. Furthermore, the address page table maintenance, cache coherence protocols, and out-of-order execution mechanisms required by multi-level memory architectures further exacerbate the degradation of system performance and area.

[0005] Currently, the industry mainly uses the following technologies to optimize storage performance, but there are still significant shortcomings:

[0006] (1) 3D stacking technology: This is an advanced integrated circuit packaging technology that significantly shortens the interconnect distance, improves integration and reduces power consumption by vertically stacking chips. However, this technology is currently mainly used inside memory devices (such as DRAM inter-layer stacking or SRAM and logic layer stacking), and has failed to effectively solve the high latency problem between memory and computing units.

[0007] (2) High Bandwidth Memory (HBM): This is an advanced high bandwidth memory technology that combines 3D stacking with 2.5D packaging to provide up to 1TB / s of storage bandwidth, with significantly lower power consumption than traditional DRAM. However, in current mainstream HBM solutions, the memory chip and logic chip are connected through a 2.5D interposer, which still results in a significant latency, typically reaching 500-700 nanoseconds, making it difficult to meet the real-time requirements of edge AI inference.

[0008] (3) Processing Near Memory (PNM): This method embeds computing units near storage devices to reduce latency and power consumption by reducing data movement. However, due to the high latency of existing HBM architectures, it is impossible to truly realize the design concept of "computing close to data", and the additional overhead introduced by multi-level storage management partially offsets its performance advantages.

[0009] In summary, existing solutions cannot simultaneously meet the demands of edge AI inference for low latency, high bandwidth, and lightweight storage operations. Especially in long-sequence, low-batch scenarios, storage bottlenecks severely restrict energy efficiency and real-time performance. Therefore, an innovative architecture is urgently needed to optimize data interaction between storage and computing units, reduce redundant data transmission, and lower latency and power consumption, thereby driving the further development of edge AI inference technology. Summary of the Invention

[0010] In view of the shortcomings of the prior art described above, the purpose of this application is to provide an in-memory computing device, an edge AI inference method, a medium, and a terminal to solve the technical problem that the high latency and low energy efficiency of data interaction between storage and computing units in the prior art limit the performance of edge AI inference.

[0011] To achieve the above and other related objectives, a first aspect of this application provides an in-memory computing device, comprising: a storage layer for storing multiple microinstruction codes for AI model inference and operation data required by each microinstruction code; an AI computing layer for sequentially reading each microinstruction code for AI model inference from the storage layer, parsing each read microinstruction code, and performing a transfer operation on the operation data required by each microinstruction code based on the parsing result, and performing a shaping operation on the operation data required by each microinstruction code during the transfer operation; and performing a distributed AI inference operation based on the operation data required by each microinstruction code after the shaping operation to obtain a distributed AI inference result.

[0012] In some embodiments of the first aspect of this application, the storage layer is configured with a plurality of storage groups, each storage group including a plurality of storage entities; the storage group is used to store microinstruction code for AI model inference and operation data required by the microinstruction code.

[0013] In some embodiments of the first aspect of this application, the AI ​​computing layer is configured with multiple AI computing modules, and one AI computing module is connected to one storage group; each AI computing module includes a CPU cluster, a bus channel, and an AI computing unit to realize the connection between the AI ​​computing layer and the storage layer; the CPU cluster is configured with a first register group; the bus channel is configured with a direct memory access controller; and the AI ​​computing unit is configured with a second register group; wherein, the CPU cluster is used to perform the following steps: controlling the direct memory access controller on the bus channel to read the current microinstruction code for AI model inference from the storage group of the storage layer, and after moving the read current microinstruction code to the first register group, controlling the direct memory access controller on the bus channel to read the next microinstruction code for AI model inference from the storage group of the storage layer; parsing the current microinstruction code stored in the first register group, and according to the parsing result, performing a transfer operation on the operation data required by the current microinstruction code to move the operation data required by the current microinstruction code to the second register group, and during the transfer operation, controlling the direct memory access controller on the bus channel to perform a shaping operation on the operation data required by the current microinstruction code.

[0014] In some embodiments of the first aspect of this application, the CPU cluster performs a transfer operation on the operation data required by the current microinstruction code by controlling the direct memory access controller on the bus channel. The method includes: controlling the direct memory access controller to transfer the operation data required by the current microinstruction code between the storage group connected to the AI ​​computing module and the second register group; or, controlling the direct memory access controller to transfer the operation data required by the current microinstruction code between the second register groups of different AI computing modules.

[0015] In some embodiments of the first aspect of this application, the AI ​​computing module further includes a memory controller, and the CPU cluster controls the memory controller to perform operations on the operation data required by the current microinstruction code, in the following ways: controlling the memory controller to perform operations on the operation data required by the current microinstruction code between different storage groups; or controlling the memory controller to perform operations on the operation data required by the current microinstruction code between different storage banks in the same storage group.

[0016] In some embodiments of the first aspect of this application, the AI ​​computing layer is configured with a multi-core scheduler, and the AI ​​computing layer synchronizes the state information of each AI computing module through the multi-core scheduler in order to adjust the allocation of computing tasks for AI model inference.

[0017] In some embodiments of the first aspect of this application, the CPU cluster is further configured to control the memory controller to move multiple microinstruction codes for AI model inference and the operation data required by each microinstruction code from external memory to the corresponding memory group.

[0018] To achieve the above and other related objectives, a second aspect of this application provides an edge AI inference method applied to the in-memory computing device described above, the in-memory computing device including a storage layer and an AI computing layer; wherein, the edge AI inference method includes: the AI ​​computing layer sequentially reading each micro-instruction code of the AI ​​model inference from the storage layer, and parsing each read micro-instruction code, so as to perform a transfer operation on the operation data required by each micro-instruction code according to the parsing result, and performing a shaping operation on the operation data required by each micro-instruction code during the transfer operation; the AI ​​computing layer performs a distributed AI inference operation according to the operation data required by each micro-instruction code after the shaping operation, so as to obtain a distributed AI inference result.

[0019] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the edge AI inference method as described above.

[0020] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the edge AI inference method as described above.

[0021] As described above, the in-memory computing device, edge AI inference method, medium, and terminal of this application have the following beneficial effects:

[0022] (1) A collaborative architecture scheme between a 3D storage system and an AI system is proposed. Multiple AI computing modules are configured on the AI ​​computing layer. Each storage pin on each storage body of a storage group is connected to the memory controller of the AI ​​computing module. This realizes that one AI computing module is connected to one storage group, optimizes storage latency, improves storage bandwidth, and eliminates serial-to-parallel conversion overhead.

[0023] (2) The operation data required by the micro-instruction code of AI model inference is transferred through the second register group. Under the performance guarantee of high bandwidth storage, cache management overhead is avoided, the power consumption and area of ​​prefetch logic are saved, the complexity of the storage system itself and data transportation is reduced, storage transportation is made lightweight, and latency is further reduced.

[0024] (3) In terms of data transfer and data synchronization, during the execution of distributed AI inference operations, data transfer can be carried out synchronously between different storage groups, between storage units in the same storage group, between storage groups and second register groups, and between second register groups of different computing modules. This further increases the parallelism of data synchronization, reduces storage access latency and idle waiting of AI computing units, and makes the data supply rate match the computing rate, thereby improving the overall throughput and energy efficiency of the system. Attached Figure Description

[0025] Figure 1 The diagram shown is a schematic representation of an in-memory computing device according to one embodiment of this application.

[0026] Figure 2 The diagram shown is a top-down view of the storage layer in one embodiment of this application.

[0027] Figure 3 The diagram shown is a top-down view of the AI ​​computing layer in one embodiment of this application.

[0028] Figure 4 The diagram shown is a flowchart of an edge AI inference method in one embodiment of this application.

[0029] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0032] <1> Llama2-7B is a high-performance, open-source, large-scale language model suitable for a variety of natural language processing tasks.

[0033] <2> Token: The basic unit after the model segments the input text.

[0034] <3> Transformer is a deep learning model architecture used for multimodal tasks such as computer vision and speech recognition. It relies on a self-attention mechanism to solve the problem of long-distance dependencies and achieve efficient parallel computing.

[0035] <4> DRAM (Dynamic Random Access Memory): A type of computer memory used to store data for fast access by the central processing unit.

[0036] <5> SRAM: Static Random Access Memory, is a very important type of semiconductor memory. As long as it is powered on, the data stored in it will not be lost.

[0037] <6> 3D stacking: is an advanced chip packaging technology that improves integration and performance by vertically stacking multiple layers of chips or wafers (dies).

[0038] <7> 2.5D packaging is a technology that integrates multiple chips (such as processors, memory, sensors, etc.) into a single package. Unlike 3D packaging, the chips in a 2.5D package are not stacked vertically, but rather placed side by side on an interposer layer.

[0039] <8> DMA (Direct Memory Access): Direct memory access is a mechanism that allows external devices to bypass the CPU and directly exchange data with system memory (such as DRAM).

[0040] <9> Batch: Batch processing refers to processing multiple input samples simultaneously (such as processing 16 sentences at the same time).

[0041] <10> NoC: Network on Chip is a communication subsystem architecture used for communication between multiple processing elements in an integrated circuit.

[0042] <11> SDIO interface: Secure Digital Input / Output interface, a secure data transmission interface based on the SD memory card protocol extension, supporting external storage devices.

[0043] <12> ETH interface: Ethernet interface, which is a wired network communication interface.

[0044] <13> USB 3.1 interface: Universal Serial Bus 3.1, is an ultra-high-speed universal serial transmission interface.

[0045] <14> PCIe Gen3 / 4 interface: High-speed peripheral interconnect interface, point-to-point high-speed serial expansion bus, used to connect acceleration devices.

[0046] <15> LPDDR5 / LPDDR5X Controller Interface: Low-power memory controller interface.

[0047] <16> TSV (Through-Silicon Via): A silicon via is an advanced semiconductor packaging technology that enables vertical interconnection within or between chips by creating vertical through-holes on silicon chips.

[0048] <17> Hybrid bonding is an advanced packaging technology that combines multiple bonding methods to achieve efficient connections between different materials or chip structures.

[0049] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1-3 Detailed explanation. Figure 1 This illustration shows a schematic diagram of an in-memory computing device according to an embodiment of the present invention. Vertically stacked DRAM1 and DRAM2 chips form a memory layer. A Logic Die represents the AI ​​computing layer, and an RDL represents a redirected trace, providing an interconnect path between the AI ​​computing layer and the memory layer. A Substrate represents the substrate. The in-memory computing device is stacked on the upper surface of the substrate using stacked packaging technology. The in-memory computing device includes:

[0050] The storage layer is used to store multiple micro-instruction codes for AI model inference, as well as the operation data required by each micro-instruction code;

[0051] The AI ​​computing layer is used to sequentially read each micro-instruction code of the AI ​​model inference from the storage layer, and parse each read micro-instruction code. Based on the parsing results, it performs data transfer operations on the operation data required by each micro-instruction code. During the data transfer operation, it performs shaping operations on the operation data required by each micro-instruction code. Based on the operation data required by each micro-instruction code after the shaping operation, it executes distributed AI inference operations to obtain distributed AI inference results.

[0052] In this embodiment, Figure 2This diagram illustrates a top-down view of the storage layer in an embodiment of the present invention. Multiple storage groups are configured on the storage layer, each storage group comprising multiple memory banks. The storage groups are used to store microinstruction code for AI model inference and the operational data required by the microinstruction code. Multiple DRAM chips are vertically stacked on the storage layer using vertical interconnect technology. Each DRAM chip includes multiple memory banks, and the memory banks of each DRAM chip are grouped to form multiple storage groups.

[0053] In this embodiment, the vertical interconnect technology includes TSV technology or Hybrid bonding technology. Hybrid bonding supports connections between multi-level DRAM layers and AI computing layers. The TSV solution ensures flexible configuration of the top and bottom layers between different layers, while the parallel storage interface guarantees high storage bandwidth and eliminates serial-to-parallel conversion overhead.

[0054] In this embodiment, based on the parameters of each configured AI model, combined with a preset intermediate cache and an activation parameter having the largest single operator size, the peak capacity capable of accommodating all AI models is calculated. The number of DRAM chips is then configured according to this peak capacity. For example, if the single-layer density of each DRAM chip is 2.5GB, and the calculated peak capacity is 5GB, then two DRAM chips, such as DRAM1 and DRAM2, are vertically stacked on the storage layer, resulting in a storage layer with a memory capacity of 5GB, sufficient to accommodate all AI models. Thus, by utilizing this customized method to evaluate the storage bandwidth and storage metrics required for different model algorithms, cost and power consumption are saved while ensuring performance.

[0055] In this embodiment, each memory bank is provided with multiple memory pins, preferably 256 memory pins, which can achieve a total memory bandwidth of 1.28TB / s for the memory layer. Under this configuration, the operating speed of each memory pin can reach up to 500Mbps.

[0056] In this embodiment, Figure 3 This diagram illustrates a top-down view of the AI ​​computing layer in an embodiment of the present invention. The AI ​​computing layer is configured with multiple AI computing modules, each connected to a storage group. Each AI computing module includes a CPU cluster, a bus channel, and an AI computing unit to connect the AI ​​computing layer to the storage layer. The CPU cluster is configured with a first register group. The bus channel is configured with a direct memory access controller (DMA). The AI ​​computing unit is configured with a second register group. The CPU cluster is used to perform the following steps:

[0057] (1) Control the direct memory access controller (DMA) on the bus channel to read the current microinstruction code for AI model inference from the storage group of the storage layer, and after moving the read current microinstruction code to the first register group, control the direct memory access controller (DMA) on the bus channel to read the next microinstruction code for AI model inference from the storage group of the storage layer;

[0058] (2) The current microinstruction code stored in the first register group is parsed, and the operation data required by the current microinstruction code is moved according to the parsing result, so as to move the operation data required by the current microinstruction code to the second register group. During the moving operation, the direct memory access controller (DMA) on the bus channel is controlled to perform shaping operation on the operation data required by the current microinstruction code.

[0059] In this embodiment, the AI ​​computing module performs distributed AI inference operations based on the operation data required by each microinstruction code after the shaping operation, in order to obtain distributed AI inference results. During the execution of the distributed AI inference operation, the operation data is simultaneously transferred and shaped.

[0060] In this embodiment, the AI ​​computing module further includes a memory controller (MC). The number of AI computing modules is the same as the number of storage groups. One AI computing module is connected to one storage group. Each storage pin on each memory bank of a storage group is connected to the memory controller (MC) of the AI ​​computing module, so that one AI computing module is connected to one storage group.

[0061] In this embodiment, for example, there are 80 memory banks, divided into 4 memory groups: Memory Group 0, Memory Group 1, Memory Group 2, and Memory Group 3. The AI ​​computing layer is configured with 4 AI computing modules (Cores): AI Computing Module 0 (Core0), AI Computing Module 1 (Core1), AI Computing Module 2 (Core2), and AI Computing Module 3 (Core3). AI Computing Module 0 (Core0) is connected to Memory Group 0, AI Computing Module 1 (Core1) is connected to Memory Group 1 (Core1), AI Computing Module 2 (Core2) is connected to Memory Group 2 (Core2), and AI Computing Module 3 (Core3) is connected to Memory Group 3 (Core3).

[0062] In this embodiment, the AI ​​computing unit is equipped with a second register group for relaying the operation data required by the microinstruction code for AI model inference. It avoids the multi-level storage units and corresponding prefetch and data retention operation units found in traditional storage structures, thus avoiding cache management overhead, saving power and area of ​​the prefetch logic, and preventing bandwidth waste caused by erroneous prefetching. The second register group is an L0 register group, a zero-level static register storage structure.

[0063] In this embodiment, the first register group is also an L0 register group. The CPU cluster is also configured with multiple control core units. These control core units control the Direct Memory Access Controller (DMA) to move the microinstruction code for AI model inference to the first register group. The control core units also parse the microinstruction code stored in the first register group and, based on the parsing results, perform data transfer operations on the operation data required by the microinstruction code to move it to the second register group. Furthermore, they control the DMA to perform shaping operations on the operation data required by the microinstruction code.

[0064] In this embodiment, the CPU cluster controls the Direct Memory Access Controller (DMA) on the bus channel to perform operations on the operation data required by the current microinstruction code. The methods include: controlling the Direct Memory Access Controller (DMA) to perform operations on the operation data required by the current microinstruction code between the storage group connected to the AI ​​computing module and the second register group; or, controlling the Direct Memory Access Controller (DMA) to perform operations on the operation data required by the current microinstruction code between the second register groups of different AI computing modules.

[0065] In this embodiment, the AI ​​computing unit is also configured with a shared memory group. The direct memory access controller (DMA) first moves the operation data required by the current microinstruction code to the shared memory group, and then moves it from the shared memory group to the second register group.

[0066] In this embodiment, the data transfer operations supported by the Direct Memory Access Controller (DMA) include the following: (1) data transfer between the storage group connected to the AI ​​computing module and the second register group of the AI ​​computing module; (2) data transfer between the second register groups of different computing modules. For example, for AI computing module 0 (Core 0), if the operation data required by the current microinstruction code is in storage group 0 (Bank Group 0), the Direct Memory Access Controller (DMA) will first transfer the operation data required by the current microinstruction code from storage group 0 (Bank Group 0) to the shared memory group of AI computing module 0 (Core 0), and then from the shared memory group of AI computing module 0 (Core 0) to the second register group. Alternatively, the data in the second register group of AI computing module 0 (Core 0) can be transferred to storage group 0 (Bank Group 0) through the shared memory group. For AI computing module 0 (Core0), if the operation data required by the current microinstruction code is in the second register group of AI computing module 1 (Core1), the direct memory access controller (DMA) will move the operation data required by the current microinstruction code from the second register group of AI computing module 1 (Core1) to the second register group of AI computing module 0 (Core0).

[0067] In this embodiment, the data transfer operations not supported by the Direct Memory Access Controller (DMA) include the following: (1) data transfer between different memory groups; (2) data transfer between different memory banks within the same memory group.

[0068] In this embodiment, the direct memory access controller (DMA) performs shaping operations on the operation data required by the microinstruction code. The shaping operations include, but are not limited to, dimension transformation operations (Reshape), padding operations, matrix transpose operations, etc.

[0069] In this embodiment, the dimension transformation operation changes the dimensional view of the data without altering the original values, such as converting a 4×4 matrix into a 16×1 vector to adapt to the input and output formats of different layers. The padding operation fills in fixed values ​​(usually 0) around specific dimensions (usually edges) of the data. For example, in convolutional neural networks, to maintain the spatial size of the output feature map consistent with the input, zeros are padded around the edges of the input image. The matrix transpose operation swaps two dimensions of the data; for a matrix, this means interchangeing rows and columns. For example, after transposing a 2x3 matrix, the order of the data in memory is rearranged, resulting in a new 3x2 matrix.

[0070] In this embodiment, the bus channel is further configured with a router, an xbar bus, and a first memory management unit. The router is used to configure data paths between different storage groups, data paths between different memory banks within the same storage group, and data paths between the storage group and the second register group. The xbar bus is used to configure data paths between memory controllers (MCs) of different AI computing modules, and data paths between the memory controller (MC) of the same AI computing module and other components. The first memory management unit is used to configure data transfer paths, such as where data is transferred from and to.

[0071] In this embodiment, the AI ​​computing module further includes a memory controller (MC). The CPU cluster controls the memory controller (MC) to perform data transfer operations required by the current microinstruction code. This can be achieved by: controlling the memory controller (MC) to transfer the data required by the current microinstruction code between different storage groups; or, controlling the memory controller (MC) to transfer the data required by the current microinstruction code between different storage banks within the same storage group.

[0072] In this embodiment, the data transfer operations supported by the memory controller (MC) include the following: (1) data transfer between different storage groups; (2) data transfer between different storage banks within the same storage group. For example, for AI computing module 0 (Core0), if the operation data required by the current microinstruction code is in storage group 1 (Bank Group1), the memory controller (MC) will transfer the operation data required by the current microinstruction code from storage group 1 (Bank Group1) to storage group 0 (Bank Group0), and then transfer the operation data required by the current microinstruction code from storage group 0 (Bank Group0) to the shared memory bank of AI computing module 0 (Core0) through the direct memory access controller (DMA), and then transfer it from the shared memory bank of AI computing module 0 (Core0) to the second register bank. For AI computing module 0 (Core0), if the operation data required by the current microinstruction code is in different memory banks of memory group 0, the control memory controller (MC) will move the operation data required by the current microinstruction code from the different memory banks to the target memory bank of memory group 0. Then, the direct memory access controller (DMA) will move the operation data required by the current microinstruction code from the target memory bank of memory group 0 to the shared memory bank of AI computing module 0 (Core0), and then from the shared memory bank of AI computing module 0 (Core0) to the second register bank.

[0073] In this embodiment, the data transfer operations supported by the memory controller (MC) also include the following: (1) Two-dimensional tensor block transfer (2D Tensor), batch transfer of matrix blocks, supporting row / column step size configuration, such as for data with obvious two-dimensional spatial structure such as matrices, image blocks, convolution windows; (2) One-dimensional data transfer, such as linear continuous data such as vectors, gradients, parameter update amounts, etc.

[0074] In this embodiment, the data transfer operations not supported by the memory controller (MC) include the following: (1) data transfer between the storage group connected to the AI ​​computing module and the second register group of the AI ​​computing module; (2) data transfer between the second register groups of different computing modules.

[0075] In this embodiment, the AI ​​computing unit is further configured with a tensor computing unit, a vector computing unit, a special function computing unit, a second memory management unit, and a barrier unit. The tensor computing unit performs large-scale, regularized parallel matrix multiplication and convolution operations. The vector computing unit performs the same operation on all elements of a one-dimensional array (vector). The special function computing unit calculates transcendental functions, nonlinear functions, or loss functions. The second memory management unit manages access to and protects the second register group of the AI ​​computing unit. The barrier unit synchronizes the execution progress of multiple parallel threads or multiple AI computing modules within the same AI computing unit, ensuring that all parallel tasks reach a predetermined synchronization point in the program before allowing all tasks to continue executing subsequent instructions, thereby preventing data races.

[0076] In this embodiment, a multi-core scheduler is configured on the AI ​​computing layer. The AI ​​computing layer uses the multi-core scheduler to synchronize the status information of each AI computing module, allowing for adjustments to the allocation of computational tasks for AI model inference. The multi-core scheduler continuously monitors the real-time status of each AI computing module, including computational load, storage access pressure, and task completion progress. When an AI computing module experiences delays due to processing complex operators (such as attention mechanisms), the multi-core scheduler dynamically migrates some of the computational tasks from that AI computing module to currently idle or less loaded AI computing modules. Simultaneously, the multi-core scheduler synchronizes the intermediate result storage address information of all AI computing modules, ensuring that when one AI computing module needs data generated by another, it can access it quickly and accurately through the Direct Memory Access Controller (DMA), avoiding resource idleness and waiting, thereby improving the overall inference throughput.

[0077] In this embodiment, the CPU cluster is also used to control the memory controller (MC) to move multiple microinstruction codes for AI model inference and the operation data required by each microinstruction code from external memory to the corresponding memory group.

[0078] In this embodiment, the AI ​​computing module further includes a single-core scheduler, which comprises a resource dedependency unit and a data dedependency unit. The resource dedependency unit checks and manages the availability of hardware resources required for microinstruction code execution to avoid structural hazards and ensure that multiple instructions do not simultaneously contend for the same physical hardware resource. The data dedependency unit checks whether the operands required by the instruction are ready to avoid data hazards.

[0079] In this embodiment, the AI ​​computing layer is configured with multiple peripheral interfaces, including but not limited to: SDIO interface, ETH interface, USB 3.1 interface, PCIe Gen3 / 4 interface, and LPDDR5 / LPDDR5X Controller interface. These peripheral interfaces are used to connect to an external compiler, which compiles the configured AI model inference program into multiple microinstruction codes for AI model inference and the operation data required for each microinstruction code; it also stores each microinstruction code in the instruction space of external memory and stores the operation data required to execute each microinstruction code in the data space of external memory. During AI model inference, the CPU cluster controls the memory controller (MC) to move the multiple microinstruction codes for AI model inference and the operation data required for each microinstruction code from external memory to the corresponding memory group.

[0080] It is worth noting that the in-memory computing device of this application has the following advantages:

[0081] (1) A collaborative architecture scheme between a 3D storage system and an AI system is proposed. Multiple AI computing modules are configured on the AI ​​computing layer. Each storage pin on each storage body of a storage group is connected to the memory controller of the AI ​​computing module. This realizes that one AI computing module is connected to one storage group, optimizes storage latency, improves storage bandwidth, and eliminates serial-to-parallel conversion overhead.

[0082] (2) The operation data required by the micro-instruction code of AI model inference is transferred through the second register group. Under the performance guarantee of high bandwidth storage, cache management overhead is avoided, the power consumption and area of ​​prefetch logic are saved, the complexity of the storage system itself and data transportation is reduced, storage transportation is made lightweight, and latency is further reduced.

[0083] (3) In terms of data transfer and data synchronization, during the execution of distributed AI inference operations, data transfer can be carried out synchronously between different storage groups, between storage units in the same storage group, between storage groups and second register groups, and between second register groups of different computing modules. This further increases the parallelism of data synchronization, reduces storage access latency and idle waiting of AI computing units, and makes the data supply rate match the computing rate, thereby improving the overall throughput and energy efficiency of the system.

[0084] The in-memory computing device of this application is not only suitable for edge-side inference acceleration, but can also be extended to training scenarios and other data-intensive tasks.

[0085] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first memory management unit" and "second memory management unit" are used only to distinguish different memory management units and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0086] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0087] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0088] Figure 4 This is a flowchart illustrating the edge AI inference method provided in an embodiment of this application. Figure 4 As shown, this edge AI inference method is applied to the in-memory computing device described above, which includes a storage layer and an AI computing layer; wherein, the edge AI inference method includes:

[0089] S401: The AI ​​computing layer sequentially reads each micro-instruction code of the AI ​​model inference from the storage layer, and parses each read micro-instruction code. Based on the parsing results, it performs transfer operations on the operation data required by each micro-instruction code, and performs shaping operations on the operation data required by each micro-instruction code during the transfer operation.

[0090] S402: The AI ​​computing layer performs distributed AI inference operations based on the operation data required by each micro-instruction code after the shaping operation, so as to obtain distributed AI inference results.

[0091] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0092] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0093] Figure 5 This is a schematic block diagram of an electronic terminal provided in an embodiment of this application. The electronic terminal includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the edge AI inference method described above. Figure 5 As shown, the electronic terminal 500 includes at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0094] The user interface 505 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0095] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0096] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the electronic terminal 500. Examples of this data include: any executable program for operation on the electronic terminal 500, such as the operating system 5021 and application programs 5022; the operating system 5021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 5022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The implementation of the edge AI inference method provided in this embodiment of the invention can be included in the application program 5022.

[0097] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0098] In an exemplary embodiment, the electronic terminal 500 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0099] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the edge AI inference method as described above.

[0100] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0101] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0102] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0106] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0107] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] In summary, this application provides an in-memory computing device, an edge AI inference method, a medium, and a terminal. Through direct interconnection between the storage group and the AI ​​computing module, register relay, and a parallel data transport mechanism, it improves system performance and energy efficiency, optimizes storage latency and bandwidth, eliminates serial-to-parallel conversion overhead, avoids cache management overhead, saves power consumption and area of ​​prefetch logic, and reduces the complexity of the storage system itself and data transport, thus lightweighting storage transport. During the execution of distributed AI inference operations, data transport and shaping can be performed simultaneously, improving data synchronization parallelism, matching the data supply rate with the computing rate, reducing idle waiting time of AI computing units, and achieving improvements in system throughput and energy efficiency. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0110] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An in-memory computing device, comprising: The application relates to a distributed AI model inference system, which comprises the following parts: a storage layer for storing a plurality of microinstruction codes of AI model inference and operation data required by each microinstruction code; a plurality of storage groups are arranged on the storage layer, each of the storage groups comprises a plurality of storage banks; the storage groups are used for storing microinstruction codes of AI model inference and operation data required by the microinstruction codes; an AI computing layer for sequentially reading each microinstruction code of AI model inference from the storage layer, analyzing each microinstruction code read, carrying out operation data required by each microinstruction code according to the analysis result, and performing shaping operation on the operation data required by each microinstruction code in the process of the carrying operation; performing distributed AI inference operation according to the operation data required by each microinstruction code after the shaping operation to obtain a distributed AI inference result; a plurality of AI computing modules are arranged on the AI computing layer, one AI computing module is connected with one storage group; each AI computing module comprises a CPU cluster, a bus channel and an AI computing unit to realize the connection between the AI computing layer and the storage layer; a first register group is arranged in the CPU cluster; a direct memory access controller is arranged on the bus channel; a second register group is arranged in the AI computing unit; wherein the CPU cluster is used for performing the following steps: controlling the direct memory access controller on the bus channel to read the current microinstruction code of AI model inference from the storage group of the storage layer, and then controlling the direct memory access controller on the bus channel to read the next microinstruction code of AI model inference from the storage group of the storage layer after carrying the current microinstruction code read to the first register group; analyzing the current microinstruction code stored in the first register group, and carrying out operation data required by the current microinstruction code according to the analysis result to carry the operation data required by the current microinstruction code to the second register group, and controlling the direct memory access controller on the bus channel to perform shaping operation on the operation data required by the current microinstruction code in the process of the carrying operation. 2.The all-memory computing device of claim 1, wherein, The CPU cluster carries out the carrying operation on the operation data required by the current microinstruction code by controlling the direct memory access controller on the bus channel, which includes: controlling the direct memory access controller to carry out the carrying operation on the operation data required by the current microinstruction code between the storage group connected with the AI computing module and the second register group; or controlling the direct memory access controller to carry out the carrying operation on the operation data required by the current microinstruction code between the second register groups of different AI computing modules. 3.The all-memory computing device of claim 1, wherein, The AI computing module further comprises a memory controller, and the CPU cluster controls the memory controller to perform a carrying operation on operation data required by the current microinstruction code in the following manner: controlling the memory controller to perform a carrying operation on the operation data required by the current microinstruction code between different storage groups; or, controlling the memory controller to perform a carrying operation on the operation data required by the current microinstruction code between different storage banks of the same storage group. 4.The all-memory computing device of claim 1, wherein, The AI computing layer is configured with a multi-core scheduler, and the AI computing layer synchronizes state information of each AI computing module through the multi-core scheduler to adjust the allocation of a computing task of AI model reasoning. 5.The all-memory computing device of claim 3, wherein, The CPU cluster is further configured to control the memory controller to carry, from an external memory, a plurality of microinstruction codes of AI model reasoning and operation data required by each microinstruction code to a corresponding storage group.

6. An end-side AI inference method, characterized by, The application is applied to the storage-computing integrated device as claimed in any one of claims 1-5, and the storage-computing integrated device comprises a storage layer and an AI computing layer; wherein the end-side AI reasoning method comprises: The AI computing layer sequentially reads each microinstruction code of AI model reasoning from the storage layer, and analyzes each read microinstruction code to perform a carrying operation on operation data required by each microinstruction code according to an analysis result, and performs a shaping operation on the operation data required by each microinstruction code during the carrying operation; The AI computing layer performs a distributed AI reasoning operation according to the operation data required by each microinstruction code after the shaping operation to obtain a distributed AI reasoning result; The storage layer is configured with a plurality of storage groups, and each storage group comprises a plurality of storage banks; the storage groups are used to store microinstruction codes of AI model reasoning and operation data required by the microinstruction codes; The AI computing layer is configured with a plurality of AI computing modules, and one AI computing module is connected to one storage group; each AI computing module comprises a CPU cluster, a bus channel, and an AI computing unit to realize the connection between the AI computing layer and the storage layer; the CPU cluster is configured with a first register group; the bus channel is configured with a direct memory access controller; the AI computing unit is configured with a second register group; wherein the CPU cluster is used to perform the following steps: controlling the direct memory access controller on the bus channel to read a current microinstruction code of AI model reasoning from a storage group of the storage layer, and carrying the read current microinstruction code to the first register group, and then controlling the direct memory access controller on the bus channel to read a next microinstruction code of AI model reasoning from the storage group of the storage layer; The current microinstruction code stored in the first register group is parsed, and according to the parsing result, the operation data required by the current microinstruction code is carried out to carry the operation data required by the current microinstruction code to the second register group, and in the process of carrying operation, the direct memory access controller on the bus channel is controlled to perform the shaping operation on the operation data required by the current microinstruction code.

7. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the end-side AI inference method of claim 6.

8. An electronic terminal comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the end-side AI inference method of claim 6. The computer program is executed by the processor to implement the end-side AI inference method of claim 6.

Images