In-memory computing neural network accelerator supplemented by digital multiplication and accumulation core and acceleration method

Abstract

The present invention provides an in-memory computing neural network accelerator supplemented by a digital multiplication-accumulation core and an acceleration method. The accelerator includes: a top-level controller for dynamically reconfiguring the data path between the in-memory computing core module, an on-chip storage module, and the digital multiplication-accumulation core module according to operating instructions; the on-chip storage module for storing data of the current neural network model during operation; and the in-memory computing core module and the digital multiplication-accumulation core module for obtaining data from the on-chip storage module and performing corresponding multiplication-accumulation operations based on the dynamically reconfigured data path according to operating instructions. By introducing a lightweight digital multiplication-accumulation core module into the original in-memory computing neural network accelerator to assist the in-memory computing core module in processing neural network layers that it is not proficient in, the accelerator can run multiple neural network models with high computing resource utilization, thereby improving the computing resource utilization of the accelerator and thus enhancing the overall performance of the accelerator.

Description

Technical Field

[0001] This invention relates to the field of circuit design technology, and in particular to an in-memory computing neural network accelerator and acceleration method supplemented by a digital multiply-accumulate core. Background Technology

[0002] Computing-In-Memory (CIM) is an emerging circuit architecture designed to address the "memory wall" problem caused by the separation of memory and processor in traditional von Neumann architectures. In traditional architectures, data needs to be constantly moved from memory to the processor for computation and then written back to memory, a process that consumes significant time and energy. CIM, on the other hand, performs computations directly within the storage unit, reducing the need for data movement and improving processing speed and energy efficiency.

[0003] For applications such as neural networks that require a large number of parallel multiplication and accumulation operations, in-memory computing offers significant advantages. This is because: firstly, in-memory computing is typically designed to perform multiple multiplication and accumulation operations simultaneously, exhibiting high parallelism, which is well-suited for matrix operations in deep learning; secondly, by reducing the transfer of data between memory and the processor, in-memory computing can significantly reduce power consumption.

[0004] However, in-memory computation encounters some challenges when applied to certain types of neural network layers. Specifically, in-memory computation has high parallelism in its multiply-accumulate operations, and the parallelism of the in-memory computation core is typically a power of 2 (e.g., 32, 64, 128, etc.). Therefore, there is a problem of low utilization of computing resources when running some neural network models.

[0005] For example, in lightweight convolutional neural networks like MobileNet, depthwise separable convolution is a key component. Depthwise separable convolution involves two steps: first, a spatial convolution (typically a 3x3 convolution) is applied to each input channel using a single kernel; then, a pointwise convolution is applied to all output channels. The 3x3 convolution here involves a multiply-accumulate parallelism of 9 (i.e., 3x3). If such an operation were mapped to a single in-memory computational core with a multiply-accumulate parallelism of 32 or higher, only a portion of the computational resources would be used, leaving the rest idle, resulting in low overall computational resource utilization (typically below 30%).

[0006] The same problem exists in standard 3x3 convolutional layers, which are fundamental building blocks in many neural network models. Because the parallelism of a 3x3 convolution is a multiple of 9, while the parallelism of an in-memory computational core is usually a power of 2, the two are not divisible, thus leading to a waste of resources in the in-memory computational core.

[0007] Therefore, how to solve the problem of low computing resource utilization caused by the mismatch between the parallelism of existing in-memory neural network accelerators and the multiply-accumulate parallelism of the neural network models they run is an important issue that urgently needs to be addressed in the field of circuit design. Summary of the Invention

[0008] This invention provides an in-memory computing neural network accelerator and acceleration method supplemented by a digital multiply-accumulate core, which overcomes the defect of low computing resource utilization caused by the mismatch between the parallelism of existing in-memory computing neural network accelerators and the multiply-accumulate parallelism of the running neural network model, improves the computing resource utilization of in-memory computing neural network accelerators, and thus improves the overall performance of the accelerator.

[0009] On one hand, this invention provides an in-memory computing neural network accelerator supplemented by a digital multiply-accumulate core, comprising: a top-level controller, an in-memory computing core module, an on-chip storage module, and a digital multiply-accumulate core module; wherein, the top-level controller is used to receive operation instructions from off-chip users and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the operation instructions; wherein, the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; the on-chip storage module is used to store data of the current neural network model during operation, the data including at least model weight parameters, input feature maps, and output feature maps; the in-memory computing core module is used to obtain first data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and execute the corresponding multiply-accumulate operation; the digital multiply-accumulate core module is used to obtain second data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and execute the corresponding multiply-accumulate operation; wherein, both the first data and the second data are included in the data.

[0010] Furthermore, the current neural network model is MobileNet, which includes depthwise separable convolutional layers and pointwise convolutional layers. In the computational allocation scheme corresponding to MobileNet, the digital multiply-accumulate core module runs the depthwise separable convolutional layers, and the in-memory computational core module runs the pointwise convolutional layers. Correspondingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computational core module, the on-chip storage module, and the digital multiply-accumulate core module according to the network structure of MobileNet and its corresponding computational allocation scheme. In the first data path, the digital multiply-accumulate core module reads the input feature map of the depthwise separable convolutional layer from the on-chip storage module, and after completing the depthwise separable convolutional layer operation, sends its output feature map to the in-memory computational core module. After completing the pointwise convolutional layer operation, the in-memory computational core module writes its output feature map back to the on-chip storage module. In this process, the digital multiply-accumulate core module and the in-memory computational core module operate in a pipelined manner.

[0011] Furthermore, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiply-accumulate operations, and the digital multiply-accumulate core module performs 1 / 9 of the multiply-accumulate operations; correspondingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme; in the second data path, the in-memory computing core module receives data from the on-chip storage module... The system reads 8 / 9 of the input feature map from the standard 3×3 convolutional layer from the on-chip storage module, and the digital multiply-accumulate core module reads 1 / 9 of the input feature map from the on-chip storage module. The in-memory computing core module and the digital multiply-accumulate core module perform their respective multiply-accumulate operations, and the output feature map obtained by summing the operation results of the in-memory computing core module and the digital multiply-accumulate core module is written back to the on-chip storage module. In this process, the digital multiply-accumulate core module and the in-memory computing core module run in parallel.

[0012] Furthermore, the core module for digital multiplication and accumulation includes several processing units, each processing unit comprising a digital multiplier, a digital adder, and an input / output register; wherein, the digital multiplier is used to perform multiplication operations on the input feature map and model weight parameters in the second data; the digital adder is used to perform accumulation operations on the output of the digital multiplier; and the input / output register is used to temporarily store the input data of the digital multiplier, the intermediate results of the digital adder, and the final result obtained from the operation.

[0013] Secondly, the present invention also provides a method for accelerating in-memory computing neural networks supplemented by a digital multiply-accumulate core, comprising: receiving an operation instruction from an off-chip user, the operation instruction including the network structure of the current neural network model and its corresponding operation allocation scheme; dynamically reconstructing the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the operation instruction; and, based on the data path and the operation allocation scheme, obtaining the data of the current neural network model during operation from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and performing multiply-accumulate operations to obtain the final operation result.

[0014] Furthermore, the current neural network model is MobileNet, which includes depthwise separable convolutional layers and pointwise convolutional layers. In the computational allocation scheme corresponding to MobileNet, the digital multiply-accumulate core module runs the depthwise separable convolutional layers, and the in-memory computing core runs the pointwise convolutional layers. Accordingly, according to the operation instructions, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed, including: according to the network structure of MobileNet and its corresponding computational allocation scheme, dynamically reconstructing the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module.

[0015] Further, based on the data path and computational allocation scheme, the data of the current neural network during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the final computation result is obtained. This includes: based on the first data path and computational allocation scheme, reading the input feature map of the depthwise separable convolutional layer from the on-chip storage module through the digital multiply-accumulate core module, and performing multiply-accumulate operations to obtain the first output feature map; based on the first data path and computational allocation scheme, performing multiply-accumulate operations on the first output feature map through the in-memory computing core module to obtain the final computation result.

[0016] Furthermore, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiply-accumulate operations, and the digital multiply-accumulate core module performs 1 / 9 of the multiply-accumulate operations; correspondingly, according to the operation instructions, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed, including: according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme, dynamically reconstructing the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module.

[0017] Further, based on the data path and computational allocation scheme, the data of the current neural network during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the final computation result is obtained. This includes: based on the second data path and computational allocation scheme, the in-memory computing core module reads 8 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module and performs the corresponding proportion of multiply-accumulate operation to obtain a first intermediate feature map; based on the second data path and computational allocation scheme, the digital multiply-accumulate core module reads 1 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module and performs the corresponding proportion of multiply-accumulate operation to obtain a second intermediate feature map; the first intermediate feature map and the second intermediate feature map are summed to obtain the final computation result.

[0018] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the in-memory computational neural network acceleration method with a digital multiply-accumulate core as described above.

[0019] The present invention provides an in-memory computing neural network accelerator supplemented by a digital multiply-accumulate core, comprising a top-level controller, an in-memory computing core module, an on-chip storage module, and a digital multiply-accumulate core module. The top-level controller receives operation instructions from off-chip users and dynamically reconstructs the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module based on these instructions. The operation instructions include the network structure of the current neural network model and its corresponding computation allocation scheme. The on-chip storage module stores data from the current neural network model during operation, including at least model weight parameters, input feature maps, and output feature maps. The in-memory computing core module, based on the dynamically reconstructed data path and the computation allocation scheme, retrieves first data from the on-chip storage module and executes the corresponding multiply-accumulate operation. The digital multiply-accumulate core module, based on the dynamically reconstructed data path and the computation allocation scheme, retrieves second data from the on-chip storage module and executes the corresponding multiply-accumulate operation. Both the first and second data are included in the data. This accelerator introduces a lightweight digital multiply-accumulate core module into the existing in-memory computing neural network accelerator to assist the in-memory computing core module in handling neural network layers that it is not good at. This enables the accelerator to run a variety of neural network models with higher computing resource utilization, thereby improving the overall performance of the accelerator. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the architecture of an in-memory computing neural network accelerator supplemented by a digital multiply-accumulate core provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the data path when the current neural network model is MobileNet, as provided in this embodiment of the invention.

[0023] Figure 3 This is a schematic diagram of the data path when the current neural network model provided in this embodiment of the invention includes a standard 3×3 convolutional layer.

[0024] Figure 4 This is a flowchart illustrating the in-memory computation neural network acceleration method supplemented by a digital multiplication-accumulation core provided in an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention.

[0026] Reference numerals: 110: Top-level controller; 120: In-memory computing core module; 130: On-chip storage module; 140: Digital multiply-accumulate core module. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] It should be noted that in-memory computation suffers from low computational resource utilization when running some neural network models because the parallelism of multiplication and accumulation operations is high and usually a power of 2 (common parallelism is 32, 64, 128 or higher).

[0029] For example, in the common neural network model MobileNet, the depthwise separable convolution has a multiply-accumulate parallelism of only 9. When it is mapped to the in-memory computing core for operation, the computing resource utilization is usually less than 30%.

[0030] Furthermore, the standard 3×3 convolution is the most common layer in neural networks, and its cumulative parallelism is a multiple of 9. When it is mapped to an in-memory computation core with a cumulative parallelism that is usually a power of 2, the in-memory computation core will be underutilized because it cannot be divided evenly.

[0031] In view of this, the present invention proposes a new in-memory computation neural network accelerator, namely an in-memory computation neural network accelerator supplemented with a digital multiply-accumulate core.

[0032] Specifically, Figure 1 This diagram illustrates the architecture of an in-memory computing neural network accelerator supplemented by a digital multiply-accumulate core, as provided in an embodiment of the present invention.

[0033] like Figure 1 As shown, the in-memory computing neural network accelerator provided in this embodiment of the invention includes a top-level controller 110, an in-memory computing core module 120, an on-chip storage module 130, and a digital multiply-accumulate core module 140. The following will provide a detailed description of the constituent modules of the in-memory computing neural network accelerator.

[0034] First, regarding the top-level controller.

[0035] The top-level controller is used to receive operation instructions from off-chip users and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the operation instructions; among them, the operation instructions include the current network structure of the neural network and its corresponding operation allocation scheme.

[0036] Understandably, based on the interface for communication with external systems, the top-level controller receives operation instructions from users outside the chip in real time. These instructions contain information about the network structure and computation allocation strategy of the currently running neural network model. The top-level controller then parses the received instructions to determine how to allocate multiplication and accumulation operations among the network layers in the current neural network model.

[0037] Based on the parsed operation instructions, the top-level controller dynamically adjusts the data communication between the on-chip storage module, the in-memory computing core module, and the digital multiply-accumulate core module. This data path determines whether the on-chip storage module, the in-memory computing core module, and the digital multiply-accumulate core module can read and transmit data to each other, ensuring that the data flow between these modules meets the requirements for running the current neural network model.

[0038] It is worth mentioning that the top-level controller has the ability to dynamically reconstruct the data path. It can adjust the allocation scheme of multiply-accumulate operations according to the characteristics of different neural network layers (such as convolutional layers, fully connected layers, etc.) in the neural network model, so that the accelerator can run multiple neural network models with the highest computing resource utilization.

[0039] Regarding the on-chip storage module.

[0040] The on-chip storage module is used to store data during the operation of the current neural network model. The data includes at least the model weight parameters, input feature maps, and output feature maps.

[0041] Understandably, on-chip memory modules are an important component of many hardware accelerators (such as GPUs, FPGAs, and ASICs), especially when performing deep neural network inference or training.

[0042] During the operation of the current neural network model, the on-chip storage module is mainly used to store the model weight parameters, input feature maps, and output feature maps, so that they can be read by the on-chip computation core module and / or the digital multiplication and accumulation core module.

[0043] The model weight parameters are the main parameters learned by the current neural network model, used to transform the input feature map in each network layer. The model weight parameters are read-only during inference, but readable and writable during training for parameter updates.

[0044] Input feature maps refer to the data input into the current neural network model, such as pixel values ​​of an image or the output of a previous network layer. During forward propagation, the input feature maps are read and used for computation.

[0045] The output feature map is the computation result of each network layer in the current neural network model and will be used as the input to the next network layer. During forward propagation, the output feature map is written to storage; during backward propagation, the output feature map needs to be read to calculate the gradient.

[0046] Regarding the core module of in-memory computing.

[0047] The in-memory computing core module is used to obtain the first data from the on-chip storage module and perform the corresponding multiply-accumulate operation according to the operation allocation scheme based on the dynamically reconstructed data path.

[0048] Understandably, the in-memory computing core module refers to a processing unit capable of performing logical and arithmetic operations within a memory cell. It typically consists of several in-memory computing macros and is a fundamental component of existing in-memory computing neural network accelerators. Based on the data path reconstructed by the top-level controller, the in-memory computing core module can directly access the primary data stored in the on-chip storage module, including model weight parameters and input feature maps, according to the computation allocation scheme. This avoids the frequent data transfer between storage and computing units as in the traditional von Neumann architecture.

[0049] Then, the corresponding multiply-accumulate operation is performed on the first data read, the intermediate results are temporarily saved, and the next set of data is processed until the calculation of the entire batch is completed, and the final result of the multiply-accumulate operation is obtained.

[0050] Finally, the final result of the multiplication and accumulation operation is written back to the on-chip storage module.

[0051] It should be noted that the in-memory computing core module can not only read the first data from the on-chip storage module, but also read the output feature map from the digital multiplication and accumulation core module according to the operation requirements. The implementation of these data reading operations all depend on the data path reconstructed by the top-level controller.

[0052] Regarding the core of digital multiplication and accumulation.

[0053] The core module for digital multiply-accumulate is used to obtain second data from the on-chip storage module and perform the corresponding multiply-accumulate operation according to the operation allocation scheme based on the dynamically reconstructed data path; wherein, both the first data and the second data are included in the data.

[0054] Understandably, the digital multiply-accumulate core module is a hardware module specifically designed to perform multiplication and accumulation operations common in current neural network models.

[0055] The core module for digital multiplication and accumulation includes several processing units, each containing a digital multiplier, a digital adder, and an input / output register. The digital multiplier performs multiplication on the input feature map and model weight parameters in the second data set; the digital adder accumulates the output of the digital multiplier; and the input / output register temporarily stores the input data of the digital multiplier, the intermediate results of the digital adder, and the final result obtained from the calculation.

[0056] Based on the data path reconstructed by the top-level controller, the digital multiply-accumulate core module can flexibly read the second data from the on-chip storage module and perform multiply-accumulate operations according to the operation allocation scheme, and then write the final result of the multiply-accumulate operation back to the on-chip storage module.

[0057] In this embodiment, the digital multiply-accumulate core module is a new module added compared to the existing in-memory computing neural network accelerator architecture. It can assist the in-memory computing core module in handling neural network layers that it is not good at handling.

[0058] In one specific embodiment, the first category network layer of the current neural network model is run. The multiply-accumulate parallelism of this category network layer is higher than a first threshold and is a power of 2. In this case, only the in-memory computing core module is needed to run this category network layer. The in-memory computing core module reads the input feature map and weights from the on-chip storage module, performs multiply-accumulate operations, and writes the final result of the multiply-accumulate operations back to the on-chip storage module.

[0059] In another specific embodiment, the second type of network layer of the current neural network model is run. The multiply-accumulate parallelism of this type of network layer is higher than the first threshold but is not a power of 2. In this case, the multiply-accumulate operation of this type of network layer is split. The multiply-accumulate operations that are powers of 2 are assigned to the in-memory computing core module, and the remaining multiply-accumulate operations that are not divisible by 2 are assigned to the digital multiply-accumulate core module. The in-memory computing core module and the digital multiply-accumulate core module each perform multiply-accumulate operations, and the results of the two calculations are added together as the final result of the multiply-accumulate operation and written back to the on-chip storage module.

[0060] It should be noted that the first threshold can be set according to the actual parallelism of the in-memory computing core module, and no specific limitation is made here.

[0061] In another specific embodiment, the third category network layer of the current neural network model is run. The multiply-accumulate parallelism of this category network layer is lower than the second threshold. In this case, only the digital multiply-accumulate core module is needed to run this category network layer. The digital multiply-accumulate core module reads the input feature map and weights from the on-chip storage module, performs multiply-accumulate operations, and writes the final result of the multiply-accumulate operations back to the on-chip storage module.

[0062] It should be noted that the second threshold is lower than the first threshold, and can be set according to the minimum parallelism of the in-memory computing core module; no specific limitation is made here.

[0063] Furthermore, the in-memory computing core module and the digital multiply-accumulate core module can also be pipelined neural network layers. For example, the digital multiply-accumulate core module can handle multiply-accumulate operations with relatively flexible parallelism, and its operation result can be used as the input of the in-memory computing core module to handle multiply-accumulate operations with high parallelism and powers of 2, thereby obtaining the final operation result.

[0064] As can be seen from the above, in this embodiment, the in-memory computing core module is good at handling multiply-accumulate operations with high parallelism and powers of 2, while the digital multiply-accumulate core module is good at handling multiply-accumulate operations with arbitrary parallelism. Thus, the digital multiply-accumulate core module can assist the in-memory computing module in handling multiply-accumulate operations with different parallelism, especially those neural network layers that the in-memory computing core module is not good at running.

[0065] It can also be seen that the operation instructions received by the top-level controller are actually determined by the network structure of the current running neural network model and the working characteristics of the in-memory computing core module and the digital multiply-accumulate core module, which helps to improve the utilization rate of the accelerator's computing resources.

[0066] In this embodiment, the in-memory computing neural network accelerator includes a top-level controller, an in-memory computing core module, an on-chip storage module, and a digital multiply-accumulate core module. The top-level controller receives operation instructions from off-chip users and dynamically reconstructs the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module based on these instructions. The operation instructions include the network structure of the current neural network model and its corresponding computation allocation scheme. The on-chip storage module stores data from the current neural network model during operation, including at least model weight parameters, input feature maps, and output feature maps. The in-memory computing core module retrieves first data from the on-chip storage module based on the dynamically reconstructed data path and the computation allocation scheme, and performs the corresponding multiply-accumulate operation. The digital multiply-accumulate core module retrieves second data from the on-chip storage module based on the dynamically reconstructed data path and the computation allocation scheme, and performs the corresponding multiply-accumulate operation. Both the first and second data are included in the data. This accelerator introduces a lightweight digital multiply-accumulate core module into the existing in-memory computing neural network accelerator to assist the in-memory computing core module in handling neural network layers that it is not good at. This enables the accelerator to run a variety of neural network models with higher computing resource utilization, thereby improving the overall performance of the accelerator.

[0067] Based on the above embodiments, the following will further describe in detail the processing procedure when the current neural network model is MobileNet.

[0068] The current neural network model is MobileNet, a common neural network model. MobileNet includes depthwise separable convolutional layers and pointwise convolutional layers. In the computation allocation scheme corresponding to MobileNet, the core module of digit multiplication and accumulation runs the depthwise separable convolutional layers, while the core module of in-memory computation runs the pointwise convolutional layers.

[0069] Correspondingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module based on the MobileNet network structure and its corresponding computation allocation scheme. The first data path can be found in [reference needed]. Figure 2 , Figure 2 This diagram illustrates the data path when the current neural network model provided in this embodiment of the invention is MobileNet.

[0070] like Figure 2 As shown, in the first data path, the digital multiply-accumulate core module reads the input feature map of the depthwise separable convolutional layer from the on-chip storage module, and after completing the depthwise separable convolutional layer operation, sends its output feature map (i.e., the input feature map of the pointwise convolutional layer) to the in-memory computation core module. The in-memory computation core module then writes its output feature map back to the on-chip storage module after completing the pointwise convolutional layer operation. In this process, the digital multiply-accumulate core module and the in-memory computation core module operate in a pipelined manner.

[0071] In this embodiment, since the multiply-accumulate parallelism of the depthwise separable convolutional layer is very low, while the digital multiply-accumulate core module can flexibly adjust the multiply-accumulate parallelism to ensure high computing resource utilization; the multiply-accumulate parallelism of the pointwise convolutional layer is high, and it can also ensure high computing resource utilization when running on the in-memory computing core module. Therefore, the in-memory computing neural network accelerator with digital multiply-accumulate core proposed in this embodiment of the invention can run the MobileNet neural network model with high computing resource utilization.

[0072] Based on the above embodiments, the following will further describe in detail the processing procedure when the current neural network model includes a standard 3×3 convolutional layer.

[0073] It should be noted that the standard 3×3 convolution is the most common layer in convolutional neural network models. Because its cumulative parallelism is a multiple of 9, it cannot be divided evenly by the multiplicative cumulative parallelism of the in-memory computation core module, which is usually a power of 2. This results in low utilization of the in-memory computation core module's computational resources. Therefore, this implementation will specifically focus on the case where current neural network models include standard 3×3 convolutional layers.

[0074] Specifically, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiplication-accumulation operations, and the digital multiplication-accumulation core module performs 1 / 9 of the multiplication-accumulation operations.

[0075] Correspondingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module based on the standard 3×3 convolutional layer and its corresponding operation allocation scheme. The first data path can be found in [reference needed]. Figure 3 , Figure 3 This diagram illustrates the data path of a current neural network model provided by an embodiment of the present invention, which includes a standard 3×3 convolutional layer.

[0076] like Figure 3 As shown, in the second data path, the in-memory computing core module reads 8 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module, and the digital multiply-accumulate core module reads 1 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module. The in-memory computing core module and the digital multiply-accumulate core module complete their respective multiply-accumulate operations in parallel. The operation results of the in-memory computing core module and the operation results of the digital multiply-accumulate core module are summed in the adder of the reconfigurable data path to obtain the result of the complete multiply-accumulate operation of the standard 3×3 convolutional layer (i.e., the output feature map), and the output feature map is written back to the on-chip storage module.

[0077] In this embodiment, by splitting the multiply-accumulate operation of the standard 3×3 convolutional layer, 1 / 9 of the (non-divisible) part of the multiply-accumulate operation of the standard 3×3 convolutional layer is run by the digital multiply-accumulate core module, and the parallelism of the multiply-accumulate core module can be flexibly adjusted. Therefore, both the in-memory computing core module and the digital multiply-accumulate core can achieve high computing resource utilization. The in-memory computing neural network accelerator with digital multiply-accumulate core provided in this embodiment of the invention can run standard 3×3 convolution and neural network models based on standard 3×3 convolution with high computing resource utilization.

[0078] It should be noted that the in-memory computing neural network accelerator with digital multiply-accumulate core provided in this embodiment of the invention can not only run the more common MobileNet neural network model, standard 3×3 convolution, and neural network model based on standard 3×3 convolution as described above, but also run other types of neural network models. The processing process can be referred to the above embodiments, and will not be described in detail here.

[0079] Corresponding to the in-memory computing neural network accelerator with digital multiply-accumulate core described in the above embodiments, the present invention also provides an in-memory computing neural network acceleration method with digital multiply-accumulate core.

[0080] Specifically, Figure 4 The diagram shows a flowchart of the in-memory computation neural network acceleration method with a digital multiply-accumulate core provided by an embodiment of the present invention.

[0081] like Figure 4 As shown, the method includes steps S410-S430, and steps S410-S430 and related steps will be described in detail below.

[0082] S410 receives operation instructions from an external user. The operation instructions include the network structure of the current neural network model and its corresponding computation allocation scheme.

[0083] S420, according to the operation instruction, dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module.

[0084] Understandably, based on the interface for communication with external systems, the top-level controller receives operation instructions from users outside the chip in real time. These instructions contain information about the network structure and computation allocation strategy of the currently running neural network model. The top-level controller then parses the received instructions to determine how to allocate multiplication and accumulation operations among the network layers in the current neural network model.

[0085] Based on the parsed operation instructions, the top-level controller dynamically adjusts the data paths between the on-chip storage module, the in-memory computing core module, and the digital multiply-accumulate core module. These data paths determine whether the on-chip storage module, the in-memory computing core module, and the digital multiply-accumulate core module can read and transmit data to each other, ensuring that the data flow between these modules meets the requirements for running the current neural network model.

[0086] It is worth mentioning that the top-level controller has the ability to dynamically reconstruct the data path. It can adjust the allocation scheme of multiply-accumulate operations according to the characteristics of different neural network layers (such as convolutional layers, fully connected layers, etc.) in the neural network model, so that the accelerator can run multiple neural network models with the highest computing resource utilization.

[0087] S430, based on the data path and operation allocation scheme, the data of the current neural network model during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the multiply-accumulate operation is performed to obtain the final operation result.

[0088] Understandably, both the in-memory computing core module and the digital multiply-accumulate core module can read data from the on-chip storage module during the current operation of the neural network model, based on the data path and operation allocation scheme, and then perform multiply-accumulate operations and sum them up, or complete the multiply-accumulate operations in a pipelined manner to obtain the final calculation result. Subsequently, the final calculation result is written back to the on-chip storage module.

[0089] In a specific embodiment, the current neural network model is MobileNet, which includes depthwise separable convolutional layers and pointwise convolutional layers. In the computation allocation scheme corresponding to MobileNet, the digital multiply-accumulate core module runs the depthwise separable convolutional layers, and the in-memory computation core module runs the pointwise convolutional layers.

[0090] Accordingly, based on the network structure of MobileNet and its corresponding computation allocation scheme, the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed.

[0091] Furthermore, based on the first data path and operation allocation scheme, the input feature map of the depthwise separable convolutional layer is read from the on-chip storage module through the digital multiply-accumulate core module, and multiply-accumulate operation is performed to obtain the first output feature map; based on the first data path and operation allocation scheme, the multiply-accumulate operation is performed on the first output feature map through the in-memory computing core module to obtain the final operation result.

[0092] In another specific embodiment, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiply-accumulate operations, and the digital multiply-accumulate core module performs 1 / 9 of the multiply-accumulate operations.

[0093] Accordingly, based on the standard 3×3 convolutional layer and its corresponding operation allocation scheme, the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed.

[0094] Furthermore, based on the second data path and computation allocation scheme, the in-memory computing core module reads 8 / 9 of the input feature map of the standard 3×3 convolution from the on-chip storage module and performs the corresponding multiply-accumulate operation to obtain the first intermediate feature map; based on the second data path and computation allocation scheme, the digital multiply-accumulate core module reads 1 / 9 of the input feature map of the standard 3×3 convolution layer from the on-chip storage module and performs the corresponding multiply-accumulate operation to obtain the second intermediate feature map; the first intermediate feature map and the second intermediate feature map are summed to obtain the final computation result.

[0095] It should be noted that the in-memory computing neural network acceleration method with digital multiply-accumulate core provided in the embodiments of the present invention can be referred to in correspondence with the in-memory computing neural network accelerator with digital multiply-accumulate core described in the above embodiments, and will not be described in detail here.

[0096] In this embodiment, by receiving operation instructions from an off-chip user, including the network structure of the current neural network model and its corresponding computation allocation scheme, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed according to the operation instructions. Based on the data path and computation allocation scheme, the data of the current neural network model during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and multiply-accumulate operations are performed to obtain the final computation result. This method introduces a lightweight digital multiply-accumulate core module into the original in-memory computing neural network accelerator to assist the in-memory computing core module in handling neural network layers that it is not good at, enabling the operation of multiple neural network models with higher computing resource utilization, improving the computing resource utilization of the accelerator, and thus improving the overall performance of the accelerator.

[0097] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an in-memory computation neural network acceleration method supplemented by a digital multiply-accumulate core. This method includes: receiving operation instructions from an off-chip user, the operation instructions including the network structure of the current neural network model and its corresponding computation allocation scheme; dynamically reconstructing the data path between the in-memory computation core module, the on-chip storage module, and the digital multiply-accumulate core module according to the operation instructions; and, based on the data path and computation allocation scheme, obtaining data from the on-chip storage module during the operation of the current neural network model through the in-memory computation core module and / or the digital multiply-accumulate core module, and performing multiply-accumulate operations to obtain the final computation result.

[0098] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An in-memory computing neural network accelerator supplemented with a digital multiply-accumulate core, characterized in that, include: The system comprises a top-level controller, an in-memory computing core module, an on-chip storage module, and a digital multiply-accumulate core module; among which, The top-level controller is used to receive operation instructions from off-chip users and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the operation instructions; wherein, the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; The on-chip storage module is used to store data of the current neural network model during operation, and the data includes at least model weight parameters, input feature maps, and output feature maps. The in-memory computing core module is used to obtain first data from the on-chip storage module and perform corresponding multiply-accumulate operations based on the dynamically reconstructed data path and the operation allocation scheme. The digital multiply-accumulate core module is used to obtain second data from the on-chip storage module and perform the corresponding multiply-accumulate operation according to the operation allocation scheme based on the dynamically reconstructed data path. Wherein, both the first data and the second data are included in the data; In the case where the current neural network model is MobileNet, MobileNet includes depthwise separable convolutional layers and pointwise convolutional layers; In the computation allocation scheme corresponding to MobileNet, the digital multiply-accumulate core module runs the depthwise separable convolutional layer, and the in-memory computation core module runs the pointwise convolutional layer; Correspondingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the network structure of MobileNet and its corresponding operation allocation scheme. In the first data path, the digital multiply-accumulate core module reads the input feature map of the depthwise separable convolutional layer from the on-chip storage module, and sends its output feature map to the in-memory computing core module after completing the depthwise separable convolutional layer operation. After completing the pointwise convolutional layer operation, the in-memory computing core module writes its output feature map back to the on-chip storage module. In this process, the digital multiply-accumulate core module and the in-memory computing core module operate in a pipeline manner. In the case where the current neural network model includes a standard 3×3 convolutional layer; In the computational allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computation core module performs 8 / 9 of the multiply-accumulate operations, and the digital multiply-accumulate core module performs 1 / 9 of the multiply-accumulate operations. Correspondingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme. In the second data path, the in-memory computing core module reads 8 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module, and the digital multiply-accumulate core module reads 1 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module. The in-memory computing core module and the digital multiply-accumulate core module perform multiply-accumulate operations of their respective proportions, and write the output feature map obtained by summing the operation results of the in-memory computing core module and the operation results of the digital multiply-accumulate core module back to the on-chip storage module. In this process, the digital multiply-accumulate core module and the in-memory computing core module run in parallel.

2. The in-memory computing neural network accelerator with a digital multiply-accumulate core as described in claim 1, characterized in that, The core module for digital multiplication and accumulation includes: Several processing units, each processing unit including a digital multiplier, a digital adder, and an input / output register; wherein, The digital multiplier is used to perform multiplication operations on the input feature map and model weight parameters in the second data; The digital adder is used to perform an accumulation operation on the output of the digital multiplier; The input / output register is used to temporarily store the input data of the digital multiplier, the intermediate results of the digital adder, and the final result obtained from the operation.

3. A method for accelerating in-memory computation neural networks with a digital multiply-accumulate core, applied to the in-memory computation neural network accelerator with a digital multiply-accumulate core as described in claim 1 or 2, characterized in that, include: Receive operation instructions from external users, the operation instructions including the network structure of the current neural network model and its corresponding computation allocation scheme; According to the operation instructions, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed. Based on the data path and computation allocation scheme, the data of the current neural network model during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the multiply-accumulate operation is performed to obtain the final computation result.

4. The in-memory computation neural network acceleration method with a digital multiplication-accumulation core as described in claim 3, characterized in that, The current neural network model is MobileNet, which includes depthwise separable convolutional layers and pointwise convolutional layers. In the computation allocation scheme corresponding to MobileNet, the digital multiply-accumulate core module runs the depthwise separable convolutional layer, and the in-memory computing core runs the pointwise convolutional layer; Accordingly, based on the operation instructions, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed, including: Based on the network structure of MobileNet and its corresponding computation allocation scheme, the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed.

5. The in-memory computation neural network acceleration method with a digital multiplication-accumulation core as described in claim 4, characterized in that, Based on the aforementioned data path and computational allocation scheme, the data of the neural network during its current operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the final computation result is obtained, including: Based on the first data path and operation allocation scheme, the input feature map of the depthwise separable convolutional layer is read from the on-chip storage module through the digital multiply-accumulate core module, and multiply-accumulate operation is performed to obtain the first output feature map; Based on the first data path and computation allocation scheme, the first output feature map is multiplied and accumulated through the in-memory computation core module to obtain the final computation result.

6. The in-memory computation neural network acceleration method with a digital multiplication-accumulation core as described in claim 3, characterized in that, The current neural network model includes a standard 3×3 convolutional layer; In the computational allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computation core module performs 8 / 9 of the multiply-accumulate operations, and the digital multiply-accumulate core module performs 1 / 9 of the multiply-accumulate operations. Accordingly, based on the operation instructions, the data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed, including: Based on the standard 3×3 convolutional layer and its corresponding operation allocation scheme, the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiply-accumulate core module is dynamically reconstructed.

7. The in-memory computation neural network acceleration method with a digital multiplication-accumulation core as described in claim 6, characterized in that, Based on the aforementioned data path and computational allocation scheme, the data of the neural network during its current operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiply-accumulate core module, and the final computation result is obtained, including: Based on the second data path and computation allocation scheme, the in-memory computing core module reads the input feature map of the standard 3×3 convolutional layer (8 / 9) from the on-chip storage module and completes the corresponding multiplication and accumulation operation to obtain the first intermediate feature map. Based on the second data path and operation allocation scheme, the digital multiply-accumulate core module reads 1 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module and completes the corresponding proportion of multiply-accumulate operation to obtain the second intermediate feature map; The first intermediate feature map and the second intermediate feature map are summed to obtain the final calculation result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the in-memory computation neural network acceleration method with a digital multiply-accumulate core as described in any one of claims 3 to 7.

Images