In-memory computing neural network accelerator assisted by digital multiply-accumulate core and acceleration method

Abstract

The invention provides an in-memory computing neural network accelerator assisted by a digital multiply-accumulate core and an acceleration method. The accelerator comprises a top layer controller, an on-chip memory module and a digital multiply-accumulate core module, wherein the top layer controller is used for dynamically reconstructing a data path among the in-memory computing core module, the on-chip memory module and the digital multiply-accumulate core module according to an operation instruction; the on-chip storage module is used for storing data of the current neural network model in the running process; and the in-memory calculation core module and the digital multiply-accumulate core module are used for obtaining data from the on-chip storage module according to the operation instruction based on the dynamically reconstructed data path and executing corresponding multiply-accumulate operation. According to the accelerator, a lightweight digital multiply-accumulate core module is introduced into an original in-memory computing neural network accelerator to assist the in-memory computing core module in processing a neural network layer which is not good at the in-memory computing core module, so that various neural network models can be operated at a relatively high computing resource utilization rate, and the computing resource utilization rate of the accelerator is improved; therefore, the overall performance of the accelerator is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of circuit design, and in particular to an in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core and an acceleration method. Background Art

[0002] Computing-In-Memory (CIM) is an emerging circuit architecture that aims to solve the "memory wall" problem caused by the separation of memory and processor in the traditional von Neumann architecture. In the traditional architecture, data needs to be constantly moved from memory to the processor for calculation, and then the results are written back to the memory, which consumes a lot of time and energy. Computing-In-Memory performs calculations directly inside the storage unit, thereby reducing the need for data movement and improving processing speed and energy efficiency.

[0003] For applications such as neural networks that require a large number of parallel multiplication and accumulation operations, in-memory computing can provide significant advantages. This is because: on the one hand, in-memory computing is usually designed to perform multiple multiplication and accumulation operations at the same time, with high parallelism, which is very suitable for matrix operations in deep learning; on the other hand, since the data transmission between memory and processor is reduced, in-memory computing can significantly reduce power consumption.

[0004] However, when in-memory computing is applied to certain types of neural network layers, some challenges arise. Specifically, the multiplication and accumulation operations of in-memory computing are highly parallelized and the parallelism of in-memory computing cores is usually a power of 2 (such as 32, 64, 128, etc.). Therefore, there is a problem of low computing resource utilization when running some neural network models.

[0005] For example, in lightweight convolutional neural networks such as MobileNet, depthwise separable convolution is a key component. Depthwise separable convolution consists of two steps: first, spatial convolution with a single convolution kernel is applied to each input channel (usually 3x3 convolution), and then pointwise convolution is applied to all output channels. The 3x3 convolution here involves a multiplication-accumulation parallelism of 9 (i.e. 3x3). If such an operation is mapped to an in-memory computing core with a multiplication-accumulation parallelism of 32 or higher, only part of the computing resources will be used, and the rest of the resources will be idle, resulting in low overall computing resource utilization (usually less than 30%).

[0006] The same problem also exists in the standard 3x3 convolution layer, which is a basic building block in many neural network models. Because the parallelism of 3x3 convolution is a multiple of 9, while the parallelism of the in-memory computing core is usually a power of 2, the two cannot be divided evenly, so it will also cause a waste of resources of the in-memory computing core.

[0007] Therefore, how to solve the problem of low computing resource utilization caused by the mismatch between the multiplication and accumulation parallelism of the existing in-memory computing neural network accelerator and the running neural network model is an important issue that needs to be urgently solved in the field of circuit design. Summary of the invention

[0008] The present invention provides an in-memory computing neural network accelerator assisted by a digital multiplication-accumulation core and an acceleration method, so as to overcome the defect of low computing resource utilization caused by the mismatch between the multiplication-accumulation parallelism of the existing in-memory computing neural network accelerator and the running neural network model, improve the computing resource utilization of the in-memory computing neural network accelerator, and further enhance the overall performance of the accelerator.

[0009] On the one hand, the present invention provides an in-memory computing neural network accelerator assisted by a digital multiplication-accumulation core, comprising: a top-level controller, an in-memory computing core module, an on-chip storage module and a digital multiplication-accumulation core module; wherein the top-level controller is used to receive operation instructions from an off-chip user, and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication-accumulation core module according to the operation instructions; wherein the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; the on-chip storage module is used to store data of the current neural network model during operation, and the data at least includes model weight parameters, input feature maps and output feature maps; the in-memory computing core module is used to obtain first data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; the digital multiplication-accumulation core module is used to obtain second data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; wherein the first data and the second data are both included in the data.

[0010] Furthermore, the current neural network model is MobileNet, and MobileNet includes a depthwise separable convolution layer and a pointwise convolution layer; in the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depthwise separable convolution layer, and the in-memory computing core module runs the pointwise convolution layer; accordingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the network structure of MobileNet and its corresponding operation allocation scheme; in the first data path, the digital multiplication and accumulation core module reads the input feature map of the depthwise separable convolution layer from the on-chip storage module, and sends its output feature map to the in-memory computing core module after completing the depthwise separable convolution layer operation, and the in-memory computing core module writes its output feature map back to the on-chip storage module after completing the pointwise convolution layer operation. In this process, the digital multiplication and accumulation core module and the in-memory computing core module operate in a pipeline manner.

[0011] Furthermore, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiplication and accumulation operations, and the digital multiplication and accumulation core module performs 1 / 9 of the multiplication and accumulation operations; accordingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme; in the second data path, the in-memory computing core module receives the data from the on-chip storage module The digital multiplication and accumulation core module reads 8 / 9 of the input feature map of the standard 3×3 convolution layer from the on-chip storage module, the in-memory computing core module and the digital multiplication and accumulation core module complete the multiplication and accumulation operations of their respective corresponding proportions, and write the output feature map obtained by summing the operation result of the in-memory computing core module and the operation result of the digital multiplication and accumulation core module back to the on-chip storage module. In this process, the digital multiplication and accumulation core module and the in-memory computing core module run in parallel.

[0012] Furthermore, the digital multiplication and accumulation core module includes: a number of processing units, each processing unit includes a digital multiplier, a digital adder and an input-output register; wherein the digital multiplier is used to perform multiplication operations on the input feature map and model weight parameters in the second data; the digital adder is used to perform accumulation operations on the results output by the digital multiplier; the input-output register is used to temporarily store the input data of the digital multiplier, the intermediate results of the digital adder and the final results obtained by the operation.

[0013] In a second aspect, the present invention also provides an in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core, comprising: receiving an operation instruction from an off-chip user, the operation instruction including the network structure of the current neural network model and its corresponding operation allocation scheme; according to the operation instruction, dynamically reconstructing the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module; based on the data path and the operation allocation scheme, obtaining the data of the current neural network model during operation from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and performing multiplication and accumulation operations to obtain the final operation result.

[0014] Furthermore, the current neural network model is MobileNet, and MobileNet includes a depthwise separable convolution layer and a pointwise convolution layer; in the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depthwise separable convolution layer, and the in-memory computing core runs the pointwise convolution layer; accordingly, according to the operation instruction, the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed, including: according to the network structure of MobileNet and its corresponding operation allocation scheme, the first data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

[0015] Furthermore, based on the data path and the computational allocation scheme, the data of the current neural network during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and the final operation result is obtained by execution, including: based on the first data path and the computational allocation scheme, the input feature map of the depthwise separable convolutional layer is read from the on-chip storage module through the digital multiplication and accumulation core module, and a multiplication and accumulation operation is performed to obtain a first output feature map; based on the first data path and the computational allocation scheme, a multiplication and accumulation operation is performed on the first output feature map through the in-memory computing core module to obtain the final operation result.

[0016] Furthermore, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module runs 8 / 9 multiplication and accumulation operations, and the digital multiplication and accumulation core module runs 1 / 9 multiplication and accumulation operations; accordingly, according to the operation instruction, the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed, including: according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme, the second data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

[0017] Further, based on the data path and the operation allocation scheme, the data of the current neural network in the running process is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and the final operation result is obtained by execution, including: based on the second data path and the operation allocation scheme, the in-memory computing core module runs to read 8 / 9 of the input feature map of the standard 3×3 convolution layer from the on-chip storage module and completes the multiplication and accumulation operations of the corresponding proportion to obtain a first intermediate feature map; based on the second data path and the operation allocation scheme, the digital multiplication and accumulation core module runs to read 1 / 9 of the input feature map of the standard 3×3 convolution layer from the on-chip storage module and completes the multiplication and accumulation operations of the corresponding proportion to obtain a second intermediate feature map; the first intermediate feature map and the second intermediate feature map are summed to obtain the final operation result.

[0018] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements any of the above-described in-memory computing neural network acceleration methods assisted by a digital multiplication and accumulation core.

[0019] The in-memory computing neural network accelerator assisted by a digital multiplication-accumulation core provided by the present invention comprises a top-level controller, an in-memory computing core module, an on-chip storage module and a digital multiplication-accumulation core module; wherein the top-level controller is used to receive operation instructions from an off-chip user, and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication-accumulation core module according to the operation instructions; wherein the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; the on-chip storage module is used to store the data of the current neural network model during operation, and the data at least includes model weight parameters, input feature maps and output feature maps; the in-memory computing core is used to obtain first data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; the digital multiplication-accumulation core module is used to obtain second data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; wherein the first data and the second data are both included in the data. The accelerator introduces a lightweight digital multiplication and accumulation core module into the original in-memory computing neural network accelerator to assist the in-memory computing core module in processing the neural network layers that it is not good at. It can run a variety of neural network models with higher computing resource utilization, improve the computing resource utilization of the accelerator, and thus improve the overall performance of the accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the architecture of an in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core provided in an embodiment of the present invention.

[0022] Figure 2 It is a schematic diagram of a data path when the current neural network model provided by an embodiment of the present invention is MobileNet.

[0023] Figure 3 It is a schematic diagram of the data path when the current neural network model provided by an embodiment of the present invention includes a standard 3×3 convolutional layer.

[0024] Figure 4 It is a flow chart of a method for accelerating an in-memory computing neural network assisted by a digital multiplication and accumulation core provided in an embodiment of the present invention.

[0025] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention.

[0026] Reference numerals: 110: top-level controller; 120: in-memory computing core module; 130: on-chip storage module; 140: digital multiplication and accumulation core module. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that since the in-memory calculation has a high degree of parallelism in multiplication and accumulation operations and is usually a power of 2 (common degrees of parallelism are 32, 64, 128 or higher), there is a problem of low computing resource utilization when running some neural network models.

[0029] For example, for the depthwise separable convolution in the common neural network model MobileNet, its multiplication and accumulation parallelism is only 9. When it is mapped to the in-memory computing core for operation, the computing resource utilization is usually less than 30%.

[0030] In addition, the standard 3×3 convolution is the most common layer in neural networks, and its accumulation parallelism is a multiple of 9. When it is mapped to the in-memory computing core, which usually has a multiplication and accumulation parallelism of a power of 2, the in-memory computing core utilization rate is also low because it cannot be divided integerly.

[0031] Taking this into consideration, the present invention proposes a new in-memory computing neural network accelerator, namely, an in-memory computing neural network accelerator supplemented by a digital multiplication and accumulation core.

[0032] Specifically, Figure 1 A schematic diagram of the architecture of an in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core provided by an embodiment of the present invention is shown.

[0033] like Figure 1 As shown, the in-memory computing neural network accelerator provided by the embodiment of the present invention includes a top-level controller 110, an in-memory computing core module 120, an on-chip storage module 130, and a digital multiplication and accumulation core module 140. The following will describe in detail the constituent modules of the in-memory computing neural network accelerator.

[0034] First, a word about the top-level controller.

[0035] The top-level controller is used to receive operation instructions from off-chip users and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the operation instructions; wherein the operation instructions include the network structure of the current neural network and its corresponding operation allocation plan.

[0036] It is understandable that, based on the interface for external communication, the top-level controller will receive operation instructions from users outside the chip in real time, and the operation instructions contain the network structure information and operation allocation strategy of the current neural network model being run. Then, the top-level controller will parse the received operation instructions to determine how to allocate the multiplication and accumulation operations of each network layer in the current neural network model.

[0037] Based on the parsed operation instructions, the top-level controller will dynamically adjust the data communication between the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module. The data path here determines whether the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module can read and transmit data to each other, ensuring that the data flow between the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module can meet the requirements of running the current neural network model.

[0038] It is worth mentioning that the top-level controller has the ability to dynamically reconstruct the data path. It can adjust the distribution scheme of multiplication and accumulation operations according to the characteristics of different neural network layers (such as convolutional layers, fully connected layers, etc.) in the neural network model, so that the accelerator can run multiple neural network models with the highest computing resource utilization.

[0039] About on-chip storage modules.

[0040] The on-chip storage module is used to store the data of the current neural network model during operation, and the data includes at least model weight parameters, input feature maps, and output feature maps.

[0041] Understandably, on-chip memory blocks are an essential component of many hardware accelerators (e.g., GPUs, FPGAs, ASICs), especially when performing deep neural network inference or training.

[0042] During the operation of the current neural network model, the on-chip storage module is mainly used to store model weight parameters, input feature maps, and output feature maps for reading by the in-memory computing core module and / or the digital multiplication and accumulation core module.

[0043] The model weight parameters refer to the main parameters learned by the current neural network model, which are used to transform the input feature map in each network layer. The model weight parameters are read-only during inference, but readable and writable during training to update the parameters.

[0044] The input feature map refers to the data input into the current neural network model, such as the pixel values ​​of an image or the output of the previous network layer. During the forward propagation process, the input feature map is read and used for calculation.

[0045] The output feature map is the calculation result of each network layer of the current neural network model, and will be used as the input of the next network layer. During the forward propagation process, the output feature map is written to the storage; during the backward propagation process, the output feature map needs to be read to calculate the gradient.

[0046] About the in-memory computing core module.

[0047] The in-memory computing core module is used to obtain first data from the on-chip storage module and perform corresponding multiplication and accumulation operations based on a dynamically reconstructed data path according to an operation allocation scheme.

[0048] It can be understood that the in-memory computing core module refers to a processing unit that can perform logical and arithmetic operations inside a storage unit, which is usually composed of a number of in-memory computing macros and is a basic component of existing in-memory computing neural network accelerators. The in-memory computing core module can directly access the first data stored in the on-chip storage module according to the calculation allocation scheme based on the data path reconstructed by the top-level controller, including model weight parameters and input feature maps, thereby avoiding the frequent data transfer process between storage and computing units under the traditional von Neumann architecture.

[0049] Then, the corresponding multiplication-accumulation operation is performed on the first data read, the intermediate result is temporarily saved, and the next set of data is processed until the calculation of the entire batch is completed to obtain the final result of the multiplication-accumulation operation.

[0050] Finally, the final result of the multiply-accumulate operation is written back to the on-chip memory module.

[0051] It should be noted that the in-memory computing core module can not only read the first data from the on-chip storage module, but also read its output feature map from the digital multiplication and accumulation core module according to the calculation requirements, and the implementation of these data reading operations depends on the data path reconstructed by the top-level controller.

[0052] About digital multiplication and accumulation core.

[0053] The digital multiplication-accumulation core module is used to obtain second data from the on-chip storage module and perform corresponding multiplication-accumulation operations based on a dynamically reconstructed data path according to an operation allocation scheme; wherein the first data and the second data are both included in the data.

[0054] It can be understood that the digital multiplication and accumulation core module is a hardware module specially designed to perform multiplication and accumulation operations commonly used in current neural network models.

[0055] The digital multiplication and accumulation core module includes several processing units, each of which includes a digital multiplier, a digital adder, and an input and output register. The digital multiplier is used to perform multiplication operations on the input feature map and the model weight parameter in the second data; the digital adder is used to perform accumulation operations on the results output by the digital multiplier; and the input and output register is used to temporarily store the input data of the digital multiplier, the intermediate results of the digital adder, and the final results obtained by the operation.

[0056] Based on the data path reconstructed by the top-level controller, the digital multiplication-accumulation core module can flexibly read the second data from the on-chip storage module and perform multiplication-accumulation operations according to the operation allocation scheme, and then write the final result of the multiplication-accumulation operations back to the on-chip storage module.

[0057] In this embodiment, the digital multiplication and accumulation core module is a newly added module compared to the existing in-memory computing neural network accelerator architecture, which can assist the in-memory computing core module in processing the neural network layers that it is not good at processing.

[0058] In a specific embodiment, the first category of network layers of the current neural network model is run, and the multiplication and accumulation parallelism of this category of network layers is higher than the first threshold and is a power of 2. In this case, it is only necessary to use the in-memory computing core module to run this category of network layers. The in-memory computing core module reads the input feature map and weights from the on-chip storage module, performs multiplication and accumulation operations, and writes the final results of the multiplication and accumulation operations back to the on-chip storage module.

[0059] In another specific embodiment, a second category of network layers of the current neural network model is run, and the multiplication and accumulation parallelism of this category of network layers is higher than the first threshold but is not a power of 2. In this case, the multiplication and accumulation operations of this category of network layers are split, and the multiplication and accumulation operations of the power of 2 are assigned to the in-memory computing core module, and the remaining multiplication and accumulation operations that are not divisible by 2 are assigned to the digital multiplication and accumulation core module. The in-memory computing core module and the digital multiplication and accumulation core module each perform multiplication and accumulation operations, and the calculation results of the two are added together and written back to the on-chip storage module as the final result of the multiplication and accumulation operation.

[0060] It should be noted that the first threshold can be set according to the actual parallelism of the in-memory computing core module and is not specifically limited here.

[0061] In another specific embodiment, the third category of network layers of the current neural network model is run, and the multiplication and accumulation parallelism of this category of network layers is lower than the second threshold. In this case, only a digital multiplication and accumulation core module is needed to run this category of network layers. The digital multiplication and accumulation core module reads the input feature map and weights from the on-chip storage module, performs multiplication and accumulation operations, and writes the final results of the multiplication and accumulation operations back to the on-chip storage module.

[0062] It should be noted that the second threshold is lower than the first threshold and can be set according to the minimum parallelism processed by the in-memory computing core module, and is not specifically limited here.

[0063] In addition, the in-memory computing core module and the digital multiplication and accumulation core module can also form a neural network layer in the form of a pipeline. For example, the digital multiplication and accumulation core module can process multiplication and accumulation operations with relatively flexible parallelism, and its operation results can be used as input to the in-memory computing core module to process multiplication and accumulation operations with high parallelism and a power of 2, thereby obtaining the final operation result.

[0064] According to the above, in this embodiment, the in-memory computing core module is good at processing multiplication and accumulation operations with high parallelism and a power of 2, while the digital multiplication and accumulation core module is good at processing multiplication and accumulation operations with arbitrary parallelism. Therefore, the digital multiplication and accumulation core module can assist the in-memory computing module in processing multiplication and accumulation operations with different degrees of parallelism, especially those neural network layers that the in-memory computing core module is not good at running.

[0065] It can also be seen from this that the operation instructions received by the top-level controller are actually determined by the operation allocation plan based on the network structure of the current neural network model being run and the working characteristics of the in-memory computing core module and the digital multiplication and accumulation core module, which is conducive to improving the computing resource utilization of the accelerator.

[0066] In this embodiment, the in-memory computing neural network accelerator includes a top-level controller, an in-memory computing core module, an on-chip storage module and a digital multiplication-accumulation core module; wherein the top-level controller is used to receive operation instructions from off-chip users, and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication-accumulation core module according to the operation instructions; wherein the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; the on-chip storage module is used to store data of the current neural network model during operation, and the data at least includes model weight parameters, input feature maps and output feature maps; the in-memory computing core module is used to obtain first data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; the digital multiplication-accumulation core module is used to obtain second data from the on-chip storage module according to the operation allocation scheme based on the dynamically reconstructed data path and perform corresponding multiplication-accumulation operations; wherein the first data and the second data are both included in the data. The accelerator introduces a lightweight digital multiplication and accumulation core module into the original in-memory computing neural network accelerator to assist the in-memory computing core module in processing the neural network layers that it is not good at. It can run a variety of neural network models with higher computing resource utilization, improve the computing resource utilization of the accelerator, and thus improve the overall performance of the accelerator.

[0067] On the basis of the above embodiments, the processing process when the current neural network model is MobileNet will be described in detail below.

[0068] The current neural network model is MobileNet, which is a common neural network model. MobileNet includes depthwise separable convolutional layers and pointwise convolutional layers. In the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depthwise separable convolutional layers, and the in-memory computing core module runs the pointwise convolutional layers.

[0069] Correspondingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computing core module, the on-chip storage module, and the digital multiplication and accumulation core module according to the network structure of MobileNet and its corresponding operation allocation scheme. The first data path can be found in Figure 2 , Figure 2 A schematic diagram of a data path when the current neural network model provided by an embodiment of the present invention is MobileNet is shown.

[0070] like Figure 2 As shown, in the first data path, the digital multiplication and accumulation core module reads the input feature map of the depthwise separable convolution layer from the on-chip storage module, and after completing the depthwise separable convolution layer operation, sends its output feature map (i.e., the input feature map of the point-by-point convolution layer) to the in-memory computing core module, and the in-memory computing core module writes its output feature map back to the on-chip storage module after completing the point-by-point convolution layer operation. In this process, the digital multiplication and accumulation core module and the in-memory computing core module operate in the form of a pipeline.

[0071] In this embodiment, since the multiplication and accumulation parallelism of the depthwise separable convolution layer is very low, and the digital multiplication and accumulation core module can flexibly adjust the multiplication and accumulation parallelism to ensure a higher computing resource utilization rate; the multiplication and accumulation parallelism of the point-by-point convolution layer is relatively high, and it can also ensure a higher computing resource utilization rate when running on the in-memory computing core module. Therefore, the in-memory computing neural network accelerator assisted by the digital multiplication and accumulation core proposed in the embodiment of the present invention can run the MobileNet neural network model with a higher computing resource utilization rate.

[0072] On the basis of the above-mentioned embodiment, the processing process when the current neural network model includes a standard 3×3 convolutional layer will be described in detail below.

[0073] It should be noted that the standard 3×3 convolution is the most common layer in the convolutional neural network model. Since its accumulation parallelism is a multiple of 9, it cannot be divided by the multiplication and accumulation parallelism of the in-memory computing core module, which is usually a power of 2, resulting in low computing resource utilization of the in-memory computing core module. Therefore, this implementation will be described in particular for the current neural network model including the standard 3×3 convolution layer.

[0074] Specifically, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module runs 8 / 9 of the multiplication and accumulation operations, and the digital multiplication and accumulation core module runs 1 / 9 of the multiplication and accumulation operations.

[0075] Correspondingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module, and the digital multiplication and accumulation core module according to the standard 3×3 convolution layer and its corresponding operation allocation scheme. Figure 3 , Figure 3 A schematic diagram of a data path when the current neural network model provided by an embodiment of the present invention includes a standard 3×3 convolutional layer is shown.

[0076] like Figure 3 As shown, in the second data path, the in-memory computing core module reads 8 / 9 of the input feature maps of the standard 3×3 convolutional layer from the on-chip storage module, and the digital multiplication and accumulation core module reads 1 / 9 of the input feature maps of the standard 3×3 convolutional layer from the on-chip storage module. The in-memory computing core module and the digital multiplication and accumulation core module complete the multiplication and accumulation operations of their corresponding proportions in parallel, and the operation results of the in-memory computing core module and the operation results of the digital multiplication and accumulation core module are summed in the adder of the reconfigurable data path, thereby obtaining the complete multiplication and accumulation operation result of the standard 3×3 convolutional layer (i.e., the output feature map), and the output feature map is written back to the on-chip storage module.

[0077] In this embodiment, the multiplication and accumulation operations of the standard 3×3 convolutional layer are split, and 1 / 9 of the multiplication and accumulation operations of the standard 3×3 convolutional layer (which cannot be divided evenly) is run by the digital multiplication and accumulation core module, and the multiplication and accumulation parallelism of the digital multiplication and accumulation core module can be flexibly adjusted. Therefore, both the in-memory computing core module and the digital multiplication and accumulation core can achieve higher computing resource utilization. The in-memory computing neural network accelerator assisted by the digital multiplication and accumulation core provided in the embodiment of the present invention can run standard 3×3 convolution and a neural network model based on standard 3×3 convolution with higher computing resource utilization.

[0078] It should be noted that the in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core provided in the embodiment of the present invention can not only run the more common MobileNet neural network model, standard 3×3 convolution, and neural network model based on standard 3×3 convolution described above, but also other types of neural network models. The processing process can refer to the above embodiments and will not be described one by one here.

[0079] Corresponding to the in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core described in the above embodiments, the present invention also provides an in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core.

[0080] Specifically, Figure 4 A schematic flow chart of an in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core provided in an embodiment of the present invention is shown.

[0081] like Figure 4 As shown, the method includes steps S410-S430, and steps S410-S430 and related steps will be described in detail below.

[0082] S410, receiving an operation instruction from an off-chip user, where the operation instruction includes the network structure of the current neural network model and its corresponding operation allocation scheme.

[0083] S420, dynamically reconstructing the data path between the in-memory computing core module, the on-chip storage module, and the digital multiplication and accumulation core module according to the operation instruction.

[0084] It is understandable that, based on the interface for external communication, the top-level controller will receive operation instructions from users outside the chip in real time, and the operation instructions contain the network structure information and operation allocation strategy of the current neural network model being run. Then, the top-level controller will parse the received operation instructions to determine how to allocate the multiplication and accumulation operations of each network layer in the current neural network model.

[0085] Based on the parsed operation instructions, the top-level controller will dynamically adjust the data path between the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module. The data path here determines whether the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module can read and transmit data to each other, ensuring that the data flow between the on-chip storage module, the in-memory computing core module, and the digital multiplication and accumulation core module can meet the requirements of running the current neural network model.

[0086] It is worth mentioning that the top-level controller has the ability to dynamically reconstruct the data path. It can adjust the distribution scheme of multiplication and accumulation operations according to the characteristics of different neural network layers (such as convolutional layers, fully connected layers, etc.) in the neural network model, so that the accelerator can run multiple neural network models with the highest computing resource utilization.

[0087] S430, based on the data path and operation allocation scheme, obtain the data of the current neural network model during operation from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and perform multiplication and accumulation operations to obtain the final operation result.

[0088] It is understandable that both the in-memory computing core module and the digital multiplication and accumulation core module can read the data of the current neural network model in operation from the on-chip storage module based on the data path and operation allocation scheme, and perform multiplication and accumulation operations on each of them and then sum them, or complete the multiplication and accumulation operations in the form of a pipeline to obtain the final operation result. Subsequently, the final operation result is written back to the on-chip storage module.

[0089] In a specific embodiment, the current neural network model is MobileNet, which includes a depthwise separable convolution layer and a pointwise convolution layer; in the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depthwise separable convolution layer, and the in-memory computing core module runs the pointwise convolution layer.

[0090] Accordingly, according to the network structure of MobileNet and its corresponding operation allocation scheme, the first data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

[0091] Furthermore, based on the first data path and the operation allocation scheme, the input feature map of the depthwise separable convolutional layer is read from the on-chip storage module through the digital multiplication and accumulation core module, and a multiplication and accumulation operation is performed to obtain a first output feature map; based on the first data path and the operation allocation scheme, a multiplication and accumulation operation is performed on the first output feature map through the in-memory computing core module to obtain a final operation result.

[0092] In another specific embodiment, the current neural network model includes a standard 3×3 convolutional layer; in the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module runs 8 / 9 of the multiplication and accumulation operations, and the digital multiplication and accumulation core module runs 1 / 9 of the multiplication and accumulation operations.

[0093] Correspondingly, according to the standard 3×3 convolutional layer and its corresponding operation allocation scheme, the second data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

[0094] Furthermore, based on the second data path and the operation allocation scheme, the in-memory computing core module is used to run the input feature map of 8 / 9 of the standard 3×3 convolution read from the on-chip storage module and complete the multiplication and accumulation operations of the corresponding proportion to obtain a first intermediate feature map; based on the second data path and the operation allocation scheme, the digital multiplication and accumulation core module is used to run the input feature map of 1 / 9 of the standard 3×3 convolution layer read from the on-chip storage module and complete the multiplication and accumulation operations of the corresponding proportion to obtain a second intermediate feature map; the first intermediate feature map and the second intermediate feature map are summed to obtain the final operation result.

[0095] It should be noted that the in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core provided in the embodiment of the present invention can correspond to the in-memory computing neural network accelerator assisted by a digital multiplication and accumulation core described in the above embodiments, and they will not be described one by one here.

[0096] In this embodiment, by receiving an operation instruction from an off-chip user, the operation instruction includes the network structure of the current neural network model and its corresponding operation allocation scheme; according to the operation instruction, dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module; based on the data path and the operation allocation scheme, the in-memory computing core module and / or the digital multiplication and accumulation core module obtain the data of the current neural network model during operation from the on-chip storage module, and perform multiplication and accumulation operations to obtain the final operation result. This method introduces a lightweight digital multiplication and accumulation core module into the original in-memory computing neural network accelerator to assist the in-memory computing core module in processing the neural network layer that it is not good at, and can run a variety of neural network models with a higher computing resource utilization rate, thereby improving the computing resource utilization rate of the accelerator and thus improving the overall performance of the accelerator.

[0097] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the in-memory computing neural network acceleration method assisted by the digital multiplication and accumulation core, the method comprising: receiving an operation instruction from an off-chip user, the operation instruction including the network structure of the current neural network model and its corresponding operation allocation scheme; according to the operation instruction, dynamically reconstructing the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module; based on the data path and the operation allocation scheme, obtaining the data of the current neural network model during operation from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and performing multiplication and accumulation operations to obtain the final operation result.

[0098] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An in-memory computing neural network accelerator supplemented with a digital multiply-accumulate core, characterized in that: include: Top-level controller, in-memory computing core module, on-chip storage module and digital multiplication and accumulation core module; among them, The top-level controller is used to receive operation instructions from off-chip users and dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the operation instructions; wherein the operation instructions include the network structure of the current neural network model and its corresponding operation allocation scheme; The on-chip storage module is used to store data of the current neural network model during operation, wherein the data at least includes model weight parameters, input feature maps, and output feature maps; The in-memory computing core module is used to obtain first data from the on-chip storage module and perform corresponding multiplication and accumulation operations according to the operation allocation scheme based on the dynamically reconfigured data path; The digital multiplication-accumulation core module is used to obtain second data from the on-chip storage module and perform corresponding multiplication-accumulation operations according to the operation allocation scheme based on the dynamically reconfigured data path; Wherein, the first data and the second data are both included in the data.

2. The in-memory computing neural network accelerator with digital multiplication and accumulation core according to claim 1, characterized in that: The current neural network model is MobileNet, which includes a depth-separable convolution layer and a point-by-point convolution layer; In the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depth-wise separable convolution layer, and the in-memory computing core module runs the point-by-point convolution layer; Accordingly, the top-level controller is specifically used to dynamically reconstruct the first data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the network structure of MobileNet and its corresponding operation allocation scheme; In the first data path, the digital multiplication and accumulation core module reads the input feature map of the depthwise separable convolution layer from the on-chip storage module, and sends its output feature map to the in-memory computing core module after completing the depthwise separable convolution layer operation. The in-memory computing core module writes its output feature map back to the on-chip storage module after completing the point-by-point convolution layer operation. In this process, the digital multiplication and accumulation core module and the in-memory computing core module operate in a pipeline manner.

3. The in-memory computing neural network accelerator with digital multiplication and accumulation core according to claim 1, characterized in that: The current neural network model includes a standard 3×3 convolutional layer; In the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiplication and accumulation operations, and the digital multiplication and accumulation core module performs 1 / 9 of the multiplication and accumulation operations; Accordingly, the top-level controller is specifically used to dynamically reconstruct the second data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module according to the standard 3×3 convolution layer and its corresponding operation allocation scheme; In the second data path, the in-memory computing core module reads 8 / 9 of the input feature maps of the standard 3×3 convolutional layer from the on-chip storage module, and the digital multiplication and accumulation core module reads 1 / 9 of the input feature maps of the standard 3×3 convolutional layer from the on-chip storage module. The in-memory computing core module and the digital multiplication and accumulation core module complete the multiplication and accumulation operations of their respective corresponding proportions, and write the output feature map obtained by summing the operation result of the in-memory computing core module and the operation result of the digital multiplication and accumulation core module back to the on-chip storage module. In this process, the digital multiplication and accumulation core module and the in-memory computing core module run in parallel.

4. The in-memory computing neural network accelerator with digital multiplication and accumulation core according to claim 1, characterized in that: The digital multiplication and accumulation core module includes: A plurality of processing units, each processing unit comprising a digital multiplier, a digital adder and an input and output register; wherein, The digital multiplier is used to perform a multiplication operation on the input feature map and the model weight parameter in the second data; The digital adder is used to perform accumulation operation on the results output by the digital multiplier; The input and output registers are used to temporarily store the input data of the digital multiplier, the intermediate results of the digital adder, and the final results obtained by operation.

5. A method for accelerating neural network computing in memory with a digital multiplication and accumulation core, characterized in that: include: Receiving an operation instruction from an off-chip user, wherein the operation instruction includes a network structure of a current neural network model and a corresponding operation allocation scheme; According to the operation instruction, dynamically reconstruct the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module; Based on the data path and operation allocation scheme, the data of the current neural network model during operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and multiplication and accumulation operations are performed to obtain the final operation result.

6. The in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core according to claim 5, characterized in that: The current neural network model is MobileNet, which includes a depth-separable convolution layer and a point-by-point convolution layer; In the operation allocation scheme corresponding to MobileNet, the digital multiplication and accumulation core module runs the depth-wise separable convolution layer, and the in-memory computing core runs the point-by-point convolution layer; Accordingly, according to the operation instruction, the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed, including: According to the network structure of MobileNet and its corresponding operation allocation scheme, the first data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

7. The in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core according to claim 6, characterized in that: Based on the data path and the operation amount distribution scheme, the data of the current neural network in operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and the final operation result is obtained by execution, including: Based on the first data path and the operation allocation scheme, the input feature map of the depthwise separable convolutional layer is read from the on-chip storage module through the digital multiplication and accumulation core module, and a multiplication and accumulation operation is performed to obtain a first output feature map; Based on the first data path and the operation allocation scheme, the first output feature map is multiplied and accumulated by the in-memory computing core module to obtain a final operation result.

8. The in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core according to claim 5, characterized in that: The current neural network model includes a standard 3×3 convolutional layer; In the operation allocation scheme corresponding to the standard 3×3 convolutional layer, the in-memory computing core module performs 8 / 9 of the multiplication and accumulation operations, and the digital multiplication and accumulation core module performs 1 / 9 of the multiplication and accumulation operations; Accordingly, according to the operation instruction, the data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed, including: According to the standard 3×3 convolutional layer and its corresponding operation allocation scheme, the second data path between the in-memory computing core module, the on-chip storage module and the digital multiplication and accumulation core module is dynamically reconstructed.

9. The in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core according to claim 8, characterized in that: Based on the data path and the operation amount distribution scheme, the data of the current neural network in operation is obtained from the on-chip storage module through the in-memory computing core module and / or the digital multiplication and accumulation core module, and the final operation result is obtained by execution, including: Based on the second data path and operation allocation scheme, the in-memory computing core module runs to read 8 / 9 of the input feature maps of the standard 3×3 convolutional layer from the on-chip storage module and complete the multiplication and accumulation operations of the corresponding proportions to obtain a first intermediate feature map; Based on the second data path and operation allocation scheme, the digital multiplication and accumulation core module is used to read 1 / 9 of the input feature map of the standard 3×3 convolutional layer from the on-chip storage module and complete the multiplication and accumulation operation of the corresponding proportion to obtain a second intermediate feature map; The first intermediate feature map and the second intermediate feature map are summed to obtain a final calculation result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the in-memory computing neural network acceleration method assisted by a digital multiplication and accumulation core as described in any one of claims 5 to 9.

Images