DSP resource reuse method for convolutional neural network hardware calculation

Abstract

The invention discloses a DSP (Digital Signal Processor) resource reuse method for convolutional neural network hardware calculation, which is suitable for structure optimization of a multiplication and addition unit in FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) hardware. The method is based on a mathematical model P = (A + D) * B, operands A and D are 25-bit wide, and the operands A and D are obtained by performing left shift expansion and zero filling splicing on 8-bit input data; the operand B is 18-bit wide and is formed by expanding an 8-bit input feature map or a weight signal; an output result P is 48 bits and represents a multiplication and addition calculation result. By means of the method, two multiplication results can be obtained through one DSP resource calculation at the same time, almost 50% of multiplier resources are saved while the operation precision and the correctness of the convolution result are kept, the DSP unit utilization rate is increased, and the hardware area and power consumption are reduced. The scheme is suitable for various neural network acceleration scenes, has good universality and engineering implementation value, and is particularly suitable for being applied to embedded deep learning chips with limited resources.

Description

Technical Field

[0001] This invention belongs to the field of convolutional neural network hardware acceleration, specifically to a method for efficiently reusing digital signal processor (DSP) resources during convolutional neural network (CNN) computations. This method can be widely applied in scenarios such as embedded systems, system-on-chip (SoC), and edge computing devices. By optimizing DSP time and resource scheduling, it improves the computational efficiency of the neural network inference phase and is suitable for the design of low-power, high-performance accelerators specifically for deep learning. Background Art

[0002] With the rapid development of artificial intelligence (AI), convolutional neural networks (CNNs) have become the mainstream model in deep learning applications due to their outstanding performance in tasks such as image recognition, speech processing, object detection, and natural language processing. Convolutional neural network architectures contain numerous convolution, activation, pooling, and fully connected operations, with convolutional layers accounting for the vast majority of the computational load. To meet the demands of real-time computing and low-power applications, hardware accelerators, as a key support method, have garnered widespread attention from both academia and industry.

[0003] In the design of convolutional neural network hardware accelerators, DSPs are widely used to perform computationally intensive tasks such as convolution, thanks to their highly optimized multiplication and addition operations, strong parallel computing capabilities, and low power consumption. Typically, the computational process of a convolutional neural network can be decomposed into a large number of matrix operations, and the multipliers and adders within the DSP are able to efficiently handle these operations, making it a crucial computing core within the accelerator.

[0004] However, as convolutional neural network models become increasingly complex, hardware computing tasks have gradually become multi-layered, non-uniform, and dynamically changing, leading to the following challenges for DSP resources: on the one hand, different network layers occupy computing resources unevenly, and in some stages the DSP is in a low-load or even idle state, resulting in a waste of computing resources; on the other hand, when processing multiple tasks or multiple channels in parallel, there is a lack of flexible multiplexing mechanisms between DSPs, resulting in uneven resource allocation or scheduling conflicts, reducing the overall system throughput and energy efficiency.

[0005] Some current hardware architectures attempt to manage DSP resources through static partitioning or centralized allocation, but these still suffer from issues such as rigid scheduling, poor scalability, and low resource utilization, making them difficult to adapt to the dynamic and diverse CNN model structures and complex application scenarios. This is especially true in embedded systems and edge computing devices, where power consumption, area, and performance are highly sensitive, placing higher demands on DSP resource reuse.

[0006] Therefore, a DSP resource reuse method suitable for convolutional neural network hardware acceleration is proposed. Through the reasonable division of computing tasks, dynamic resource scheduling and parallel utilization mechanism design, efficient utilization of DSP resources is achieved, and the overall performance and power consumption ratio of the accelerator are improved. This is of great significance for promoting the design of deep learning dedicated chips and the popularization of smart terminals. Summary of the Invention

[0007] This paper addresses the problems of low DSP resource utilization, inflexible scheduling, and uneven task load distribution in existing convolutional neural network hardware acceleration. By proposing a DSP resource reuse method for convolutional neural network hardware computation, the paper proposes a DSP resource reuse method for convolutional neural network hardware computation. Unlike traditional accelerators that statically allocate or bind DSP resources, this paper devises a DSP dynamic reuse mechanism based on task feature recognition and scheduling control, enabling flexible allocation and efficient coordination of DSP resources across different computational stages, significantly improving resource utilization and overall system performance.

[0008] The technical solution adopted in the present invention is:

[0009] The core mathematical model of this invention is: P = (A + D) × B. This model is highly compatible with the structure of the multiplication-addition unit in modern FPGAs and ASICs, such as the Xilinx DSP48E1, facilitating integration and hardware implementation. Operands A and D are each formed by left-shifting and zero-padding 8-bit input data; operand B is obtained by expanding the 8-bit input signal; and the output, P, represents the result of the multiplication-addition calculation. Typically, the high-order bits of P are used to indicate overflow and sign extension, while the low-order bits are the valid calculation result. Based on specific application requirements, the required effective bit width is extracted through truncation, rounding, and other methods to meet the accuracy and range requirements of subsequent calculations.

[0010] In the above scheme, the operands A and D are both 25 bits wide, and are specifically formed by 8-bit input data through left shift expansion and zero padding; among them, A is formed by a[7:0] left shifted 16 bits and padded with zeros, and D is expanded to 25 bits wide by d[7:0].

[0011] In the above scheme, the operand B is 18 bits wide and is obtained by sign extension or zero extension of the 8-bit input signal b[7:0] to meet the multiplication width requirement.

[0012] In the above scheme, the output result P is 48 bits wide, which fully represents the result of the multiplication and addition operation, of which the lower 32 bits are the effective product and sum result, and the upper 16 bits are used for sign extension and overflow flag.

[0013] In the above scheme, the multiplication and addition unit structure refers to Xilinx DSP48E1, which supports 25×18-bit multiplication and 48-bit addition.

[0014] In the above scheme, the multiplication and addition result P is cached through the pipeline register, and the pipeline depth is generally set to 3 levels to ensure high-speed data transmission and synchronization.

[0015] In the above scheme, the input data width is 8 bits, which supports the multiplication and addition calculation requirements of different convolution kernels and biases, ensuring calculation flexibility and accuracy.

[0016] In the above scheme, according to the computational characteristics of different layers of the convolutional neural network, the splicing method of operands A and D is dynamically adjusted to support multiple operation modes.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] (1) Through the operand splicing and expansion mechanism, the multi-function reuse of the multiplication and addition unit is realized, which greatly improves the DSP resource utilization and avoids the idleness of hardware resources.

[0019] (2) Through intelligent scheduling strategies, it supports parallel execution of multiple tasks and multiple channels, improving computing efficiency and system throughput.

[0020] (3) Through a lightweight scheduling interface, fast task switching and data consistency are ensured to meet the needs of real-time reasoning scenarios.

[0021] (4) Two multiplication and addition operations are completed through one DSP, which reduces power consumption and chip area, and improves the economy and practicality of hardware design.

[0022] (5) By unifying the multiplication and addition units to reuse multiple input data streams, the hardware computing throughput is effectively improved, the system latency is reduced, and the demand for efficient computing of convolutional neural networks is met.

[0023] (6) Compatible with existing mainstream DSP hardware architecture, multiplication and addition operations are reasonably mapped to the DSP48E1 unit in FPGA or ASIC, which is easy to integrate and expand. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of a typical DSP multiplication-addition unit structure according to an embodiment of the present invention.

[0025] Figure 2 Schematic diagram of a mathematical model for operand concatenation and expansion of a multiplier-accumulator input A according to an embodiment of the present invention.

[0026] Figure 3 Schematic diagram of a mathematical model for operand concatenation and expansion of a multiplier-accumulator input D according to an embodiment of the present invention.

[0027] Figure 4 Schematic diagram of a mathematical model for operand concatenation and expansion of the multiplier-accumulator input B according to an embodiment of the present invention.

[0028] Figure 5 Schematic diagram of a mathematical model for operand concatenation and expansion of a multiplier-accumulator input P according to an embodiment of the present invention.

[0029] Figure 6 This is a DSP computing resource allocation diagram for an embodiment of the present invention, where the weights of the convolutional neural network serve as B input.

[0030] Figure 7 This is a DSP computing resource allocation diagram for an embodiment of the present invention, with the feature map of the convolutional neural network serving as input B. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples.

[0032] like Figure 1 Figure 2 shows the typical DSP multiply-add unit architecture. It includes an input register, a pre-adder, a 25×18 multiplier, a 48-bit accumulation / logic unit, and a pattern detector. A and D are 25 bits wide and first pass through the pre-adder for addition and subtraction. B, 18 bits wide, is multiplied with the pre-adder output. The product is then fed into a 48-bit accumulator for addition, subtraction, or logic operation, resulting in a 48-bit output, P.

[0033] like Figure 2 Figure 2 shows the concatenation of operand A. Input A is 8 bits wide (a[7:0]). Before entering the DSP operation, it is left-shifted 16 bits, padded with 0s in the lower bits, and padded with a[7] at the 25th bit, expanding it to 25 bits. This means: A_ext[24:0] = ((a[7:0]<<16)|0)|((a[7]<<24)|0). This operation aligns the low-precision input data to the data width supported by the DSP multiply-add unit to ensure the validity of the multiplication calculation.

[0034] like Figure 3 Figure 2 shows the concatenation and expansion of operand D. Input D is 8 bits wide (d[7:0]) and is sign-extended to 25 bits before entering the DSP. Specifically, the highest bit, d[7], is used as the sign bit, replicated 16 times toward the upper bits, and an additional d[7] is appended after the lowest 8 bits to form a complete 25-bit signed data. This means: D_ext[24:0] = {{16{d[7]}}, d[7], d[7:0]}. This extension ensures sign correctness in operations such as multiplication and accumulation.

[0035] like Figure 4Figure 2 shows the expansion of operand B. Input B is 8 bits wide (b[7:0]), which is expanded to the 18-bit width required by the DSP unit. Before entering the DSP operation, the sign bit is extended to 18 bits. This is done by using the highest bit, b[7], as the sign bit and replicating it 10 times toward the upper bits, i.e., B_ext[17:0] = {{10{b[7]}}, b[7:0]}.

[0036] like Figure 5 The figure shows the concatenated structure of the output result P. According to the standard output configuration of DSP48E1, the multiplication and addition result P can be up to 48 bits wide. Here we only focus on the lower 36 bits, of which the upper 18 bits store the result of a[7:0]*b[7:0], and the lower 18 bits store the result of d[7:0]*b[7:0]. By intercepting the data of the corresponding bits, we can get the results of the two multiplication operations respectively. The specific interception operation is as follows: directly intercept l[15:0], which is the multiplication result of d[7:0]*b[7:0]. According to the positive or negative condition of d[7:0]*b[7:0], the specific result of h[15:0] is determined. If d[7:0]*b[7:0]>=0, then h[15:0]=a[7:0]*b[7:0]; otherwise, h[15:0]=a[7:0]*b[7:0]-1.

[0037] like Figure 6 As shown in the figure, combined with the above DSP multiplexing operation, in resource allocation method 1, the weight parameters of the convolutional neural network are input as operand B, and different data streams enter the DSP core respectively by multiplexing A and D, realizing the operation of multiple input channels sharing the same convolution kernel, improving the efficiency of convolution kernel multiplexing, and being suitable for the common input feature map channel grouping convolution strategy.

[0038] like Figure 7 As shown in the figure, combined with the above DSP multiplexing operation, in resource allocation method 2, the input feature map of the convolutional neural network is input as operand B, while A and D are dynamically provided by the weight group, which improves the overall hardware execution efficiency and reduces the bus bandwidth pressure.

[0039] This embodiment adapts and optimizes the traditional DSP multiplication-addition unit structure by rationally constructing the mathematical model P = (A + D) × B. This method fully utilizes the 25 × 18-bit multiplication and 48-bit addition supported by the DSP48E1 architecture in Xilinx FPGAs. This method not only balances the accuracy and complexity of convolution calculations, but also significantly improves the efficiency of neural network acceleration calculations, offering excellent portability and engineering application value.

Claims

1. A DSP resource reuse method for convolutional neural network hardware computing, characterized by: This method is suitable for structural reuse of multiplication-addition units in convolutional neural networks. Using the mathematical model P = (A + D) × B, the input operands A and D are concatenated and expanded as addition operands, and the input operand B is concatenated and expanded as multiplication operands. The calculation result is output as P. A and D are 25-bit concatenated inputs, B is an 18-bit expanded input, and P is a 48-bit output (actually only 36 bits are considered). This method allows two multiplication results to be calculated simultaneously using one DSP resource, saving nearly 50% of multiplier resources while maintaining operational accuracy and the correctness of the convolution results.

2. The DSP resource reuse method according to claim 1, wherein: Both A and D are generated by left-shifting and expanding two 8-bit input channels according to a specified rule and splicing them by zero filling. The total bit width of the splicing result is 25 bits.

3. The DSP resource reuse method according to claim 1, wherein: The B operand is spliced ​​into 18 bits after the most significant bit of the 8-bit input is extended and input to the multiplier path in the DSP module.

4. The DSP resource reuse method according to claim 1, wherein: The P is the multiplication and addition result output by the DSP unit, with a bit width of 36 bits. According to the specified rules, different bits are intercepted to obtain the calculation results of two multiplications, which can be directly sent to the next-level adder or accumulator register for neural network convolution operation.

5. The DSP resource reuse method according to claim 1, wherein: The A input path corresponds to the A port in the XilinxDSP48E1 structure, D corresponds to the D port, B input corresponds to the B port, and the output P is the result of the multiplication and addition path.

6. The DSP resource reuse method according to claim 1, characterized in that: The DSP internally adds A and D through the ALU module, multiplies the result by B, and obtains the multiplication and addition result P, logically realizing P = (A + D) × B.

7. The DSP resource reuse method according to claim 1, wherein: The splicing structure supports convolution kernel weights or feature maps as B inputs, and the B channel has configuration switching logic to adapt to different convolution stages.

8. The DSP resource reuse method according to claim 1, wherein: This method realizes the resource merging of two multipliers in convolutional neural network calculation by sharing the input splicing path and reusing the DSP core, saving at least one 18×25-bit multiplier resource.

9. The DSP resource reuse method according to claim 1, wherein: The resource reuse structure supports a bit width range of 8 bits for the convolution input feature map, a bit width range of 8 bits for the convolution kernel weight, and supports 36-bit high-precision operations for the convolution accumulation output.

Images