Acceleration method and device of 3D convolutional neural network based on multi-dimensional dynamic parallel computing
By adopting multi-dimensional dynamic parallel computing method in 3D CNN hardware accelerator, dynamically selecting parallel strategies and adjusting parallelism degrees, the problems of low computing efficiency and waste of resources of existing 3D CNN hardware accelerators are solved, and higher computing efficiency and performance optimization are achieved.
Patent Information
- Application Number
- CN202411869843.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-11
AI Technical Summary
The existing 3D CNN hardware accelerators lack efficient parallel computing methods for time-oriented dimensions, resulting in low computing efficiency, serious resource waste, and inflexible parallel computing strategies, which are unable to adapt to the differences between different network structures and FPGA devices.
The 3D convolutional neural network acceleration method based on multi-dimensional dynamic parallel computing is adopted to dynamically select the time depth, filter and feature image pixel three-dimensional parallel strategy or filter and channel two-dimensional parallel strategy, and optimize the computing efficiency by dynamically adjusting the parallelism degree and data flow storage and transmission mode.
It effectively improves the acceleration efficiency of 3D CNN, adapts to the calculation acceleration of different 3D CNN models, reduces the amount of intermediate redundant data, reduces data transmission delay and energy consumption, and optimizes performance.
Smart Images
Figure CN119808842A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for accelerating a 3D convolutional neural network based on multi-dimensional dynamic parallel computing. Background Art
[0002] The input information of the 3D CNN model is a multi-frame feature map that is continuous in time. Since the convolution kernel in 3D CNN extracts features in the time dimension by sliding between multiple feature maps and exhibits good performance, it is widely used in application fields such as action recognition. Figure 1 Demonstrates the specific operation of 3D CNN.
[0003] Previous studies have shown that 2D CNN has a higher computational load than traditional machine learning algorithms. 3D CNN also requires convolution in the time dimension, which increases the computational load by 10x to 100x, greatly increasing the computing power requirements of edge computing platforms. Therefore, how to deploy 3D CNN on resource-constrained edge devices is a current research difficulty. The pseudo code for 3D convolution operations is as follows: Figure 2 As shown, the first two rows are two sets of loop operations added by 3D CNN compared to 2D CNN.
[0004] In terms of CNN hardware acceleration, current research focuses on the hardware acceleration architecture of CNN computing, and its core technologies include hardware-friendly methods such as loop unrolling and data tiling. The core operation of loop unrolling is to expand a single for loop in the pseudocode into multiple inner loops and process them in parallel, thereby increasing the instruction set parallelism (i.e., LIP) of the inner loop. The application of loop unrolling in hardware acceleration is to input multiple data in a single loop and perform parallel calculations separately. This method not only improves computing efficiency, but also realizes data reuse and effectively reduces the number of storage accesses. Data tiling is to cut and store input data in blocks according to the needs of parallel computing. This method not only effectively reduces the overhead of on-chip storage resources and avoids data redundancy, but also reduces data transmission delays during hardware acceleration.
[0005] Existing CNN hardware accelerators usually use 2D convolution, i.e., the loop in lines 3 to 7 in the pseudocode, to achieve loop unrolling. The method based on line 3 is called filter parallelism (Pf); the parallel method based on line 4 is called channel parallelism (Pc); the parallel strategy based on the height or width of the input features is called pixel parallelism (Pv), corresponding to lines 5 and 6 in the pseudocode; and the parallel strategy based on the height and width of the convolution kernel is called convolution kernel parallelism (Pk), corresponding to lines 7 and 8. However, existing CNN hardware accelerators do not fully consider the computational characteristics of 3D convolution, which results in the time dimension characteristics in 3D convolution calculations not being fully utilized.
[0006] When the existing convolutional neural network hardware accelerator design scheme is applied to 3D CNN, the main problems are as follows:
[0007] (1) Lack of efficient parallel computing methods for the time dimension. Most of the existing hardware accelerators are designed based on 2D CNN, and the computing architecture for 3D CNN fails to fully consider its time dimension characteristics. If these hardware architectures are used to accelerate 3D CNN, it will lead to a large amount of intermediate data redundancy and data transmission delay.
[0008] (2) There is a lack of a universal high-performance 3D CNN hardware architecture. Most existing 3D CNN hardware accelerators are hardware accelerators for specific models. If different 3-D CNN network models are deployed, it will cause a lot of resource waste and low computing efficiency.
[0009] (3) Inefficient parallel computing strategies. Figure 3 As shown in the figure, the change pattern of the features and weight parameters of 3D CNN with the depth of the model is similar to that of 2D CNN: since the pooling operation will reduce the size of the network model, as the number of network layers increases, the number of pooling operations increases, and the parameters of the input features will also tend to decrease relatively, while the weight parameters will tend to increase due to the increase in the number of input channels and filters. Existing 3D CNN hardware accelerators usually adopt a fixed parallel strategy. As the number of network layers changes, the computing array designed based on a fixed degree of parallelism may face load imbalance, resulting in low computing efficiency.
[0010] In general, due to the structural characteristics of 3D convolutional neural networks, parallel computing schemes designed based on a fixed degree of parallelism cannot guarantee high efficiency in the deployment of each layer of CNN, which leads to the inability to fully utilize computing resources and the great differences in computing efficiency on different network structures and different FPGA devices. Summary of the invention
[0011] The present invention provides a method and device for accelerating a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, aiming to improve the acceleration efficiency of the 3D CNN.
[0012] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:
[0013] An acceleration method for a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, dynamically selecting a first parallel strategy or a second parallel strategy according to the structural parameters of each convolution layer of the 3D convolutional neural network;
[0014] The first parallel strategy is time depth, filter and feature map pixel three-dimensional parallelism, and P total =Pt1 *P 1 *Pv 1 ; Among them, P total represents the total computational parallelism of the 3D convolutional neural network; Pt 1 , Pf 1 , Pv 1 They are the parallelism of the first parallel strategy in time depth, filter and feature map pixel dimensions, and their values are all integer powers of 2;
[0015] The second parallel strategy is to parallelize the filters and channels in two dimensions, and P total =Pf 2 *Pc 2 ; Pf 2 、Pc 2 They are the degrees of parallelism of the second parallel strategy in the filter and channel dimensions, respectively, and their values are integer powers of 2.
[0016] Furthermore, if the size of a convolutional layer input feature map is larger than Pt set in the first parallel strategy 1 , Pv 1 And the number of channels is relatively smaller than the Pc set in the second parallel strategy 2 , the first parallel strategy is selected; if the size of a convolutional layer input feature map is smaller than Pt set in the first parallel strategy 1 , Pv 1 And the number of channels is greater than or equal to Pc set in the second parallel strategy 2 , select the second parallel strategy.
[0017] Furthermore, the first parallel strategy adopts a three-dimensional array structure composed of a plurality of MAC units for parallel computing, wherein each MAC unit is composed of a multiplier and an accumulator; wherein a single two-dimensional array constituting the three-dimensional array structure receives feature data from different time frames, and each two-dimensional array shares weight data of the same time frame; each column of a single two-dimensional array receives feature data from different Pf 1 The weight parameters of the filter group, different rows in each column share the same weight data from the same filter and the Pv of the same row of feature data from the same time frame 1 Characteristic data;
[0018] The second parallel strategy uses a two-dimensional array structure composed of several MAC units for parallel computing; wherein,
[0019] Each column in the two-dimensional array structure receives the 2 The weight parameters of the filter bank, different rows in each column receive the weights from Pc 2 The characteristic parameters of the input channels and Pc 2The weight parameter of the input channels.
[0020] Furthermore, an addition tree is configured for each column of each two-dimensional array;
[0021] In the first parallel strategy, parallel calculation of the channel dimension is not involved, so the addition tree of each column in the two-dimensional array is disabled, and each column of MAC units calculates Pv in parallel on the feature map pixel dimension 1 The three-dimensional array structure uses a total accumulator to perform time dimension accumulation operations on the output results of all two-dimensional arrays;
[0022] In the second parallel strategy, each column of the two-dimensional array enables an adder tree to implement channel-dimensional addition operations on the output results of all MAC units, and finally outputs a summation result for each column.
[0023] Furthermore, in the first parallel strategy, the weights are stored as follows: Ks of the same channel of different filter groups on the same time dimension 2 The weight data are stored at the same address; the feature data is stored in the following way: a row of input feature data is stored at each address, and data shifting is achieved through multi-level on-chip buffering to meet the data requirements of feature pixel dimension parallelism.
[0024] Furthermore, in the second parallel strategy, the weights are stored as follows: 2 The Pc in the filters with the same spatial position 2 The weight data is stored in the same address; the feature data is stored in the following way: Pc is stored in each address 2 data to meet the data requirements of channel dimension parallelism.
[0025] The present invention also provides an acceleration device for a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, the acceleration device comprising a weight buffer, a data buffer, a control module, a convolution engine, a pooling module and an activation function module; the implementation steps of the acceleration device include:
[0026] The host computer dynamically selects the first parallel strategy or the second parallel strategy for the current convolution layer according to the structural parameters of the current convolution layer, and sends the selected parallel strategy to the control module;
[0027] The control module sends a dynamic adjustment instruction to other hardware modules according to the received parallel strategy that needs to be executed currently, and the other hardware modules make corresponding adjustments according to the received adjustment instructions; the other hardware modules include: a weight buffer, a data buffer, a convolution engine, a pooling module and an activation function module;
[0028] The control module generates a buffer read signal and sends it to the weight buffer and the feature buffer. The two buffers transmit the weight data and feature data to the convolution engine.
[0029] The control module generates a calculation signal and sends it to the convolution engine;
[0030] The convolution engine performs convolution calculations based on the received weight data and feature data;
[0031] After receiving the receipt signal from the convolution engine, the control module generates a pooling and activation module enable signal;
[0032] The pooling and activation modules perform pooling and activation processing on the convolutional data in turn;
[0033] The control module sends a write signal to the weight buffer and feature buffer to transmit the activated data to the off-chip DDR;
[0034] The first parallel strategy is time depth, filter and feature map pixel three-dimensional parallelism, and P total =Pt 1 *P 1 *Pv 1 ; Among them, P total is the total computational parallelism of the 3D convolutional neural network, Pt 1 , Pf 1 , Pv 1 They are the parallelism of the first parallel strategy in time depth, filter and feature map pixel dimensions, and their values are all integer powers of 2;
[0035] The second parallel strategy is to parallelize the filters and channels in two dimensions, and P total =Pf 2 *Pc 2 ; Pf 2 、Pc 2 They are the degrees of parallelism of the first parallel strategy in the filter and channel dimensions, respectively, and their values are integer powers of 2.
[0036] Compared with the prior art, the present invention has the following beneficial technical effects:
[0037] 1. The present invention utilizes the data characteristics of the spatiotemporal dimension of 3-D convolution, adopts a multi-dimensional collaborative dynamic parallel computing method, and proposes an acceleration method for 3D convolutional neural network, which effectively avoids the problem of low computational efficiency of existing 3D CNN acceleration methods;
[0038] 2. The hardware architecture of the present invention is based on multi-dimensional collaborative dynamic parallel computing in time and space, that is, the dynamic array structure therein, which can adapt to the computing acceleration of different 3D CNN models and obtain higher computing efficiency and performance compared with the existing 3D CNN accelerators.
[0039] 3. The present invention designs the data flow storage and transmission mode of the accelerator based on the multi-dimensional collaborative parallel strategy of time and space, which can realize data reuse in the calculation process, reduce the amount of intermediate redundant data, reduce the delay caused by data transmission and the energy consumption caused by data access, so as to achieve the purpose of saving storage resources and optimizing performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a 3D convolution operation;
[0041] Figure 2 It is the pseudo code of 3D convolution calculation;
[0042] Figure 3 It is the 3D convolution feature, weight parameter quantity and network layer change characteristics;
[0043] Figure 4 It is a dynamic computing array of the first parallel strategy in the embodiment of the present application;
[0044] Figure 5 It is a dynamic computing array of the second parallel strategy in the embodiment of the present application;
[0045] Figure 6 It is the addition tree structure described in the embodiment of the present application;
[0046] Figure 7 This is the weight storage mode of the first parallel strategy in the embodiment of the present application;
[0047] Figure 8 This is the weight storage mode of the second parallel strategy in the embodiment of the present application;
[0048] Fig. 9 This is the characteristic data storage mode of the first parallel strategy in the embodiment of the present application;
[0049] Fig.10 This is the characteristic data storage mode of the first parallel strategy in the embodiment of the present application;
[0050] Fig.11 It is the overall hardware architecture of the accelerator described in the embodiment of the present application. DETAILED DESCRIPTION
[0051] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.
[0052] According to the unique time dimension characteristics of 3D convolution, this solution introduces time depth parallelism, which is represented by Pt here. The specific method is: Figure 2The first line of the pseudocode represents the time-depth loop, which is expanded and then operated in parallel. When using Pt for parallel computing, it is necessary to input Pt frame feature data, which shares the same weight data. The intermediate data cache required for a D-dimensional loop will be reduced by Pt times. This design effectively reduces the burden of on-chip storage and is also conducive to the accelerator's data reuse in the convolution kernel time dimension (Kt).
[0053] Considering the problems existing in the existing 3D CNN accelerator, the present invention proposes an acceleration method for a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, which adopts three-dimensional parallel computing of Pt, Pf, and Pv combined with two-dimensional parallel computing of Pc and Pf. Specifically: according to the structural parameters of each convolution layer of the 3D convolutional neural network, the first parallel strategy or the second parallel strategy is dynamically selected; the first parallel strategy is three-dimensional parallel computing of time depth, filter and feature map pixel, and P total =Pt 1 *P 1 *Pv 1 ; Among them, P total P represents the total computational parallelism of the 3D convolutional neural network, which is determined by analyzing the hardware resources of the target computing platform. total The value of Pt 1 , Pf 1 , Pv 1 are the parallelism of the first parallel strategy in time depth, filter and feature map pixel dimensions, and their values are all integer powers of 2; the second parallel strategy is two-dimensional parallelism of filters and channels, and P total =Pf 2 *Pc 2 ; Pf 2 、Pc 2 They are the degrees of parallelism of the second parallel strategy in the filter and channel dimensions, respectively, and their values are integer powers of 2.
[0054] According to the parameter changes of the existing 3D CNN, if the size of the input feature map of a convolutional layer is larger than Pt set in the first parallel strategy 1 , Pv 1 And the number of filter groups and channels is relatively smaller than the Pf set in the second parallel strategy 2 、Pc 2 , the first parallel strategy is selected; if the size of a convolutional layer input feature map is smaller than Pt set in the first parallel strategy 1 , Pv 1 And the number of filter groups and channels is greater than or equal to Pf set in the second parallel strategy 2 、Pc 2When , the second parallel strategy is selected. This method can effectively improve the load imbalance of the convolution engine and keep the utilization rate and computational efficiency of parallel resources as high as possible in each cycle when accelerating different convolution layers.
[0055] Assume that the total computational parallelism of the 3D convolutional neural network is P total =256. Set the parallelism of the first parallel strategy to: Pv1=8, Pt1=8, Pf1=4; Set the parallelism of the second parallel strategy to: Pf2=16; Pc2=16. There is a lightweight convolutional neural network. The width (W), height (H), channel (C), filter group (F), and time depth (D) of the first convolution layer are 8, 8, 3, 16, 8 respectively; the width, height, channel, filter group, and time depth group of the second data layer are 4, 4, 16, 32, 4 respectively. The width and height of the first layer are very suitable for strategy 1, and the number of channels (3) is not suitable for Pc2 in the second parallel strategy. Therefore, it is more suitable for the first data layer to use the first parallel strategy; while the width (4) and height of the second layer are less than the Pv parallelism (8) of the first parallel strategy, Pt parallelism will also face the same problem, which will cause 4 of the 8 parallel computing units to be unable to receive data for work, that is, there will be a load imbalance problem. The input channels and filter groups of the second layer are larger than Pc2 and Pf2 respectively, which can meet the amount of data required for parallel processing in one cycle. Therefore, the second parallel strategy is more suitable for accelerating the second convolutional layer.
[0056] In this embodiment, the array structure of MAC (accumulated multiplication and addition operation) unit and addition tree unit is used to realize the convolution engine of 3D convolutional neural network based on multi-dimensional dynamic parallel computing. Since the MAC unit is a relatively independent block on the FPGA chip, it can be dynamically deployed into a corresponding array structure according to the different parallel strategies implemented in each layer. Figure 4 , Figure 5 shown.
[0057] When a three-dimensional array structure is applied, a single two-dimensional array constituting the three-dimensional array receives feature data from different frames, wherein each two-dimensional array shares the same frame of convolution kernel data. Each column of the single two-dimensional array receives weight parameters from a different filter group (F), and different rows in each column share the same set of feature parameters.
[0058] When a two-dimensional array structure is applied, each column receives weight parameters from a different filter bank (F), and different rows in each column receive feature parameters from different input channels.
[0059] Assume that the maximum total parallelism P that can be supported based on the target device resource analysis total=256, the shallower layers of the network (such as 1 to 4 layers) can use Pt=4, Pv=16, and Pf=4; as the network depth increases and the input feature size becomes smaller (such as 5 to 9 layers), Pt and Pv should be reduced to a certain extent, and Pf should be increased. At this time, Pt=2, Pv=8, and Pf=16 are adopted; when the network model reaches a deeper layer, the size of the input feature is too small to support large values of Pv and Pt in parallel. At this time, the three-dimensional array can be reorganized into a two-dimensional array, and the parallel strategy of Pf=16 and Pc=16 is adopted. This example also illustrates that the dynamic array in the present invention is not only reflected in the mutual conversion between the three-dimensional array and the two-dimensional array, but also can dynamically adjust the parallelism size in the face of models of different depths and types.
[0060] In order to reduce the accumulation calculation time of Pc, the addition tree module is used in the convolution engine. After receiving the enable signal, the addition tree module performs the corresponding accumulation operation. The addition tree structure is as follows Figure 6 As shown, because the degree of parallelism is often set to an exponential power of 2, the addition tree is usually composed of log 2 The number of adders required for the first layer of the adder tree is P, where P represents the degree of parallelism. The number of adders required for the first layer of the adder tree is P, and then decreases by an exponential multiple of 2 layer by layer. In this implementation, Pc needs to use the adder tree for summation calculation. For example, when the current layer Pc=16, 4 layers of adders are required to build the adder tree, where: the first layer requires 16 adders, the second layer requires 8 adders, the third layer requires 4 adders, and the fourth layer requires 2, for a total of 30 adders.
[0061] The convolution operation of 3-D CNN in the time dimension means that the data after array operation still needs to be accumulated by the total accumulator to obtain the final calculation result. The total accumulator is composed of multiple independent accumulator units and is used for accumulation calculation in the time dimension. Strategy 2 does not involve parallelism in the time dimension, so the array designed based on strategy 2 does not need the total accumulator. When implementing strategy 2, the total accumulator is disabled to save hardware resources. The maximum number of accumulator units included in the total accumulator of this embodiment is: P total .
[0062] The MAC unit consists of a multiplier and an accumulator. When performing Pv parallel calculations, the intermediate data obtained by Pv calculations will be stored in the accumulator and output after the final result is obtained through accumulation.
[0063] Taking into full consideration the temporal characteristics of 3-D CNN, in a more optimal embodiment, different data stream storage and transmission methods are further designed for the two parallel strategies. Figure 7 , Figure 8 As shown in the figure: Different from the weight data of 2-D CNN, the weight data of 3-D CNN has one more dimension, namely the time depth of the Kt convolution kernel. Figure 7 The design is a storage method designed for the first parallel strategy, considering that Pt parallelism only requires one frame of weight data in one parallel calculation, Pv parallel calculation requires one weight data in space, and Pf parallelism requires Pf weight data at the same position from different filter groups (F). This storage method stores Ks of the same channel (C) of different filter groups (F) at the same time depth (Kt), that is, different time frames. 2 The weight data are stored in the same address, so that the data stream they constitute is highly adaptable to the parallel operation based on the first parallel strategy; Figure 8 The Pc weight data with the same spatial position in the Pf filters are stored in the same address. This storage method also ensures the high adaptability of the parallel computing of the second parallel strategy.
[0064] Similar to weight calculation storage, feature data storage provides two dynamic storage solutions for two different parallel strategies. Fig. 9 , Fig.10 As shown, the maximum number of on-chip memory blocks (BRAM) used by the feature buffer is Pt max , that is, the maximum parallelism in the time depth of the parallel strategy is adopted. In the schematic diagram, Pt max Simplified to Pt. This design allows the convolution engine to efficiently transmit the required data, which is beneficial to improving the performance of the hardware accelerator. Fig. 9 The block method is a feature data storage scheme designed for the first parallel strategy: the block method stores a row of input feature data at each address, and realizes data shifting through multi-level on-chip buffering to meet the data requirements of Pv parallelism; Fig.10 The block method is designed for the second parallel strategy, storing Pc data in each address to meet the data requirements of Pc parallelism. The number of input feature frames (D) when using the second parallel strategy is less than or equal to Pt max Therefore, each BRAM is designed with enable control to adapt to different memory requirements.
[0065] The present invention also provides an acceleration device for a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, wherein the acceleration device is as follows: Fig.11 As shown, it includes a weight buffer, a data buffer, a control module, a convolution engine, a pooling module and an activation function module; the implementation steps of the acceleration device include:
[0066] (1) The host computer dynamically selects the first parallel strategy or the second parallel strategy for the current convolutional layer according to the structural parameters of the current convolutional layer, and sends the selected parallel strategy to the control module;
[0067] After receiving the currently implemented parallel strategy, the control module generates an adjustment signal and sends it to the rebuffer, data buffer, convolution engine, pooling module and activation function module. Different modules will dynamically make corresponding adjustments according to the current parallel strategy.
[0068] Before dynamically selecting the parallel strategy, the trained network weights and input feature data have been stored in the off-chip DDR memory through the host computer. Some data has been pre-stored in the on-chip weight cache and data cache. The host computer determines the parallel strategy implemented by each layer of the network to the control module.
[0069] After the host computer sends a start signal to the control module, the control module starts its internal state machine, and the state machine generates control signals for different modules according to the working status.
[0070] The first parallel strategy is time depth, filter and feature map pixel three-dimensional parallelism, and P total =Pt 1 *P 1 *Pv 1 ; Among them, P total represents the total computational parallelism of the 3D convolutional neural network; Pt 1 , Pf 1 , Pv 1 They are the parallelism of the first parallel strategy in time depth, filter and feature map pixel dimensions, and their values are all integer powers of 2;
[0071] The second parallel strategy is to parallelize the filters and channels in two dimensions, and P total =Pf 2 *Pc 2 ; Pf 2 、Pc 2 They are the degrees of parallelism of the second parallel strategy in the filter and channel dimensions, respectively, and their values are integer powers of 2.
[0072] (2) The control module generates a buffer read signal and sends it to the weight buffer and feature buffer. The two buffers transmit the weight data and feature data to the convolution engine;
[0073] (3) The control module generates a calculation signal and sends it to the convolution engine;
[0074] (4) The convolution engine performs convolution calculations based on the received weight data and feature data;
[0075] (5) After receiving the receipt signal from the convolution engine, the control module generates a pooling and activation module enable signal;
[0076] (6) The pooling and activation modules perform pooling and activation processing on the convolutional data in turn;
[0077] (7) The control module sends a write signal to the weight buffer and feature buffer to transmit the activated data to the off-chip DDR.
[0078] Example verification 1
[0079] This example aims to verify the feasibility of the accelerator. This example first uses the Pytorch framework to build a three-layer 3-D CNN network model. The structural parameters of the network model are shown in Table 1. After training the network model using the UCF101 dataset, it is deployed on the hardware accelerator designed in this solution for reasoning. The first two layers of the network use parallel strategy 1, setting Pt=4, Pv=16, and Pf=4; while the last layer of the network uses parallel strategy 2, setting Pf=16 and Pc=16. The results of the hardware accelerator accelerated reasoning are compared with the network model reasoning results on the CPU. The similarity is 1, which shows that the accelerator function is correct.
[0080] Table 1 Structural parameters of three-layer 3-D CNN
[0081] Width high depth aisle 64 64 16 3 32 32 8 8 16 16 4 16
[0082] Example verification 2
[0083] In this example, the current mainstream 3-D CNN model is deployed on the accelerator, the total parallelism is set to 2048, and the parallel strategy is generated using software tools. The computational efficiency and energy efficiency of each network are shown in Table 2.
[0084] Table 2 Computational efficiency and energy efficiency
[0085] Network Model Computational efficiency Energy efficiency (GOP / S / W) C3D 99.67％ 148.96 R3D-18 99.79％ 149.15 R3D-34 99.86％ 149.26 E3D 85.39％ 127.62 I3D 98.13％ 146.75 X3D 80.06％ 119.66 R(2+1)D 93.58％ 139.86
[0086] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.
Claims
1. A method for accelerating a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, characterized in that: According to the structural parameters of each convolutional layer of the 3D convolutional neural network, the first parallel strategy or the second parallel strategy is dynamically selected; The first parallel strategy is time depth, filter and feature map pixel three-dimensional parallelism, and P total =Pt1*Pf1*Pv1; Among them, P total Represents the total computational parallelism of the 3D convolutional neural network; Pt1, Pf1, and Pv1 are the parallelisms of the first parallel strategy in the time depth, filter, and feature map pixel dimensions, respectively, and their values are all integer powers of 2; The second parallel strategy is to parallelize the filters and channels in two dimensions, and P total =Pf2*Pc2; Pf2 and Pc2 are the parallelism of the second parallel strategy in the filter and channel dimensions, respectively, and their values are both integer powers of 2.
2. The acceleration method of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 1, characterized in that: If the size of a convolutional layer input feature map is larger than Pt1 and Pv1 set in the first parallel strategy and the number of channels is relatively smaller than Pc2 set in the second parallel strategy, the first parallel strategy is selected; If the size of the input feature map of a convolutional layer is smaller than Pt1 and Pv1 set in the first parallel strategy and the number of channels is greater than or equal to Pc2 set in the second parallel strategy, the second parallel strategy is selected.
3. The acceleration method of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 1, characterized in that: The first parallel strategy adopts a three-dimensional array structure composed of several MAC units for parallel computing, and each MAC unit is composed of a multiplier and an accumulator; wherein a single two-dimensional array constituting the three-dimensional array structure receives feature data from different time frames, and each two-dimensional array shares weight data of the same time frame; each column of the single two-dimensional array receives weight parameters from different Pf1 filter groups, and different rows in each column share the same weight data from the same filter and Pv1 feature data of the same row of feature data from the same time frame; The second parallel strategy adopts a two-dimensional array structure composed of several MAC units for parallel computing; wherein each column in the two-dimensional array structure receives weight parameters from Pf2 filter groups, and different rows in each column receive feature parameters and weight parameters from Pc2 input channels.
4. The acceleration method of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 3, characterized in that: An addition tree is configured for each column of each two-dimensional array; In the first parallel strategy, parallel calculation in the channel dimension is not involved, so the addition tree of each column in the two-dimensional array is disabled, and the MAC unit of each column calculates in parallel on the feature map pixel dimension to obtain Pv1 accumulation results; the three-dimensional array structure uses a total accumulator to perform time dimension accumulation operations on the output results of all two-dimensional arrays; In the second parallel strategy, each column of the two-dimensional array enables an adder tree to implement channel-dimensional addition operations on the output results of all MAC units, and finally outputs a summation result for each column.
5. The acceleration method of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 3, characterized in that: In the first parallel strategy, the weights are stored as follows: Ks of the same channel of different filter groups on the same time dimension 2 The weight data are stored at the same address; the feature data is stored in the following way: a row of input feature data is stored at each address, and data shifting is achieved through multi-level on-chip buffering to meet the data requirements of feature pixel dimension parallelism.
6. The acceleration method of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 3, characterized in that: In the second parallel strategy, the weights are stored in the same way as follows: Pc2 weight data with the same spatial position in Pf2 filters are stored in the same address; the feature data is stored in the same way as follows: Pc2 data are stored in each address to meet the data requirements of channel dimension parallelism.
7. An acceleration device for a 3D convolutional neural network based on multi-dimensional dynamic parallel computing, characterized in that: The acceleration device includes a weight buffer, a data buffer, a control module, a convolution engine, a pooling module and an activation function module; the implementation steps of the acceleration device include: The host computer dynamically selects the first parallel strategy or the second parallel strategy for the current convolution layer according to the structural parameters of the current convolution layer, and sends the selected parallel strategy to the control module; The control module sends a dynamic adjustment instruction to other hardware modules according to the received parallel strategy that needs to be executed currently, and the other hardware modules make corresponding adjustments according to the received adjustment instructions; the other hardware modules include: a weight buffer, a data buffer, a convolution engine, a pooling module and an activation function module; The control module generates a buffer read signal and sends it to the weight buffer and the feature buffer. The two buffers transmit the weight data and feature data to the convolution engine. The control module generates a calculation signal and sends it to the convolution engine; The convolution engine performs convolution calculations based on the received weight data and feature data; After receiving the receipt signal from the convolution engine, the control module generates a pooling and activation module enable signal; The pooling and activation modules perform pooling and activation processing on the convolutional data in turn; The control module sends a write signal to the weight buffer and feature buffer to transmit the activated data to the off-chip DDR; The first parallel strategy is time depth, filter and feature map pixel three-dimensional parallelism, and P total =Pt1*Pf1*Pv1; where P total is the total computational parallelism of the 3D convolutional neural network. Pt1, Pf1, and Pv1 are the parallelisms of the first parallel strategy in the time depth, filter, and feature map pixel dimensions, respectively. Their values are all integer powers of 2. The second parallel strategy is to parallelize the filters and channels in two dimensions, and P total =Pf2*Pc2; Pf2 and Pc2 are the parallelism of the first parallel strategy in the filter and channel dimensions, respectively, and their values are both integer powers of 2.
8. The acceleration device of the 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 7, characterized in that: If the size of a convolutional layer input feature map is larger than Pt1 and Pv1 set in the first parallel strategy and the number of channels is relatively smaller than Pc2 set in the second parallel strategy, the first parallel strategy is selected; If the size of the input feature map of a convolutional layer is smaller than Pt1 and Pv1 set in the first parallel strategy and the number of channels is greater than or equal to Pc2 set in the second parallel strategy, the second parallel strategy is selected.
9. The acceleration device of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 7, characterized in that: An addition tree is configured for each column of each two-dimensional array; In the first parallel strategy, parallel calculation in the channel dimension is not involved, so the addition tree of each column in the two-dimensional array is disabled, and the MAC unit of each column calculates in parallel on the feature map pixel dimension to obtain Pv1 accumulation results; the three-dimensional array structure uses a total accumulator to perform time dimension accumulation operations on the output results of all two-dimensional arrays; In the second parallel strategy, each column of the two-dimensional array enables an adder tree to implement channel-dimensional addition operations on the output results of all MAC units, and finally outputs a summation result for each column.
10. The acceleration device of a 3D convolutional neural network based on multi-dimensional dynamic parallel computing according to claim 7, characterized in that: In the first parallel strategy, the weights are stored as follows: Ks of the same channel of different filter groups on the same time dimension 2 The weight data is stored in the same address; the feature data is stored in the following way: a row of input feature data is stored in each address, and data shifting is achieved through multi-level on-chip buffering to meet the data requirements of feature pixel dimension parallelism; In the second parallel strategy, the weights are stored in the same way as follows: Pc2 data with the same spatial position in Pf2 filters are stored in the same address; the feature data is stored in the same way as follows: Pc2 data are stored in each address to meet the data requirements of channel dimension parallelism.