One-shot network architecture search method and hardware accelerator

Abstract

The application provides a one-time network architecture search method and a hardware accelerator. The method comprises the following steps: constructing a super network, the super network being composed of a plurality of layer structures; obtaining a one-time super network training algorithm; training the super network based on the one-time super network training algorithm to obtain a target deep neural network; and based on the target deep neural network, using a genetic algorithm in combination with an FPGA accelerator, using FPGA-aware network architecture search to obtain a network architecture with optimal accuracy and hardware performance, so as to solve the problem that although the classification accuracy of a deep neural network based on a multiplication operator is high when performing an image classification task, the energy consumption and area overhead are huge when the deep neural network is implemented in hardware, and it is difficult to apply the deep neural network to mobile devices or Internet of Things devices; and although a deep neural network based on a non-multiplication operator has superior hardware execution efficiency when implemented in hardware, the classification accuracy of the deep neural network is low when the deep neural network completes an image classification task.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a one-time network architecture search method and hardware accelerator. Background Technology

[0002] Deep Neural Networks (DNNs) have achieved great success in computer vision tasks, leading to an increased demand for DNN-based methods in practical applications.

[0003] However, the multiplication operation widely used in DNNs dominates their hardware consumption and largely limits the hardware efficiency that DNNs can achieve. This has prompted the development of multiplication-free DNNs, which use hardware-friendly operators such as addition and bit shifting (which require less power and area overhead) to replace multiplication.

[0004] While multiplication-free DNNs offer superior hardware efficiency, their task accuracy is typically lower than that of multiplication-based DNNs. Therefore, it is necessary to combine the advantages of both to construct hybrid DNNs with reduced multiplication counts, thus achieving a trade-off between performance and efficiency. Summary of the Invention

[0005] To address the issues of high energy consumption and area overhead in hardware implementation of traditional deep neural networks based on multiplication operators, and insufficient task accuracy of deep neural networks without multiplication, this application provides a one-time network architecture search method and hardware accelerator through the following aspects.

[0006] The first aspect of this application provides a one-time network architecture search method, including:

[0007] A super network is constructed, which consists of multiple layers. Each layer includes: a first convolutional layer, a candidate module group connected to the first convolutional layer, a pooling module connected to the candidate module group, a pooling layer connected to the pooling module, and a classifier connected to the pooling layer. The last two layers of the super network are fixed layers, while the remaining layers are searched from the search space of the candidate module group. The search space of the candidate module group includes convolutional layers and non-convolutional layers, where the non-convolutional layers include additive layers and shift layers.

[0008] A one-time super network training algorithm is obtained, and the super network is trained based on the one-time super network training algorithm to obtain the target deep neural network;

[0009] Based on the target deep neural network, a network architecture with optimal accuracy and hardware performance is obtained by using a genetic algorithm combined with an FPGA accelerator and FPGA-aware network architecture search.

[0010] In some embodiments, the search space of the candidate module group comprises seven stages, each stage including at least one candidate module. Each candidate module includes two pointwise convolutional layers and one channelwise convolutional layer, represented by five hyperparameters. The hyperparameters of the search space of the candidate module group are:

[0011] The number of output channels for the candidate module groups includes: Phase 1 {16, 24}, Phase 2 {24, 32}, Phase 3 {32, 40}, Phase 4 {64, 72}, Phase 5 {112, 120, 128}, Phase 6 {192, 200, 208, 216}, and Phase 7 {216, 224}.

[0012] The output channel expansion ratios of the candidate module groups include: Phase 1 {1}, Phase 2 {4, 5, 6}, Phase 3 {4, 5, 6}, Phase 4 {4, 5, 6}, Phase 5 {4, 5, 6}, Phase 6 {6}, and Phase 7 {6}.

[0013] The convolutional kernel sizes for the candidate module groups include: Phase 1 {3, 5}, Phase 2 {3, 5}, Phase 3 {3, 5}, Phase 4 {3, 5}, Phase 5 {3, 5}, Phase 6 {3, 5}, and Phase 7 {3, 5}.

[0014] The number of modules in the candidate module groups includes: Phase 1 {1, 2}, Phase 2 {3, 4, 5}, Phase 3 {3, 4, 5, 6}, Phase 4 {3, 4, 5, 6}, Phase 5 {3, 4, 5, 6, 7, 8}, Phase 6 {3, 4, 5, 6, 7, 8}, and Phase 7 {1, 2}.

[0015] Layer types include: convolutional layers, additive layers, and shift layers;

[0016] In this configuration, the kernel size and layer type of the first convolutional layer and the pooling module are set to 1 and 1 respectively, and their output channel counts are searchable. The hyperparameter of the output channel count search space is:

[0017] The number of output channels of the first convolutional layer includes: {16, 24};

[0018] The number of output channels of the pooling module includes: {1792, 1984}.

[0019] In some embodiments, training the super network based on the one-time super network training algorithm to obtain the target deep neural network includes:

[0020] Based on the one-time super network training algorithm, the super network is simplified to obtain a simplified super network;

[0021] The simplified super network is trained to obtain the target deep neural network.

[0022] In some embodiments, training the simplified supernetwork includes:

[0023] The simplified supernetwork is sampled based on the sandwich rule to obtain a sampling model; the sampling model includes: the smallest subnetwork, the largest subnetwork, and randomly sampled subnetworks;

[0024] Based on the sampling model, the weights of the sampling model are optimized using an optimization model to obtain the optimized sampling model;

[0025] Based on the optimized sampling model, knowledge distillation training is performed on the optimized sampling model using α divergence to obtain the target deep neural network.

[0026] In some embodiments, the FPGA-aware network architecture search includes:

[0027] Obtain the initial network architecture set based on FPGA;

[0028] Based on the initial set of network architectures, crossover and mutation instructions are executed on the initial set of network architectures to obtain an extended set of network architectures.

[0029] Update the expanded network architecture set to obtain the target network architecture set, and use the target network architecture set for searching.

[0030] A second aspect of this application provides a hardware accelerator for accelerating a target deep neural network obtained by the one-time network architecture search method described in any one of the first aspects above, comprising: an acceleration kernel module; the acceleration kernel module includes a convolution acceleration kernel, a shift acceleration kernel, and an addition acceleration kernel;

[0031] The convolution acceleration kernel is provided with a multiply-accumulate unit for processing convolution, the shift acceleration kernel is provided with a shift-accumulate unit for supporting the shift layer, and the addition acceleration kernel is provided with an addition-accumulate unit for supporting the addition layer.

[0032] The hardware accelerator also includes: a global buffer, a controller, and an off-chip storage module;

[0033] The off-chip storage module is used to store pre-trained model parameters, input data, and output data.

[0034] The global buffer is used to read the input data and cache intermediate computation data within the chip.

[0035] The convolution acceleration kernel is used to accelerate the convolutional layer in a deep neural network and includes multiple multiply-accumulate units, which form a systolic array.

[0036] The controller is used to coordinate and control the collaboration and logic between the various modules;

[0037] The shift acceleration kernel is used to accelerate the shift layer in a deep neural network, and includes multiple shift accumulation units, which constitute a pulsating array.

[0038] The addition acceleration kernel is used to accelerate the addition layer in a deep neural network and includes multiple addition accumulation units, which form a pulsating array.

[0039] In some embodiments, the multiply-accumulate unit is provided with a digital signal processor, which has a built-in 27-bit × 18-bit multiplier for performing a multiplication operation of (A + D) × B, where A is 30 bits, D is 27 bits, and B is 18 bits.

[0040] In some embodiments, the shift accumulation unit is provided with a first lookup table, which is used to implement shift operations in the shift layer of the hybrid model.

[0041] In some embodiments, the addition and accumulation unit is provided with a second lookup table, which is used to utilize lookup table resources on the FPGA.

[0042] In some embodiments, the hardware accelerator further includes:

[0043] The allocation unit is used to allocate the digital signal processor and lookup table resources on the FPGA to the corresponding acceleration core modules according to the number of operations of different layer types.

[0044] This application provides a one-time network architecture search method and hardware accelerator. The method includes: constructing a super network, the super network being composed of multiple layer structures, each layer structure including: a first convolutional layer, a candidate module group connected to the first convolutional layer, a pooling module connected to the candidate module group, a pooling layer connected to the pooling module, and a classifier connected to the pooling layer; the last two layers of the layer structure are fixed layers, and the remaining layer structures need to be searched from the search space of the candidate module group; the search space of the candidate module group includes: convolutional layers and non-convolutional layers, the non-convolutional layers including: addition layers and shift layers; obtaining a one-time super network training algorithm, based on the one-time network architecture search method and hardware accelerator. The one-time super network training algorithm trains the super network to obtain a target deep neural network. Based on the target deep neural network, a genetic algorithm combined with an FPGA accelerator is used to perform FPGA-aware network architecture search to obtain a network architecture with optimal accuracy and hardware performance. This allows the target deep neural network obtained by the one-time network architecture search method provided in this application to be a multiplication-reduced hybrid deep neural network. It has the advantages of high accuracy of convolutional layers based on multiplication operators and high hardware efficiency of addition or shift layers, achieving a balance between hardware efficiency and task accuracy. Furthermore, thanks to the one-time super network training algorithm, retraining of the super network in different scenarios is avoided. Attached Figure Description

[0045] Figure 1 A schematic diagram illustrating the workflow of a one-time network architecture search method provided in this application embodiment;

[0046] Figure 2 A schematic diagram of the structure of a supernetwork constructed in a deep neural network architecture search method provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of the weight distribution in convolutional layers, shift layers constructed with DeepShift-PS, shift layers constructed with DeepShift-Q, and additive layers;

[0048] Figure 4 A schematic diagram of the structure of a hardware accelerator for a deep neural network provided in an embodiment of this application;

[0049] Figure 5 A schematic diagram illustrating the PE scheduling strategy process of a hardware accelerator for a deep neural network, provided in an embodiment of this application;

[0050] Figure 6 (a) is a schematic diagram of a DSP (DSP48E2) (Digital Signal Processor), and (b) is a schematic diagram of the packaging strategy of INT8 DSP (Digital Signal Processor) based on Xilinx FPGA. Detailed Implementation

[0051] The deep neural network provided in this application can be used in image classification tasks in computer vision. Currently, while deep neural networks based on multiplication operators achieve high classification accuracy in image classification tasks, their hardware implementation incurs huge power consumption and area overhead, making them difficult to apply to mobile devices or IoT devices. Conversely, deep neural networks based on non-multiplication operators offer superior hardware execution efficiency, but their classification accuracy is relatively low. To address these issues, this application provides a one-time network architecture search method and hardware accelerator through the following embodiments.

[0052] Depend on Figure 1 As can be seen, the first aspect of this application provides a one-time network architecture search method, which searches for a network architecture with optimal accuracy and hardware performance, namely a multiplication-reduced hybrid deep neural network. The method includes: constructing a super network, the super network including a first convolutional layer, a candidate module group connected to the first convolutional layer, a pooling module connected to the candidate module group, a pooling layer connected to the pooling module, and a classifier connected to the pooling layer; the last two layers of the super network are fixed layers, and the remaining layers need to be searched from a predefined search space. The search space of the candidate module group includes convolutional and non-convolutional layers, where the non-convolutional layers are additive layers and / or shift layers. A customized one-time super network training algorithm comprises two key stages: 1) Simplifying the super network: using a differentiable neural network architecture search algorithm to search for the optimal heterogeneous pattern for the super network, i.e., determining the layer type of each candidate module and the number of modules in each stage, thus obtaining a simplified super network whose weights can be directly shared among sub-networks; 2) Training the simplified super network: using a one-time super network training algorithm to train the simplified super network. The super network obtained after the one-time super network training algorithm can meet the deployment requirements of various different scenarios, thus avoiding the expensive retraining overhead; 3) Using FPGA-aware network architecture search: utilizing a genetic algorithm combined with direct feedback from the FPGA accelerator to identify the network architecture with optimal accuracy and hardware performance.

[0053] In some embodiments, the target deep neural network can be used to perform computer image classification tasks, taking an image to be classified as input and obtaining the corresponding classification result.

[0054] Depend on Figure 2It is known that the last two layers of the supernetwork are fixed, and the remaining layers need to be searched from a predefined search space. The candidate module group comprises multiple candidate modules connected sequentially. Each candidate module group contains 7 stages, and each stage contains several candidate modules. Each candidate module includes two pointwise convolutional layers (PW) and one channelwise convolutional layer (DW). The first PW is followed by a batch normalization (BN) layer and an activation layer. The channelwise convolutional layer is followed by a normalization layer and an activation layer. For example, the activation layer uses the swish activation function. The second PW is followed by a BN layer, and the output result is concatenated with the residual of the input data of the current candidate module to obtain the output data of the current candidate module.

[0055] In some embodiments, the first convolutional layer is a 1×1 convolutional layer, and the pooling module is a 1×1 convolutional layer. In other embodiments, convolutional layers with other sizes of output channels may be used.

[0056] from Figure 2 As can be seen, each candidate module is represented by five hyperparameters: the number of output channels C, the channel dilation ratio E of the candidate module, the kernel size K, the layer type T, and the number of modules in each stage N. The hyperparameters of the candidate modules are obtained from a preset search space.

[0057] In some embodiments, the search space of the candidate module group is based on AlphaNet.

[0058] In some embodiments, the calculation process of the shift layer of the hybrid shift neural network can be performed using the formula Y = ΣX. T *W shift The diagram shows a shift layer. X is the input of the shift layer, Y is the output of the shift layer, Σ() is the summation operation, and W... shift These are the weights of the shift layer. In one possible implementation, the weights W are determined using the DeepShift-Q method. shift That is, by using the original weights w in the convolutional layer * The shift layer is constructed by quantizing to powers of 2, as expressed by the following formula:

[0059]

[0060] In this context, sign() represents the sign operation, and round() represents the rounding operation.

[0061] Depend on Figure 3This diagram illustrates the weight distribution in shift layers determined using the DeepShift-PS method and the DeepShift-Q method. From... Figure 3 As can be seen, compared with other methods of determining the shift layer weights that directly optimize the sign flip bit s and shift bit p (such as DeepShift-PS), the DeepShift-Q method can match the small weights in the convolutional layer, which is more friendly to the training process of deep neural networks.

[0062] In some embodiments, the hybrid addition search space employs the addition layer proposed by AdderNet. The addition layer utilizes computationally efficient addition operations and the L1 norm to determine the correlation between lateral weights and feature maps, expressed by the following formula: Y = Σ - |XW adder |, where W adder These are the weights of the additive layer, X is the input to the additive layer, Y is the output of the additive layer, and Σ() is the summation operation. See also Figure 2 In the above embodiment, (d) represents the weight distribution of the additive layer.

[0063] In some instances, the hyperparameters in the search space are configured as follows:

[0064] C: Includes: Phase 1 {16, 24}, Phase 2 {24, 32}, Phase 3 {32, 40}, Phase 4 {64, 72}, Phase 5 {112, 120, 128}, Phase 6 {192, 200, 208, 216}, Phase 7 {216, 224}.

[0065] E: Includes: Phase 1 {1}, Phase 2 {4, 5, 6}, Phase 3 {4, 5, 6}, Phase 4 {4, 5, 6}, Phase 5 {4, 5, 6}, Phase 6 {6}, Phase 7 {6};

[0066] K: Includes: Phase 1 {3, 5}, Phase 2 {3, 5}, Phase 3 {3, 5}, Phase 4 {3, 5}, Phase 5 {3, 5}, Phase 6 {3, 5}, Phase 7 {3, 5};

[0067] N: includes: Phase 1 {1, 2}, Phase 2 {3, 4, 5}, Phase 3 {3, 4, 5, 6}, Phase 4 {3, 4, 5, 6}, Phase 5 {3, 4, 5, 6, 7, 8}, Phase 6 {3, 4, 5, 6, 7, 8}, Phase 7 {1, 2};

[0068] T includes: convolutional layers, additive layers, and shift layers.

[0069] In this configuration, the kernel size and layer type of the first convolutional layer and the pooling module are set to 1 and convolutional layer, respectively, but their output channel count can be searched. The hyperparameter configuration of the output channel count search space is as follows:

[0070] The number of output channels of the first convolutional layer, C: {16, 24};

[0071] The number of output channels of the pooling module, C: {1792, 1984}.

[0072] For example, the supernetwork includes 37 candidate modules, and ultimately, the search space of the candidate module group contains 2.4 * 10^6 modules. 14 A potential subnetwork.

[0073] Depend on Figure 3 It is known that the weights of the additive layers provided in this application typically follow a Laplace distribution, while the weights of the convolutional layers typically follow a Gaussian distribution. To prevent gradient conflicts between different heterogeneous modes and to avoid retraining and fine-tuning of sub-networks, embodiments of this application provide a customized one-time super-network training algorithm. The customized one-time super-network training algorithm includes: simplifying the super-network based on the one-time super-network training algorithm to obtain a simplified super-network; and training the simplified super-network based on the one-time super-network training algorithm to obtain a target deep neural network.

[0074] Depend on Figure 4 As can be seen, in the super network simplification stage, this application uses the Differentiable Neural Architecture Search (DNAS) algorithm to search for the optimal heterogeneous pattern for the super network, including searching for a suitable layer type T and the number of modules N in each stage for each layer in the super network, thereby obtaining a simplified super network whose weights can be directly shared among sub-networks.

[0075] The specific training process is as follows:

[0076]

[0077] in and These are the validation loss function, training loss function, and hardware loss function, respectively. θ represents the network weights of the supernetwork, and α represents the network structure coefficients.

[0078] In some embodiments, FLOPs (floating point operations) are used as the hardware loss function L. hwThe surrogate hardware metrics are used. For shift and adder layers where FLOPs cannot be measured, the shift and / or adder layers are first treated as convolutional layers to obtain FLOPs, and then scaled according to the hardware power consumption ratio of the shift and / or adder layers relative to the convolutional layers. In some embodiments, unit power consumption, latency, parameter count, etc., can also be used as hardware surrogate metrics.

[0079] In some instances, the learning rate for the structural coefficients is set to a relatively small value, such as 0.0003, and updated using the Adam optimizer, while the learning rate for the network weights is set to 0.3 and updated using the SGD optimizer.

[0080] The process of training the simplified supernetwork to obtain the target deep neural network includes: sampling the simplified supernetwork based on the sandwich rule to obtain a sampling model; the sampling model includes: the smallest subnetwork, the largest subnetwork, and randomly sampled subnetworks; optimizing the weights of the sampling model using an optimization model to obtain an optimized sampling model; and training the optimized sampling model using α-divergence through knowledge distillation to obtain the target deep neural network. The training strategy for the simplified supernetwork is crucial to its performance. Specifically, this application utilizes two training strategies to aid in the training of the simplified supernetwork: "network architecture sampling based on the sandwich rule" and "knowledge distillation based on α-divergence." Network architecture sampling based on the sandwich rule: In each training step, the following model is sampled: (i) the smallest subnetwork α s (i) The network with the minimum channel dimension C, expansion ratio E, and kernel size K; (ii) The largest subnetwork α b That is, the simplified supernetwork obtained in the step of simplifying the supernetwork; (iii) M randomly sampled subnetworks This involves randomly sampling networks C, E, and K. The weights of these models are then optimized using the following function, resulting in the optimized model:

[0081]

[0082] Knowledge Distillation Based on Alpha Divergence: To facilitate knowledge distillation from teacher to student models, KL (Kullback-Leibler) divergence is widely used to measure the difference between their output probabilities. However, KL divergence can lead to student models overestimating or underestimating the output probability distribution of the teacher model. To address this issue, this patent applies alpha divergence to encourage student models to accurately estimate the output probability distribution of the teacher model. Alpha divergence can be expressed as: D α+，α- (p||q)=max{D α+ (p||q), D α-(p||q)}, where:

[0083]

[0084] and These are two discrete distributions on m categories, where α+ and α- respectively prevent the student model from overestimating or underestimating the probability distribution of the teacher model.

[0085] Therefore, in formula (1) It can be represented by the following formula (3):

[0086]

[0087] Where A∈{α s α r}

[0088] The first aspect of this application provides a one-time network architecture search method, comprising: constructing a super network, the super network including a first convolutional layer, a candidate module group connected to the first convolutional layer, a pooling module connected to the candidate module group, a pooling layer connected to the pooling module, and a classifier connected to the pooling layer; the last two layers of the super network are fixed layers, and the remaining layers need to be searched from a predefined search space. The search space for the candidate module group includes convolutional and non-convolutional layers, where the non-convolutional layers are additive layers and / or shift layers. A customized one-time super network training algorithm comprises two key stages: simplifying the super network by using a differentiable neural network architecture search algorithm to find the optimal heterogeneous pattern for the super network, i.e., determining the layer type of each candidate module and the number of modules in each stage, thus obtaining a simplified super network whose weights can be directly shared among sub-networks; and training the simplified super network using the one-time super network training algorithm. The super network obtained after the one-time super network training algorithm can meet the deployment requirements of various scenarios, thus avoiding the expensive retraining overhead. Therefore, the final target deep neural network obtained is a multiplication-reduced hybrid deep neural network. This target deep neural network can combine the high accuracy of convolutional layers based on multiplication operators with the high hardware efficiency of additive or shift layers, achieving a balance between hardware efficiency and task accuracy.

[0089] In some instances, the FPGA-aware network architecture search includes: obtaining an initial network architecture set based on the FPGA; performing crossover and mutation instructions on the initial network architecture set to obtain an expanded network architecture set; updating the expanded network architecture set to obtain a target network architecture set; and using the target network architecture set for searching. Genetic algorithms are used in sub-network searches after the supernetwork training is complete. The network identification accuracy and hardware performance are comprehensively considered during the search process to identify the network architecture with optimal accuracy and hardware performance, i.e., FPGA-aware network architecture search. Specifically, the algorithm includes the following steps in its implementation:

[0090] (i) Initializing the network architecture set: An initial network architecture set is formed by randomly sampling subnetworks from the supernetwork that meet given hardware constraints (e.g., latency). The hardware metrics are measured by a hardware simulator based on the accelerator of this patent to provide fast and accurate estimates.

[0091] (ii) Crossover and mutation: The set of network architectures is expanded by performing crossover (i.e., randomly selecting two subnetworks from the set of network architectures and randomly replacing some values ​​in the model configuration of one subnetwork with corresponding values ​​in the model configuration of the other subnetwork to generate a new subnetwork) and mutation (i.e., randomly sampling subnetworks and randomly changing some values ​​in their model configuration to other options to generate a new subnetwork).

[0092] (iii) Update the network architecture set: Sort the candidate network architectures in the set according to accuracy, and retain only the top k sub-networks that satisfy the given hardware constraints. Iterate steps (ii) and (iii) until the pre-set maximum number of iterations is reached.

[0093] Thanks to the custom one-time supernetwork training strategy proposed in this application, there is no need to retrain / fine-tune the sampled subnetworks at this stage.

[0094] To accelerate the target deep neural network obtained by the one-time network architecture search method provided in the aforementioned embodiments, and to fully utilize heterogeneous hardware resources on the FPGA (such as DSP (Digital Signal Processor) and LUT (Lookup Table)) to accelerate heterogeneous layers (such as convolutional layers, shift layers, and additive layers) in the searched hybrid network to improve hardware utilization and throughput, this patent designs a dedicated multi-core hardware accelerator. This application also provides an embodiment of a hardware accelerator for deep neural networks. The hardware accelerator includes a global buffer, a controller, an off-chip storage module, and three custom-designed acceleration cores.

[0095] Among them, DRAM (Dynamic Random Access Memory) is an off-chip storage module. GB (Global Buffer) is the global storage module, MAC (Multiply And Accumulate) is the basic unit of convolution processing, SAC (Shift And Accumulate) is the basic unit of shift processing, and AAC (Add And Accumulate) is the basic unit of addition processing.

[0096] Depend on Figure 4 It is known that the accelerator is equipped with three custom-designed acceleration cores: a convolution acceleration core, a shift acceleration core, and an addition acceleration core. It is configured with dedicated processing units (PEs) to accelerate heterogeneous layers in the hybrid network. Specifically, the convolution acceleration core integrates a MAC unit for processing convolutions, the shift acceleration core is equipped with a SAC unit to support shift layers, and the addition acceleration core integrates an AAC unit to support addition layers.

[0097] DRAM is used to store pre-trained model parameters, as well as input and output data; GB is used to read input data and cache intermediate computation data on-chip; GB caches computation inputs and outputs on-chip to reduce the high cost of off-chip data access from DRAM. The controller manages the data reading of each acceleration core and the coordination between modules. Convolutional acceleration kernels / shift acceleration kernels / addition acceleration kernels are used to accelerate the convolutional layers / shift layers / addition layers of deep neural networks, respectively.

[0098] In some examples, the PE unit design for MAC, SAC, and AAC is as follows:

[0099] DSP implementation of MAC: such as Figure 6 As shown in (a), each DSP (DSP48E2) has a built-in 27-bit × 18-bit multiplier, capable of performing a (A+D) × B multiplication operation, where A is 30 bits, D is 27 bits, and B is 18 bits. To fully utilize the DSP's built-in high-order multiplier to support low-order multiplication (8 bits × 8 bits) in the MAC unit, this patent employs a Xilinx FPGA-based INT8 DSP packing strategy, packaging multiple low-order multiplications within a single DSP. Specifically, as... Figure 6 As shown in (b), by inputting an 8-bit weight W through port A and packing two 8-bit inputs X1 and X2 through port B, the DSP's built-in multiplier can be configured to support two 8-bit × 8-bit multiplications: W × {X1, X2} = {O1, O2}, thereby saving DSP resource consumption.

[0100] SAC LUT Implementation: To implement shift operations in the hybrid model shift layer, this patent utilizes LUT resources on the FPGA to implement the SAC cell. LUT-based shift operations consume approximately 80% less power than multiplication operations based on DSP packing strategies, highlighting the superiority of the shift layer in improving hardware efficiency.

[0101] AAC LUT Implementation: To make full use of the abundant LUT resources on the FPGA, this patent utilizes Vivado's automatic optimization capabilities and the custom IP core VIO (Virtual Input / Output) to implement the AAC unit in the addition acceleration core.

[0102] In some instances, such as Figure 5 As shown, a five-layer hybrid network is used as an example to illustrate the scheduling strategy of the deep neural network hardware accelerator provided in the above embodiments. In each processing cycle, the acceleration kernels (convolution acceleration kernel, shift acceleration kernel, and addition acceleration kernel) in the multi-core accelerator process the convolutional layer (convolution 1 and convolution 4), the shift layer (shift 2), and the addition layer (addition 3) in sequence. The data processed in each cycle is independent, and these data come from five separate input images and are the output of the previous cycle. For example, the result of convolution 1 processed by the convolution acceleration kernel in cycle i will be used as the input of shift 2 and calculated by the shift acceleration kernel in the next cycle i+1. When the convolution acceleration kernel (whose processing time is the longest in this example) finishes, a cycle is completed. Therefore, in order to reduce the processing bottleneck and improve the overall throughput, this patent uses formula (4) to allocate the DSP and LUT resources on the FPGA to the corresponding acceleration kernels:

[0103]

[0104] in, These are the number of PEs allocated to MAC, SAC, and AAC, respectively. Conv O Shift and O Adder These represent the number of operations for convolution, shift, and addition, respectively. Meanwhile, the resource consumption N of the DSP and LUT is... DSP A LUT To satisfy the given hardware resource constraint T.

[0105] Since the deep neural network provided in the foregoing embodiments includes shift and addition layers with high area and energy efficiency, the corresponding deep neural network hardware accelerator can partially replace the more expensive multiply-accumulate units with lower-cost shift processing modules and / or addition processing modules under the same area budget, thereby improving parallelism and reducing the overall latency and energy cost of the hardware accelerator.

[0106] A second aspect of this application provides a hardware accelerator that fully utilizes heterogeneous hardware resources on an FPGA (such as a DSP (Digital Signal Processor) and a LUT (Lookup Table)) to improve hardware utilization and throughput. The hardware accelerator includes an on-chip global buffer, a controller, an off-chip storage module, and three custom-designed acceleration cores: a convolution acceleration core, a shift acceleration core, and an addition acceleration core. Each acceleration core is configured with a dedicated processing unit (PE) to accelerate heterogeneous layers in a hybrid network. Specifically, the convolution acceleration core integrates a MAC (Multiply And Accumulate) unit for processing convolutions, the shift acceleration core is equipped with a SAC (Shift And Accumulate) unit to support shift layers, and the addition acceleration core integrates an AACss (Add And Accumulate) unit to support addition layers. Accumulate (addition and accumulation) unit; wherein, the off-chip storage module is used to store pre-trained model parameters, input data, and output data; the global buffer is used to read the input data and cache on-chip intermediate computation data; the convolution acceleration kernel is used to accelerate the convolutional layer in the deep neural network, including multiple multiply-accumulate units, which form a systolic array; the controller is used to coordinate and control the collaboration and logic between the various modules; the shift acceleration kernel is used to accelerate the shift layer in the deep neural network, including multiple shift-accumulate units, which form a systolic array; the addition acceleration kernel is used to accelerate the addition layer in the deep neural network, including multiple addition-accumulate units, which form a systolic array. In some embodiments, the hardware accelerator also integrates an automapper in the algorithm computation to automatically search for the optimal data stream, further improving hardware efficiency.

[0107] The hardware accelerator provided in this application can be implemented using an FPGA (Field Programmable Gate Array).

Claims

1. A one-time network architecture search method, characterized in that, include: A super network is constructed, which consists of multiple layers. Each layer includes: a first convolutional layer, a candidate module group connected to the first convolutional layer, a pooling module connected to the candidate module group, a pooling layer connected to the pooling module, and a classifier connected to the pooling layer. The last two layers of the super network are fixed layers, and the remaining layers are obtained from the search space of the candidate module group. The search space of the candidate module group includes convolutional layers and non-convolutional layers, and the non-convolutional layers include additive layers and shift layers. A one-time super network training algorithm is obtained, and the super network is trained based on the one-time super network training algorithm to obtain the target deep neural network; Based on the target deep neural network, a network architecture with optimal accuracy and hardware performance is obtained by using a genetic algorithm combined with an FPGA accelerator and FPGA-aware network architecture search. The step of training the super network based on the one-time super network training algorithm to obtain the target deep neural network includes: Based on the one-time super network training algorithm, the super network is simplified to obtain a simplified super network; The simplified supernetwork is trained to obtain the target deep neural network; Training the simplified supernetwork yields the target deep neural network, which includes: The simplified supernetwork is sampled based on the sandwich rule to obtain a sampling model; the sampling model includes: the smallest subnetwork, the largest subnetwork, and randomly sampled subnetworks; Based on the sampling model, the weights of the sampling model are optimized using an optimization model to obtain the optimized sampling model; Based on the optimized sampling model, knowledge distillation training is performed on the optimized sampling model using α divergence to obtain the target deep neural network; In the simplification phase of the super network, a differentiable neural network architecture search algorithm is used to search for the optimal heterogeneous pattern for the super network, including searching for the appropriate layer type T and the number of modules N in each stage for each layer in the super network, resulting in a simplified super network whose weights can be directly shared among sub-networks. The FPGA-aware network architecture search includes: Initialize the network architecture set: Select sub-networks that meet the given hardware constraint latency by randomly sampling from the supernetwork to form the initial network architecture set. The hardware constraint is latency. The hardware metrics are measured by an accelerator-based hardware simulator. The network architecture set is expanded by performing crossover and mutation multiple times; crossover includes randomly selecting two subnetworks from the network architecture set and randomly replacing the values ​​in the model configuration of one subnetwork with the corresponding values ​​in the model configuration of the other subnetwork to generate a new subnetwork; mutation includes randomly sampling subnetworks and randomly changing the values ​​in their model configurations to other optional values ​​to generate a new subnetwork to expand the network architecture set. Update the network architecture set: Sort the candidate network architectures in the set according to accuracy and retain only the top k sub-networks that meet the given hardware constraints; iterate the crossover and mutation steps and update the network architecture set until the preset maximum number of iterations is reached.

2. The one-time network architecture search method according to claim 1, characterized in that, The search space of the candidate module group consists of seven stages, each stage including at least one candidate module. Each candidate module includes two pointwise convolutional layers and one channelwise convolutional layer, represented by five hyperparameters. The hyperparameters of the search space of the candidate module group are: The number of output channels for the candidate module groups includes: Phase 1 {16, 24}, Phase 2 {24, 32}, Phase 3 {32, 40}, Phase 4 {64, 72}, Phase 5 {112, 120, 128}, Phase 6 {192, 200, 208, 216}, and Phase 7 {216, 224}. The output channel expansion ratios of the candidate module groups include: Phase 1 {1}, Phase 2 {4, 5, 6}, Phase 3 {4, 5, 6}, Phase 4 {4, 5, 6}, Phase 5 {4, 5, 6}, Phase 6 {6}, and Phase 7 {6}. The convolutional kernel sizes for the candidate module groups include: Phase 1 {3, 5}, Phase 2 {3, 5}, Phase 3 {3, 5}, Phase 4 {3, 5}, Phase 5 {3, 5}, Phase 6 {3, 5}, and Phase 7 {3, 5}. The number of modules in the candidate module groups includes: Phase 1 {1, 2}, Phase 2 {3, 4, 5}, Phase 3 {3, 4, 5, 6}, Phase 4 {3, 4, 5, 6}, Phase 5 {3, 4, 5, 6, 7, 8}, Phase 6 {3, 4, 5, 6, 7, 8}, and Phase 7 {1, 2}. Layer types include: convolutional layers, additive layers, and shift layers; In this configuration, the kernel size and layer type of the first convolutional layer and the pooling module are set to 1 and 1 respectively, and their output channel counts are searchable. The hyperparameter of the output channel count search space is: The number of output channels of the first convolutional layer includes: {16, 24}; The number of output channels of the pooling module includes: {1792, 1984}.

3. A hardware accelerator, said hardware accelerator being used to accelerate a target deep neural network obtained by the one-time network architecture search method according to any one of claims 1 to 2, characterized in that, include: Acceleration kernel module; the acceleration kernel module includes a convolution acceleration kernel, a shift acceleration kernel, and an addition acceleration kernel; The convolution acceleration kernel is provided with a multiply-accumulate unit for processing convolution, the shift acceleration kernel is provided with a shift-accumulate unit for supporting the shift layer, and the addition acceleration kernel is provided with an addition-accumulate unit for supporting the addition layer. The hardware accelerator also includes: a global buffer, a controller, and an off-chip storage module; The off-chip storage module is used to store pre-trained model parameters, input data, and output data. The global buffer is used to read the input data and cache intermediate computation data within the chip. The convolution acceleration kernel is used to accelerate the convolutional layer in a deep neural network and includes multiple multiply-accumulate units, which form a systolic array. The controller is used to coordinate and control the collaboration and logic between the various modules; The shift acceleration kernel is used to accelerate the shift layer in a deep neural network, and includes multiple shift accumulation units, which constitute a pulsating array. The addition acceleration kernel is used to accelerate the addition layer in a deep neural network and includes multiple addition accumulation units, which form a pulsating array.

4. The hardware accelerator according to claim 3, characterized in that, The multiply-accumulate unit is equipped with a digital signal processor, which has a built-in 27-bit × 18-bit multiplier for performing the multiplication operation of (A + D) × B, where A is 30 bits, D is 27 bits, and B is 18 bits.

5. The hardware accelerator according to claim 3, characterized in that, The shift accumulation unit is provided with a first lookup table, which is used to implement shift operations in the shift layer of the hybrid model.

6. The hardware accelerator according to claim 3, characterized in that, The addition and accumulation unit is equipped with a second lookup table, which is used to utilize the lookup table resources on the FPGA.

7. The hardware accelerator according to claim 3, characterized in that, Also includes: The allocation unit is used to allocate the digital signal processor and lookup table resources on the FPGA to the corresponding acceleration core modules according to the number of operations of different layer types.

Images