Architecture search methods and hardware accelerators for deep neural networks

Abstract

This application provides a method for architecture search of deep neural networks and a hardware accelerator. The architecture search method includes: constructing a supernetwork, which includes a first fixed layer, candidate module groups, a second fixed layer, a pooling layer, and a classifier connected in sequence; the search space of the candidate modules includes convolutional layers and non-convolutional layers, where the non-convolutional layers are additive layers and / or shift layers; pre-training the supernetwork to update its weight parameters; and using a differentiable search algorithm to search from the pre-trained supernetwork to obtain a target deep neural network. The search space of the candidate module groups includes convolutional layers, additive layers, and / or shift layers. Thus, the obtained target deep neural network is a multiplication-reduced hybrid deep neural network, which has the advantages of high accuracy of convolutional layers based on multiplication operators and high hardware efficiency of additive or shift layers, achieving a balance between hardware efficiency and task accuracy.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a method for searching the architecture of a deep neural network and a hardware accelerator. Background Technology

[0002] Deep Neural Networks (DNNs) are widely used in computer vision due to their powerful model representation capabilities. Traditionally, deep neural networks include multiplication-based convolutional neural networks. However, multiplication operators consume a lot of energy and have a large area overhead when implemented in hardware, which seriously hinders the application of DNNs in embedded devices with limited power consumption and area.

[0003] To improve hardware efficiency, existing research has used addition and shift operators, which have lower power consumption and area overhead, to replace multiplication operators and implement multiplication-free deep neural networks. However, the task accuracy of these multiplication-free deep neural networks is far inferior to that of multiplication-based deep neural networks.

[0004] Therefore, there is an urgent need for a deep neural network that can balance hardware efficiency and task accuracy. Summary of the Invention

[0005] To address the issues of high energy consumption and area overhead in hardware implementation of traditional deep neural networks based on multiplication operators, and insufficient task accuracy of deep neural networks without multiplication, this application provides a method for deep neural network architecture search and a hardware accelerator through the following aspects.

[0006] The first aspect of this application provides a method for architecture search of a deep neural network, comprising: constructing a super network, the super network including a first fixed layer, a candidate module group connected to the first fixed layer, a second fixed layer connected to the candidate module group, a pooling layer connected to the second fixed layer, and a classifier connected to the pooling layer; wherein, the candidate module group includes a plurality of candidate modules connected in sequence; the candidate module includes two pointwise operation layers, one layerwise operation layer, and corresponding batch normalization layers and activation layers; the search space of the candidate module group includes convolutional layers and non-convolutional layers, the non-convolutional layers being additive layers and / or shift layers;

[0007] Pre-train the super network to update the weight parameters of the super network, and obtain the pre-trained super network;

[0008] The target deep neural network is obtained by searching the pre-trained supernetwork using a differentiable search algorithm.

[0009] Optionally, the hyperparameters of the search space for the candidate module group are configured as follows: (channel expansion ratio of the candidate module, kernel size of the candidate module) including: (1,3), (3,3), (6,3), (1,5), (3,5), (6,5); the layer types include: convolutional layer and shift layer.

[0010] Optionally, the hyperparameters of the search space for the candidate module group are configured as follows: (channel expansion ratio of the candidate module, kernel size of the candidate module) including (1,3), (3,3), (6,3), (1,5), (3,5), (6,5); the layer types include: convolutional layer and additive layer.

[0011] Optionally, the hyperparameters of the search space for the candidate module group are configured as follows: (channel expansion ratio of the candidate module, kernel size of the candidate module) including: (1,3), (3,3), (6,3), (1,5), (3,5), (6,5); the layer types include: convolutional layer, shift layer and additive layer.

[0012] Optionally, the hyperparameters of the search space for the candidate module group can be configured to include a skip operation, which is used to skip the corresponding candidate module.

[0013] Optionally, the pre-trained supernetwork includes: pre-trained convolutional layers; fixed pre-trained convolutional layers; pre-trained additive layers; and pre-trained the entire deep neural network.

[0014] Optionally, the initial value of the scaling factor of the batch normalization layer can be set to 0, and the pre-training learning rate can be set to 0.1.

[0015] The second aspect of this application provides a hardware accelerator for deep neural networks, used to accelerate a target deep neural network obtained by using the architecture search method of the deep neural network provided in the first aspect of this application. The deep neural network hardware accelerator includes: an off-chip storage module, a global storage module and an on-chip network module connected to the off-chip storage module, a convolution processing module connected to the on-chip network module, and a shift processing module and / or an addition processing module connected to the on-chip network module, wherein the global storage module and the on-chip network module are connected.

[0016] The off-chip storage module is used to store pre-trained model parameters, as well as input and output data.

[0017] The global storage module is used to read input data and cache intermediate computation data within the chip;

[0018] The on-chip network module is used to read pre-trained network parameters and enhance on-chip data reuse;

[0019] The convolution processing module is used to accelerate the convolutional layers in deep neural networks. It includes multiple multiply-accumulate units, which form a systolic array.

[0020] The shift processing module is used to accelerate the shift layer in the deep neural network, and includes multiple shift accumulation units, which form a systolic array.

[0021] The addition processing module is used to accelerate the addition layer in deep neural networks. It includes multiple addition accumulation units, which form a pulsating array.

[0022] Optionally, the multiply-accumulate unit, shift unit, and add unit all include register sub-units.

[0023] Optionally, deep neural network accelerators may also include: automappers;

[0024] The automapper constructs a data flow search space using nested for loop descriptions determined by cyclic sorting factors and cyclic tiling factors, for automatically searching for the optimal data flow.

[0025] This application provides a method for architecture search of deep neural networks and a hardware accelerator. The architecture search method includes: constructing a supernetwork, which includes a first fixed layer, candidate module groups, a second fixed layer, a pooling layer, and a classifier connected in sequence; the search space of the candidate modules includes convolutional layers and non-convolutional layers, where the non-convolutional layers are additive layers and / or shift layers; pre-training the supernetwork to update its weight parameters; and using a differentiable search algorithm to search from the pre-trained supernetwork to obtain a target deep neural network. The search space of the candidate module groups includes convolutional layers, additive layers, and / or shift layers. Thus, the obtained target deep neural network is a multiplication-reduced hybrid deep neural network, which has the advantages of high accuracy of convolutional layers based on multiplication operators and high hardware efficiency of additive or shift layers, achieving a balance between hardware efficiency and task accuracy. Attached Figure Description

[0026] Figure 1 A schematic diagram illustrating the workflow of a deep neural network architecture search method provided in this application embodiment;

[0027] Figure 2 A schematic diagram of the structure of a supernetwork constructed in a deep neural network architecture search method provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram of the weight distribution in convolutional layers, shift layers constructed using DeepShift-PS, shift layers constructed using DeepShift-Q, and additive layers;

[0029] Figure 4A schematic diagram illustrating the workflow of a pre-trained supernetwork in a deep neural network architecture search method provided in this application embodiment;

[0030] Figure 5 A schematic diagram of the structure of a hardware reducer for a deep neural network provided in an embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the PE scheduling strategy process for a hardware accelerator of a deep neural network provided in an embodiment of this application. Detailed Implementation

[0032] The deep neural network provided in this application can be used in image classification tasks in computer vision. While traditional deep neural networks based on multiplication operators achieve high classification accuracy in image classification tasks, their hardware implementation incurs significant power consumption and area overhead, making them difficult to apply to mobile devices or IoT devices. Conversely, deep neural networks based on non-multiplication operators offer superior hardware efficiency, but their classification accuracy is relatively low. To address these issues, this application provides a deep neural network and its hardware accelerator through the following embodiments.

[0033] See Figure 1 This application provides a method for searching the architecture of a deep neural network. The architecture search method searches for a multiplicative reduced hybrid deep neural network. The architecture search method includes:

[0034] Step 11: Construct a super network, which includes: a first fixed layer, a candidate module group connected to the first fixed layer, a second fixed layer connected to the candidate module group, a pooling layer connected to the second fixed layer, and a classifier connected to the pooling layer. The search space of the candidate module group includes convolutional layers and non-convolutional layers, where the non-convolutional layers are additive layers and / or shift layers.

[0035] Step 12: Pre-train the super network to update the weight parameters of the super network and obtain the pre-trained super network;

[0036] Step 13: Use a differentiable search algorithm to search from the pre-trained supernetwork to obtain the target deep neural network.

[0037] In some embodiments, the target deep neural network can be used to perform computer image classification tasks, taking an image to be classified as input and obtaining the corresponding classification result.

[0038] like Figure 2As shown, the first and last three layers of the supernetwork are fixed, while the remaining layers need to be searched from a predefined search space. The candidate module comprises multiple candidate modules connected sequentially. Each candidate module includes two pointwise layers (PW) and one depthwise layer (DW). The first PW is followed by a batch normalization (BN) layer and an activation layer. The depthwise layer is followed by a normalization layer and an activation layer. For example, the activation layer uses the ReLU activation function. The second PW is followed by a BN layer, and the output result is concatenated with the residual input data of the current candidate module to obtain the output data of the current candidate module.

[0039] In some embodiments, the first fixed layer is a 3×3 convolutional layer, and the second fixed layer is a 1×1 convolutional layer. In other embodiments, convolutional layers with other sizes of output channels may be used.

[0040] from Figure 2 As can be seen, each candidate module is represented by three hyperparameters: the channel dilation ratio E, the kernel size K, and the layer type T. These hyperparameters are obtained from a predefined search space.

[0041] In some embodiments, the search space of the candidate module group is based on FBNet. FBNet approximates a discrete search space as a continuous space to search for the network architecture of a deep neural network.

[0042] In some embodiments, the search space of the candidate module group includes convolutional layers and shift layers. In this application, the hybrid shift search space is used to describe this search space, and the hybrid shift network is used to describe the target deep neural network obtained by using the hybrid shift search space.

[0043] In some embodiments, the calculation process of the shift layer of the hybrid shift neural network can be performed using the formula Y = ∑X. T *W shift The diagram shows a shift layer. Here, X is the input to the shift layer, Y is the output of the shift layer, ∑() is the summation operation, and W... shift These are the weights of the shift layer. In one possible implementation, the weights W are determined using the DeepShift-Q method. shift That is, by using the original weights w in the convolutional layer * The shift layer is constructed by quantizing to powers of 2, as expressed by the following formula:

[0044]

[0045] In this context, sign() represents the sign operation, and round() represents the rounding operation.

[0046] See Figure 3 This diagram illustrates the weight distribution in a convolutional layer, a shift layer defined using the DeepShift-PS method, and a shift layer defined using the DeepShift-Q method. From... Figure 3 As can be seen, compared with other methods of determining the shift layer weights that directly optimize the sign flip bit s and shift bit p (such as DeepShift-PS), the DeepShift-Q method can match the small weights in the convolutional layer, which is more friendly to the training process of deep neural networks.

[0047] The hyperparameter configuration in the hybrid shift search space is as follows: (E,K) includes: (1,3), (3,3, (6,3), (1,5), (3,5), (6,5); T includes convolutional layers and shift layers.

[0048] In some embodiments, the hybrid shift search space further includes a skip operation, which is used to skip corresponding candidate modules, thereby supporting the search for lighter-weight deep neural networks.

[0049] In some embodiments, the search space of the candidate module group includes convolutional layers and additive layers. In this application, such a search space is described by a hybrid additive search space, and the target deep neural network obtained by using the hybrid additive search space is described by a hybrid additive network.

[0050] In some embodiments, the hybrid addition search space employs the addition layer proposed by AdderNet. The addition layer utilizes computationally efficient addition operations and the L1 norm to determine the correlation between lateral weights and feature maps, expressed by the following formula: Y = ∑-|XW adder |, where W adder These are the weights of the additive layer, X is the input to the additive layer, Y is the output of the additive layer, and ∑() is the summation operation. See also Figure 2 In the above embodiment, (d) represents the weight distribution of the additive layer.

[0051] The hyperparameter configuration in the hybrid addition search space is as follows: (E,K) includes: (1,3), (3,3, (6,3), (1,5), (3,5), (6,5); T includes convolutional layers and addition layers.

[0052] In some embodiments, the hybrid addition search space further includes a skip operation, which is used to skip corresponding candidate modules, thereby supporting the search for lighter-weight deep neural networks.

[0053] In some embodiments, the search space of the candidate module group includes three types of layers: convolutional layers, shift layers, and additive layers. In this application, the pre-search space is described by a hybrid all-search space, and the target deep neural network obtained by searching in the hybrid all-search space is described by a hybrid all-network. The hyperparameter configuration of the hybrid all-search space is as follows: (E,K) includes: (1,3), (3,3), (6,3), (1,5), (3,5), (6,5); the layer types include: convolutional layers, shift layers, and additive layers.

[0054] In some embodiments, the hybrid search space also includes a skip operation, which is used to skip corresponding candidate modules, thereby supporting the search for lighter-weight deep neural networks.

[0055] As described above, the search spaces of the candidate module groups all contain the same (E,K) options, but have different T options. A hybrid shift or addition search space integrates only the shift or addition layers into the convolutional layers, while a hybrid all search space integrates both, used to construct multiplication-reduced deep neural networks.

[0056] For example, the supernetwork includes 22 candidate modules, and each candidate module can select 13 or 19 candidate blocks from the search space provided above (corresponding to option (E, K) * option T + 1). Therefore, the search space of the candidate module group has a total of 13 22 Or 19 22 A potential subnetwork.

[0057] See Figure 3 It can be seen that the weights of the additive layers in the hybrid additive search space and the hybrid all search space provided in this application typically follow a Laplace distribution, while the weights of the convolutional layers typically follow a Gaussian distribution. To prevent the deep neural network using the hybrid additive search space and the hybrid all search space from failing to converge during pre-training, embodiments of this application provide a progressive pre-training method to train the deep neural network to update the weights of the supernetwork.

[0058] Among them, participants Figure 4 The progressive pre-training method includes:

[0059] Step 21: Pre-train the convolutional layer.

[0060] Step 22: Fix the trained convolutional layer and pre-train the additive layer.

[0061] Step 23: Pre-train the entire deep neural network.

[0062] Furthermore, in step 21, in order to take full advantage of the fast convergence speed of the convolutional neural network, only the forward inference and backward gradient updates of the convolutional layers in the super network are performed; in step 22, the trained convolutional layers are fixed to optimize the remaining additive layers, which means that both the convolutional layers and the additive layers are used for inference, but only the latter is used for gradient updates; in step 23, the previously fixed convolutional layers are optimized together with the additive layers.

[0063] In some embodiments, in the progressive pre-training method, the initial scaling factor of the batch normalization layer is set to 0, and the learning rate is set to a high learning rate to support efficient searching of deep neural networks employing a hybrid additive search space and a hybrid all-search space. For example, the learning rate is increased from the commonly used 0.05 to 0.1. Although a high learning rate may lead to skipping the global optimum during pre-training, because additive networks converge slower than convolutional networks, in this embodiment, setting the pre-training learning rate to 0.1 when pre-training a deep neural network employing a hybrid additive search space and a hybrid all-search space can accelerate the convergence speed of the deep neural network.

[0064] In the deep neural network construction described in the above embodiments, the preset hybrid search space employs a differentiable network architecture search algorithm to approximate the discrete search space as a continuous space. A gradient-based optimization scheme is used to update the weight parameters w and network structure parameters α, where the weight parameters w in the hybrid additive network and the hybrid all-network are optimized based on the aforementioned progressive pre-training method. The corresponding loss function is: in, and w represents the cross-entropy loss function on the test and training sets. * The optimal value of the weight is represented by , st (subject to) represents the constraint condition in the mathematical formula, and λ is a coefficient used to weigh performance and efficiency.

[0065] In some embodiments, FLOPs (floating point operations) are used as the hardware loss function L. hw The surrogate hardware metrics are used. For shift and adder layers where FLOPs cannot be measured, the shift and / or adder layers are first treated as convolutional layers to obtain FLOPs, and then scaled according to the hardware power consumption ratio of the shift and / or adder layers relative to the convolutional layers. In some embodiments, unit power consumption, latency, parameter count, etc., can also be used as hardware surrogate metrics.

[0066] The above embodiments provide a method for searching the architecture of a deep neural network, comprising: constructing a super network, the super network including a first fixed layer, a candidate module group, a second fixed layer, a pooling layer, and a classifier connected in sequence; wherein, the candidate module group includes multiple candidate modules connected in sequence, the candidate module including two pointwise operation layers, one layerwise operation layer, and corresponding batch normalization layers and activation layers; the search space of the candidate modules includes convolutional layers and non-convolutional layers, the non-convolutional layers being additive layers and / or shift layers; pre-training the super network to update the weight parameters of the super network, obtaining a pre-trained super network; using a differentiable search algorithm to search from the pre-trained super network to obtain a target deep neural network; the search space of the candidate module group including convolutional layers, additive layers, and / or shift layers, thus, the target deep neural network finally obtained is a multiplication-reduced hybrid deep neural network. The target deep neural network can have the advantages of high accuracy of convolutional layers based on multiplication operators and high hardware efficiency of additive or shift layers, achieving a balance between hardware efficiency and task accuracy.

[0067] To accelerate the hardware-based search of the target deep neural network obtained using the deep neural network architecture search method provided in the foregoing embodiments, this application also provides an embodiment of a hardware accelerator for deep neural networks. The deep neural network hardware accelerator includes: an off-chip storage module, a global storage module and an on-chip network module connected to the off-chip storage module, and a convolution processing module, a shift processing module, and an addition processing module connected to the on-chip network module.

[0068] See Figure 5 This is an exemplary architecture diagram of the deep neural network hardware accelerator provided in this embodiment. Figure 5 As shown, in this embodiment, the deep neural network hardware accelerator employs a multi-core microarchitecture to facilitate the customization of processing elements (PEs) for heterogeneous layers in the deep neural network with automatic architecture search.

[0069] Among them, DRAM (Dynamic Random Access Memory) is an off-chip storage module. GB (Global Buffer) is the global storage module, NoC (Network on Chip) is the on-chip network module, CLP (Convolution Layer Processors) is the convolution processing module, SLP (Shift Layer Processors) is the shift processing module, and ALP (Adder Layer Processors) is the addition processing module.

[0070] DRAM is used to store pre-trained model parameters, input data, and output data; GB (Block Bar) is used to read input data and cache intermediate computation data on-chip; GB caches computation inputs and outputs on-chip to reduce the high cost of off-chip data access from DRAM. NoC (No-Chip) is a new type of on-chip communication architecture designed for multi-core SoCs (System-on-Chips). It refers to a network-based electronic system implemented on a single chip, in the form of an integrated circuit chip. NoC connects GB to CLP / SLP / ALP to further enhance data multiplexing, directly reads pre-trained network parameters (such as weights) from DRAM, reads input data from GB and assigns it to CLP / SLP / ALP, and outputs the computation results from CLP / SLP / ALP to GB. CLP / SLP / ALP are used to accelerate convolutional layers / shifting layers / addition layers of deep neural networks, respectively, and read corresponding input data and network parameters from NoC. After computation, they output the corresponding computation results to NoC.

[0071] Among them, NoC draws on the communication methods of distributed computing systems, and uses data routing and packet switching technology to replace the traditional bus structure. It solves the problems of poor scalability caused by the limited address space of the SoC bus structure, low communication efficiency caused by time-sharing communication, and power consumption and global clock synchronization caused by the architecture.

[0072] In some embodiments, the PEs in the CLP use multiplication and accumulation units (MACs), with multiple MACs forming a PE systolic array; the PEs in the SLP use shift and accumulation units, with multiple shift and accumulation units forming a PE systolic array; and the PEs in the ALP use addition and accumulation units, with multiple addition and accumulation units forming a PE systolic array. Although the CLP / SLP / ALP share DRAM, GB, and NoC, each PE has its own register sub-unit, used to store the corresponding input, weight, and partial sum of the PE. Thus, DRAM, GB, NoC, and register sub-units form a four-level storage structure to enhance data multiplexing. The register sub-units are implemented using RF (Register File).

[0073] The working principle of hierarchical data storage is based on the locality of data access. By automatically moving infrequently accessed data to lower levels of the storage hierarchy, higher-cost storage space is freed up for more frequently accessed data, resulting in better cost-effectiveness. It should be noted that in some other embodiments, other types of storage structures can also be used to construct the aforementioned deep neural network hardware accelerator.

[0074] See Figure 5 In the MAC unit, the "input", "weight", and "partial sum" are implemented using RF, with "X" representing a multiplier and "+" representing an adder; in the shift unit, the "input", "weight", and "partial sum" are implemented using RF, with ">>" representing a shift register and "+" representing an adder; in the addition unit, the "input", "weight", and "partial sum" are implemented using RF, with "+" representing an adder.

[0075] See Figure 6 Taking a five-layer hybrid network as an example, the scheduling strategy of the deep neural network hardware accelerator provided in the above embodiments is illustrated. The five-layer hybrid network consists of convolutional layers (…). Figure 6 Convolution 1) Shift layer ( Figure 6 Shift 1) Addition layer ( Figure 6 Addition 1) Shift layer ( Figure 6 2) Shifting in the middle, convolutional layer ( Figure 6 (Convolution 2 in the original text). Within each cycle, CLP, SLP, and ALP sequentially process the assigned convolutional layers (convolution 1, convolution 5), shift layers (shift 2, shift 4), and addition layers (addition 3); the input data for each processing module is independent. Figure 6 As can be seen, the throughput achievable by the deep neural network hardware accelerator is limited by the longest latency within a cycle. Therefore, the PE allocation strategy attempts to rationally allocate PE resources to all three processing modules to balance their throughput, thereby minimizing the latency of each cycle.

[0076] In one implementation, the above PE allocation strategy can be expressed as:

[0077] N CLP / O 卷积 =N SLP / O 移位 =N ALP / O 加法 ;

[0078] stA CLP +A SLP +A ALP =Area constraint;

[0079] Where, N CLP N represents the number of PEs in the convolutional processing module. SLP N represents the number of PEs in the shift processing module. ALP For the number of PEs in the addition processing module, O 卷积 O is the total number of operations in the convolutional layer. 移位 O is the total number of operations in the shift layer. 加法 This represents the total number of operations in the addition layer. A CLP A represents the hardware area occupied by the convolutional layer. SLP For the hardware area occupied by the shift layer, A ALP This refers to the hardware area occupied by the addition layer. The above PE allocation strategy ensures that the number of PEs allocated to the convolution processing module, shift processing module, and addition processing module is proportional to the total number of operations in the convolutional, shift, and addition layers of the hybrid network.

[0080] Since the deep neural network provided in the foregoing embodiments includes shift and / or addition layers with high area and energy efficiency, the corresponding deep neural network hardware accelerator can partially replace the more expensive multiply-accumulate units with lower-cost shift processing modules and / or addition processing modules under the same area budget, thereby improving parallelism and reducing the overall latency and energy cost of the hardware accelerator.

[0081] The algorithm that maps a neural network to a hardware accelerator in terms of time and space is called a dataflow. Dataflow can significantly impact the hardware efficiency of the accelerator. Because the deep neural networks provided in the above embodiments contain a large number of heterogeneous layers, there is a significant space for dataflow optimization in the deep neural network hardware accelerator. Therefore, in some embodiments, the algorithm side of the deep neural network hardware accelerator includes an automapper. The automapper uses a nested for loop description determined by a cyclic sorting factor and a cyclic tiling factor to construct the dataflow search space, automatically searching for the optimal dataflow to maximize hardware efficiency.

[0082] The cyclic sorting factor corresponds to how computations are scheduled within the target PE array and within each PE, thus determining the data reuse pattern, while the cyclic tiling factor determines how data is stored in each level of the storage structure to effectively accommodate the aforementioned cyclic tiling factor. Therefore, the data flow search space in the automapper can be summarized as follows:

[0083] A cyclic sorting factor is used to determine the data reuse mode. In this embodiment, four modes are searched for for the convolution / shift / addition processing module: Row Stationary (RS), Input Stationary (IS), Weight Stationary (WS), and Output Stationary (OS). Therefore, there are a total of 64 (4*4*4) multiplexing modes for the convolution / shift / addition processing module in the hardware accelerator.

[0084] The cyclic tiling factor is used to determine how data is stored in various levels of storage structures (e.g., DRAM, GB, NoC, and RF in the hardware accelerator) to effectively accommodate the aforementioned data reuse pattern, and can be derived from all possible options under resource constraints (e.g., storage and compute resource constraints).

[0085] The above embodiments provide a hardware accelerator for a deep neural network. The hardware accelerator includes an off-chip storage module, a global storage module and an on-chip network module connected to the off-chip storage module, and a convolution processing module, a shift processing module, and an addition processing module connected to the on-chip network module. The global storage module and the on-chip network module are connected. The off-chip storage module stores pre-trained model parameters, input data, and output data. The global storage module reads the input data and caches intermediate computational data within the chip. The on-chip network module reads the pre-trained network parameters and enhances on-chip data reuse. The convolution processing module accelerates convolutional layers in the deep neural network and includes multiple multiply-accumulate units forming a systolic array. The shift processing module accelerates shift layers in the deep neural network and includes multiple shift-accumulate units forming a systolic array. The addition processing module accelerates addition layers in the deep neural network and includes multiple addition-accumulate units forming a systolic array. The hardware accelerator for the deep neural network employs a multi-core microarchitecture to support heterogeneous layers within the deep neural network. Furthermore, the hardware accelerator's data storage structure adopts a four-layer hierarchical structure to enhance on-chip data reuse. In some embodiments, the hardware accelerator also integrates an automapper in the algorithm computation to automatically search for the optimal data stream, further improving hardware efficiency.

[0086] The deep neural network hardware accelerator provided in the above embodiments can be custom-implemented using ASIC (Application Specific Integrated Circuit).

[0087] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method for architecture search of deep neural networks, characterized in that, include: A super network is constructed, comprising a first fixed layer, a candidate module group connected to the first fixed layer, a second fixed layer connected to the candidate module group, a pooling layer connected to the second fixed layer, and a classifier connected to the pooling layer; wherein the search space of the candidate module group comprises convolutional layers and non-convolutional layers, and the non-convolutional layers are additive layers and / or shift layers. Pre-train the super network to update the weight parameters of the super network, and obtain the pre-trained super network; A target deep neural network is obtained by searching the pre-trained supernetwork using a differentiable search algorithm; the target deep neural network can be used to perform computer image classification tasks, taking an image to be classified as input and obtaining the corresponding classification result. The hyperparameters of the search space for the candidate module group are configured as follows: The channel expansion ratio of the candidate modules and the convolution kernel size of the candidate modules include: (1,3), (3,3), (6,3), (1,5), (3,5), (6,5); Layer types include: convolutional layers, shift layers, and additive layers; The formula for the shift layer includes Y = ∑X T *W shift Where X is the input of the shift layer, Y is the output of the shift layer, ∑() is the summation operation, and W shift These are the weights of the shift layer; the weights W are determined using the DeepShift-Q method. shift By using the original weights w in the convolutional layer * The shift layer is constructed by quantizing to powers of 2, as expressed by the following formula: Among them, sign() represents the sign operation, and round() represents the rounding operation; AdderNet proposes an additive layer; the additive layer is used to compute efficient addition operations and L1 norm to determine the correlation between lateral weights and feature maps. The formula for the additive layer includes: Y = ∑-|XW adder |, where W adder X is the weight of the additive layer, Y is the input of the additive layer, and ∑() is the summation operation.

2. The deep neural network architecture search method according to claim 1, characterized in that, The hyperparameter configuration of the search space of the candidate module group includes a skip operation, wherein the skip operation is used to skip the corresponding candidate module.

3. The deep neural network according to claim 1, characterized in that, The pre-trained supernetwork includes: The convolutional layers were pre-trained; Fixed, pre-trained convolutional layers; pre-trained additive layers. Pre-train the entire super network.

4. The deep neural network according to claim 3, characterized in that, Set the initial scaling factor of the batch normalization layer to 0 and the pre-training learning rate to 0.

1.

5. A hardware accelerator for a deep neural network, characterized in that, The hardware accelerator of the deep neural network is used to accelerate the target deep neural network obtained by the architecture search method of the deep neural network according to any one of claims 1-4. The hardware accelerator includes: an off-chip storage module, a global storage module and an on-chip network module connected to the off-chip storage module, a convolution processing module connected to the on-chip network module, and a shift processing module and / or an addition processing module connected to the on-chip network module, wherein the global storage module and the on-chip network module are connected. The off-chip storage module is used to store pre-trained model parameters, input data, and output data. The global storage module is used to read the input data and cache intermediate computation data within the chip. The on-chip network module is used to read the pre-trained network parameters and enhance on-chip data reuse; The convolution processing module is used to accelerate the convolutional layers in deep neural networks, and includes multiple multiply-accumulate units, which form a systolic array. The shift processing module is used to accelerate the shift layer in a deep neural network, and includes multiple shift accumulation units, which constitute a pulsating array. The addition processing module is used to accelerate the addition layer in a deep neural network, and includes multiple addition accumulation units, which form a pulsating array.

6. The hardware accelerator for deep neural networks according to claim 5, characterized in that, The multiply-accumulate unit, the shift unit, and the adder unit all include register subunits.

7. The hardware accelerator for deep neural networks according to claim 5, characterized in that, Also includes: Automatic mapper; The automapper uses a nested for loop description determined by a cyclic sorting factor and a cyclic tiling factor to construct a data flow search space for automatically searching for the optimal data flow.

Images