A hardware-aware differentiable neural architecture search method and device

Abstract

The application discloses a hardware-aware differentiable neural network structure searching method and device, relates to the technical field of neural networks, and realizes the deployment of an application based on a DNN model on an edge device with limited computing and storage resources. The main technical scheme of the application is as follows: a differentiable neural network structure is used to train a super network with scalable width and depth, and a more optimal subnetwork balancing accuracy and execution efficiency is extracted from the super network according to a hardware index table collected on a target hardware, and then the subnetwork is used to construct a target DNN model suitable for the target hardware. The application is applied to searching a neural network structure suitable for a target device and constructing a DNN model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, and in particular to a hardware-aware differentiable neural architecture search method and device. BACKGROUND

[0002] In recent years, deep neural network (DNN) technology has developed rapidly, and related models have achieved good results in practical applications such as biometric identification, autonomous driving, and speech recognition. In order to meet the needs of practical scenarios such as workstations, embedded devices, and mobile terminals, artificial intelligence applications not only require deep neural network models to have high inference accuracy, but also require them to be efficiently executed on hardware.

[0003] Currently, deep neural network models only consider accuracy requirements when training and do not consider hardware efficiency. Improving accuracy usually means more parameters and deeper network structures, resulting in longer inference delays and higher energy consumption. However, in low-cost and low-performance edge devices, they are limited by available computing and storage resources (such as on-chip memory size and the number of arithmetic units) of edge devices. The computing power of these hardware devices is much lower than that of data centers. Traditional DNN models are often computationally and storage-intensive, which makes them unable to efficiently execute on edge devices, or even sometimes unable to be successfully deployed. SUMMARY

[0004] Therefore, the present application provides a hardware-aware differentiable neural network structure search method and device, the main purpose of which is to use a differentiable neural network structure to train a super network with scalable width and depth, and to extract a more optimal subnetwork that balances accuracy and execution efficiency from the super network according to the hardware index table collected on the target hardware, and then use such a subnetwork to build a target DNN model suitable for the target hardware, thereby enabling efficient deployment of DNN model-based applications on edge devices with limited computing and storage resources.

[0005] In order to achieve the above purpose, the present application mainly provides the following technical solutions:

[0006] The first aspect of the present application provides a hardware-aware differentiable neural network structure search method, which comprises:

[0007] A plurality of computing blocks are assembled by using computing components obtained from different deep neural network structures, the computing blocks at least containing an activation layer computing component and other layer computing components, the other layer computing components at least containing a convolution layer computing component or a pooling layer computing component;

[0008] The hardware index table corresponding to the target hardware is obtained by constructing a plurality of deep neural network models using a plurality of calculation blocks and respectively deploying the plurality of deep neural network models to run on the target hardware.

[0009] A super network corresponding to the target hardware is constructed using a plurality of calculation blocks, each hidden layer in the super network comprising a plurality of calculation blocks, each calculation block having model parameters and architecture parameters.

[0010] The super network is trained and optimized according to the model parameters and architecture parameters of the plurality of calculation blocks to obtain a corresponding target super network.

[0011] A target subnetwork is derived by training an initial population of subnetworks from the target super network using the hardware index table, the target subnetwork being used to construct a target deep neural network model suitable for the target device.

[0012] In some modified embodiments of the first aspect of the application, the hardware index table at least includes a mapping relationship between the calculation block identifiers, the accuracy indicators, and the latency indicators used to construct the deep neural network model, and the target subnetwork is derived by training an initial population of subnetworks from the target super network using the hardware index table, including:

[0013] The calculation blocks included in each subnetwork in the initial population are parsed.

[0014] According to the mapping relationship between the calculation block identifiers, the accuracy indicators, and the latency indicators included in the hardware index table, the index data corresponding to the calculation blocks in each subnetwork is obtained and the index data corresponding to each subnetwork is counted.

[0015] The index data corresponding to each subnetwork is input into a preset subnetwork scoring formula, and a scoring value corresponding to each subnetwork is output.

[0016] If the scoring value corresponding to the subnetwork reaches a preset target scoring value, the subnetwork is selected as the target subnetwork.

[0017] In some modified embodiments of the first aspect of the application, the subnetworks included in the initial population are selected as a first batch of subnetworks, and if none of the subnetworks in the first batch of subnetworks has a scoring value reaching a preset target scoring value, the method further includes:

[0018] The subnetwork with the highest scoring value is selected from the first batch of subnetworks as a base subnetwork.

[0019] A second batch of subnetworks is derived from the base subnetwork.

[0020] If the score corresponding to at least one of the subnets in the second batch of subnets reaches the preset target score, the subnet is selected as the target subnet;

[0021] If the score corresponding to none of the subnets in the second batch of subnets reaches the preset target score, the operation of evolving and deriving the next batch of subnets is repeatedly performed until at least one of the subnets in a specified batch of subnets has a score reaching the preset target score, and the subnet is selected as the target subnet and the operation of evolving and deriving the next batch of subnets is stopped.

[0022] In some modified embodiments of the first aspect of the application, before the operation of repeatedly evolving and deriving the next batch of subnets is performed, the method further comprises:

[0023] obtaining the current evolution derivation times;

[0024] If the score corresponding to none of the subnets in the batch of subnets corresponding to the current evolution derivation times reaches the preset target score, but the current evolution derivation times reaches the preset evolution derivation times threshold, the subnet with the highest score is selected from the batch of subnets corresponding to the current evolution derivation times as the target subnet, and the operation of evolving and deriving the next batch of subnets is stopped.

[0025] In some modified embodiments of the first aspect of the application, the obtaining of the hardware index table corresponding to the target hardware by constructing a plurality of deep neural network models using a plurality of computing blocks and deploying them to the target hardware for running comprises:

[0026] selecting a target computing block from the plurality of computing blocks;

[0027] inserting the target computing block into a plurality of hidden layers in a first deep neural network model to obtain a second deep neural network model;

[0028] repeatedly performing the operation of inserting the first deep neural network model into a plurality of hidden layers using other computing blocks except the target computing block in the plurality of computing blocks to obtain a plurality of second deep neural network models;

[0029] deploying the plurality of second deep neural network models to the target hardware for running to obtain index data corresponding to each of the second deep neural network models, the index data at least including an accuracy index and a time delay index;

[0030] composing the hardware index table corresponding to the target hardware according to the computing block identifier included in each of the second deep neural network models, the accuracy index corresponding to the second deep neural network model, and the time delay index.

[0031] In some modified embodiments of the first aspect of the application, the training and optimization of the super network according to the model parameters and the architecture parameters of the plurality of computing blocks comprises:

[0032] The application of the plurality of computing blocks to the super network corresponding to the target hardware is used to construct a first super network;

[0033] In each hidden layer of the first super network, the model parameters and the architecture parameters of the plurality of computing blocks are trained to obtain a second super network with updated model parameters and architecture parameters;

[0034] If the accuracy of the model constructed by the second super network reaches a preset target accuracy, the second super network is used as the target super network;

[0035] If the accuracy of the model constructed by the second super network does not reach the preset target accuracy, the next round of training of the model parameters and the architecture parameters of the computing blocks in each hidden layer is performed until the accuracy of the model constructed by the next super network reaches the preset target accuracy, and the next round of training of the model parameters and the architecture parameters of the computing blocks in each hidden layer is stopped.

[0036] In some modified embodiments of the first aspect of the application, before the next round of training of the model parameters and the architecture parameters of the computing blocks in each hidden layer is performed, the method further comprises:

[0037] The number of current loop training is accumulated;

[0038] The super network corresponding to the number of current loop training is determined;

[0039] If the accuracy of the model constructed by the super network does not reach the preset target accuracy, but the number of current loop training reaches a preset maximum number of loop training, the super network is obtained as the target super network, and the next round of training of the model parameters and the architecture parameters of the computing blocks in each hidden layer is stopped.

[0040] In some modified embodiments of the first aspect of the application, the method further comprises:

[0041] The target deep neural network model is run to output a corresponding accuracy;

[0042] By comparing the accuracy with a preset accuracy threshold, a loss value is determined;

[0043] According to the loss value, back propagation of model parameters of the target deep neural network model is performed, model parameters possessed by a calculation block in a target subnetwork corresponding to the target deep neural network model are updated, and an adjusted deep neural network model is obtained.

[0044] The second aspect of the present application provides a hardware-aware differentiable neural architecture search device, which comprises:

[0045] The assembling unit is configured to assemble a plurality of calculation blocks by using calculation components obtained from different deep neural network structures, wherein each of the calculation blocks comprises at least one activation layer calculation component and other layer calculation components, and the other layer calculation components comprise at least one convolution layer calculation component or one pooling layer calculation component.

[0046] The obtaining unit is configured to obtain a hardware index table corresponding to the target hardware by constructing a plurality of deep neural network models using the plurality of calculation blocks and deploying the deep neural network models on the target hardware for running.

[0047] The constructing unit is configured to construct a super network corresponding to the target hardware using the plurality of calculation blocks, wherein each hidden layer of the super network comprises a plurality of the calculation blocks, and each of the calculation blocks has model parameters and architecture parameters.

[0048] The first training unit is configured to train and optimize the super network according to the model parameters and the architecture parameters of the plurality of calculation blocks, and obtain a corresponding target super network.

[0049] The second training unit is configured to form an initial population by sampling subnetworks from the target super network, and to train the initial population to evolve and derive a target subnetwork using the hardware index table, wherein the target subnetwork is used to construct a target deep neural network model suitable for the target device.

[0050] The third aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the hardware-aware differentiable neural architecture search method as described above.

[0051] The fourth aspect of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the hardware-aware differentiable neural architecture search method as described above when executing the computer program.

[0052] By means of the above technical solution, the technical solution provided by the present application has at least the following advantages:

[0053] The application provides a hardware-aware differentiable neural network structure search method, which comprises the following steps: using the calculation components obtained from different DNNs to assemble a plurality of calculation blocks, further constructing a DNN model based on each calculation block and deploying the DNN model on a target hardware to run, collecting a hardware index table corresponding to the target hardware, wherein the hardware index table actually comprises an accuracy index, a time delay index and the like corresponding to each calculation block; using the calculation blocks to construct a super network applied to the target hardware, optimizing the model parameters and the architecture parameters of each hidden layer in the super network, obtaining an optimized target super network, then sampling a subnetwork from the target super network to form an initial population, and training the initial population by using the hardware index table to obtain a better target subnetwork, so that the target DNN model constructed by using the target subnetwork is suitable for the target device. Compared with the prior art, the technical problem that the DNN model cannot be efficiently executed on the edge device or even cannot be effectively deployed due to the fact that only the model accuracy is considered when training the DNN model and the hardware resources of the edge device are not fully considered is solved. The application finally extracts a better subnetwork balancing the accuracy and execution efficiency, and then uses the subnetwork to construct a target DNN model suitable for the target hardware, so that the application based on the DNN model can be effectively deployed on the edge device with limited computing and storage resources.

[0054] The above description is only a summary of the technical scheme of the application, in order to enable one skilled in the art to better understand the technical means of the application and implement it according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the application to be more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several drawings to represent similar components. In the drawings:

[0056] Figure 1 A hardware-aware differentiable neural network structure search method flowchart provided by an embodiment of the application;

[0057] Figure 2 (a) a DNN model basic calculation component schematic diagram provided by an embodiment of the application, and Figure 2 (b) a Block schematic diagram provided by an embodiment of the application;

[0058] Figure 3 A DNN model sketch deployment schematic diagram provided by an embodiment of the application;

[0059] Figure 4 The following is a table of hardware specifications exemplified in the embodiments of the present invention;

[0060] Figure 5 This is a schematic diagram of a super network exemplified in an embodiment of the present invention;

[0061] Figure 6 A flowchart of another hardware-aware differentiable neural network structure search method provided in an embodiment of the present invention;

[0062] Figure 7 A block diagram illustrating the composition of a hardware-aware, differentiable neural network structure search device provided in an embodiment of the present invention;

[0063] Figure 8 A block diagram illustrating the composition of another hardware-aware, differentiable neural network structure search device provided in an embodiment of the present invention. Detailed Implementation

[0064] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0065] This invention provides a hardware-aware, differentiable neural network architecture search method, such as... Figure 1 As shown, this method considers not only model accuracy but also applicability to a specific target device when constructing the target DNN model. This ensures efficient execution of the target DNN model on the target device, achieving effective deployment of the DNN model on edge devices. The specific steps provided in this embodiment are as follows:

[0066] 101. Multiple computational blocks are constructed by utilizing computational components obtained from different deep neural network structures.

[0067] Each computation block contains at least one activation layer computation component and other layer computation components, and each other layer computation component contains at least one convolutional layer computation component or one pooling layer computation component.

[0068] In this embodiment of the invention, the most advanced deep neural network (DNN) structures, such as the VGG series, ResNet series, and MobileNet series, are summarized to obtain some basic computational components, such as convolutional layer computational components, pooling layer computational components, and activation layer computational components.

[0069] For example, such as Figure 2(a) The basic calculation component diagram of the DNN model shown, where "Conv" is a convolution layer, "Conv3*3", "Conv5*5", and "Conv7*7" are convolution layer calculation components; "DW-Conv" represents a depthwise separable convolution layer, "DW-Conv3*3", "DW-Conv5*5", and "DW-Conv7*7" are depthwise separable convolution layer calculation components; "Max Pooling" is maximum pooling, which corresponds to a pooling layer calculation component; "Average Pooling" is average pooling, which corresponds to a pooling layer calculation component; "ReLU6Activation" represents an activation layer calculation component.

[0070] In an embodiment of the present application, a calculation block (Block for short) is randomly selected from multiple calculation components according to the depth requirement of the Block, but at least one activation layer calculation component needs to be included in the selection to form a Block, and multiple Blocks are formed as {Block1, Block2, …, Blockn}, as shown in Figure 2 (b) The Block diagram is shown.

[0071] For example, if the Block depth requirement is randomly selected between 2-4, for example, the depth is 3, then the number of calculation components in the Block is determined to be 3, and an activation layer calculation component is ensured to be selected, and then the other two calculation components can include three cases, specifically: randomly selected two convolution layer calculation components; randomly selected two pooling layer calculation components; randomly selected convolution layer calculation component and pooling layer calculation component.

[0072] 102. By using multiple calculation blocks to build multiple deep neural network models and deploying them on target hardware for running, the hardware index table corresponding to the target hardware is obtained.

[0073] The calculation block (Block for short) is a basic unit for performing network structure search, so the purpose of this step is actually to: collect each Block to form a DNN model, and then deploy it on the target hardware for running, and finally the corresponding hardware index data is obtained, which can be used to form a hardware index table.

[0074] For example, as shown in Figure 3 The DNN model sketch deployment diagram is shown. If n Blocks are obtained through step 101, then each Block is used to build a DNN model, and finally n DNN models are obtained.

[0075] Among them, Figure 3The "Pre-Process" refers to pre-processing, and the "Post-Process" refers to post-processing. If an example is used using a DNN model structure, the pre-processing refers to the pre-processing before the input layer of the model, for example, the resolution of a naturally captured picture is 1000*1000, but for a specific DNN model, only pictures with a resolution of 600*600 can be processed, so the picture needs to be pre-processed to 600*600, such as edge blur processing or pre-processing of cutting out the middle part; the post-processing refers to the full connection layer to the output layer of the model, such as some binary classification processing or labeling processing.

[0076] In the embodiments of the present application, using each Block to build a DNN model refers to: selecting a Block from n Blocks, and then inserting the Block into each hidden layer of the DNN structure to build a DNN model for the Block. Since there are n Blocks, the operation of building a DNN model is repeatedly performed by selecting Blocks from the n Blocks one by one, and finally n DNN models are obtained.

[0077] For a DNN structure, the hidden layer data processing is mainly used for extracting image features, and the number of hidden layers is determined according to actual application requirements, which can be several to several hundred or the like. After inserting the Block into the hidden layer of the DNN structure to complete the building of the DNN model, the DNN model is deployed on the target hardware device for running, and then according to the model running effect on the target hardware, the index data of each Block can be indirectly understood.

[0078] Therefore, for n DNN models built by n Blocks, the index data of the n Blocks on the target hardware can be collected, which can be formed into a hardware index table corresponding to the target hardware, as shown in Table 1. Figure 4 In Table 1, "Hardware" is the identification of the target hardware, "Accuracy" is the accuracy, "Latency" is the time delay, and "Energy" is the energy consumption. Then the hardware index table at least contains the mapping relationship among the Block identification in the DNN model, the accuracy index, and the time delay index.

[0079] For example, as shown in Table 2, two hardware parts (Hareware Part) are also exemplified, Ultra96 v2 FPGA (Field Programmable Gate Array) and rk3399 pro NPU (Embedded Neural Network Processor). Figure 3

[0080] 103、Utilize a plurality of computing blocks to build a super network corresponding to the target hardware. ​

[0081] In the embodiment of the present application, a plurality of Blocks are simultaneously inserted into each hidden layer of the DNN structure, thereby using the Blocks as a search space to construct a super network, as shown in the following formula: Figure 5 The super network schematic diagram is shown in the figure, in which i represents the number of hidden layers, n represents the number of Blocks inserted in each hidden layer, and a i,n represents the corresponding associated architecture parameters of the nth Block in the ith hidden layer.

[0082] It should be noted that for the constructed super network, there are n Blocks in each hidden layer, such as {Block1, Block2, …, Blockj, …, Blockn}, and for any hidden layer, each Block is assigned an initial model parameter and an initial architecture parameter, the model parameter is used to indicate the computing power of the Block, and the architecture parameter is used to indicate the importance of a certain Block in each hidden layer participating in the operation accuracy of the obtained DNN model.

[0083] 104. Training and optimizing the super network according to the model parameters and architecture parameters of the plurality of computing blocks to obtain a corresponding target super network.

[0084] In the embodiment of the present application, the super network constructed in step 103 is actually the same based on the initial assignment of the Block. Then, training and optimizing the super network is actually training and optimizing the model parameters and architecture parameters of different Blocks in each hidden layer, so that after a plurality of rounds of training and optimization operations, although the number of Blocks in each hidden layer is still the same, the model parameters and architecture parameters of the Blocks in any hidden layer are constantly changing, so that for any two hidden layers in the super network, the Blocks in each of them are changed, and finally the target super network after training and optimization is different, and after a plurality of rounds of training and optimization, the model parameters and architecture parameters of the Blocks in the target super network are close to the optimal.

[0085] 105. Forming an initial population by sampling a sub-network from the target super network, and using the hardware index table to train the initial population to evolve and derive a target sub-network, the target sub-network is used to construct a target deep neural network model suitable for the target device.

[0086] In the embodiment of the present application, for collecting subnets from the target super network to form an initial population means that: assuming that the target super network contains i hidden layers, each hidden layer contains n blocks, then a block is randomly selected from each hidden layer to obtain a subnet, the subnet contains i hidden layers and only one block in each hidden layer, according to which, multiple subnets can be collected from a target super network, and an initial population is formed by using these subnets.

[0087] In the embodiment of the present application, for training the initial population by using the hardware index table to obtain the target subnet means that: taking an initial population as the starting point, since the hardware index table actually stores the index data corresponding to each block on the target hardware, the index data of each subnet in the initial population can be further obtained according to the index data corresponding to each block, and then it can be compared which subnet in the initial population is the best; then the optimal subnet is evolved to derive multiple subnets, and the index data of each subnet is further obtained, and then a better subnet is searched, and such iterative operation is performed for multiple rounds of training, and the subnet with the best index data can be found, which is a better subnet that balances the model processing accuracy and execution efficiency on the target hardware.

[0088] In summary, the hardware-aware differentiable neural network structure search method and device provided by the embodiment of the present application uses the calculation components obtained from different DNNs to assemble multiple calculation blocks, and further constructs a DNN model based on each calculation block and deploys it on the target hardware for running, collects the hardware index table corresponding to the target hardware, which actually includes the accuracy index, delay index and the like corresponding to each calculation block. Then, the calculation blocks are used to construct a super network applied to the target hardware and optimize the model parameters and architecture parameters of each calculation block in each hidden layer in the super network to obtain an optimized target super network, and then subnets are collected from the target super network to form an initial population, and the hardware index table obtained above is used to train the initial population to obtain a better target subnet, so that the target DNN model constructed by using the target subnet is suitable for the target device. Compared with the prior art, the technical problem that the DNN model cannot be efficiently executed on the edge device or even cannot be effectively deployed due to the fact that only model accuracy is considered when training the DNN model and the hardware resources of the edge device are not fully considered is solved. The embodiment of the present application finally extracts a better subnet that balances accuracy and execution efficiency, and then uses such a subnet to construct a target DNN model suitable for the target hardware, thereby realizing the effective deployment of the application based on the DNN model on the edge device with limited computing and storage resources.

[0089] To make a more detailed description of the above embodiments, the embodiments of the present application also provide another hardware-aware differentiable neural network architecture search method, as shown in Figure 6 The method is a refinement and supplementary statement of the above embodiments, and the embodiments of the present application provide the following specific steps:

[0090] 201. Assemble multiple computing blocks by using the computing components obtained from different deep neural network structures.

[0091] Among them, the computing block at least contains an activation layer computing component and other layer computing components, and the other layer computing components at least contain a convolution layer computing component or a pooling layer computing component.

[0092] In the embodiments of the present application, for the statement of this step, see step 101, which will not be repeated here.

[0093] 202. Obtain the hardware index table corresponding to the target hardware by using multiple computing blocks to build multiple deep neural network models and respectively deploying them to the target hardware for running.

[0094] In the embodiments of the present application, a DNN model is constructed by inserting the components in the hidden layers of the DNN structure with Blocks one by one, as shown in Figure 3 The DNN model sketch deployment diagram, for example, inserts Blockn into each hidden layer of the DNN structure, and the number of hidden layers of the DNN structure can be pre-set according to actual needs.

[0095] Specifically, the specific implementation steps of constructing a DNN model by inserting components in the hidden layers of the DNN structure with Blocks one by one can include the following:

[0096] First, select a target computing block (identified as Block1) from multiple Blocks, and insert the components in each hidden layer of the first deep neural network model with Block1 to obtain a second deep neural network model.

[0097] It should be noted that the word "first" identifies the DNN model before the Block is inserted, and the word "second" is used to identify the DNN model after the Block is inserted. The words "first" and "second" are only used to distinguish different DNN models.

[0098] Second, after the above inserting Block1 into the first DNN model to construct a second DNN model, the above inserting Block into the first DNN model to obtain a second DNN model is repeatedly performed with other Blocks, accordingly, for example, Block is {Block1, Block2, …… Blockn}, then n second DNN models will be obtained.

[0099] Finally, the plurality of second deep neural network models are respectively deployed on the target hardware to run, to obtain index data corresponding to each second deep neural network model, the index data at least including an accuracy index and a latency index. Then, according to the calculation block identifier contained in each second deep neural network model, the accuracy index and the latency index corresponding to the second deep neural network model, a hardware index table corresponding to the target hardware is formed.

[0100] For example, Block is {Block1, Block2, …… Blockn}, then n second DNN models are obtained. The n second DNN models are respectively deployed on the target hardware to run, to obtain index data corresponding to each second DNN model, the index data at least including an accuracy index and a latency index.

[0101] It should be noted that since the difference between different DNN models is only that different Blocks are used to construct, the index data corresponding to different second DNN models is equivalent to the index data corresponding to different Blocks.

[0102] Then, for n Blocks, a hardware index table can be established, for example, the hardware index table shown in FIG. 8, which includes attribute fields of “Block”, “Hardware”, “Accuracy”, “Latency”, and “Enrgy”. According to the table, the accuracy index, the latency index, and the energy consumption index corresponding to different Blocks on the same target hardware can be shown. Figure 4

[0103] 203. Constructing a super network applied to a target hardware by using a plurality of calculation blocks.

[0104] Each hidden layer in the super network contains a plurality of calculation blocks and architecture parameters corresponding to each calculation block.

[0105] In the embodiment of the application, the present step is described with reference to step 103, which will not be repeated here.

[0106] 204. Training and optimizing the super network according to the model parameters and the architecture parameters of the plurality of calculation blocks, to obtain a corresponding target super network.

[0107] In the embodiment of the application, the plurality of Blocks are simultaneously inserted into the hidden layers of each DNN structure, and a super network is constructed by using the Blocks as a search space, as shown in FIG. 7. Figure 5 i,n Each hidden layer layer i in the super network is composed of n Blocks and associated architecture parameters α ​​

[0108] For the embodiment of the application, the model accuracy and the maximum number of training cycles can be preset as two constraint conditions for training the optimized super network. Specifically, the accuracy constraint is the main constraint, and the maximum number of training cycles is the auxiliary constraint. The specific implementation steps for optimizing the training of the super network can include the following:

[0109] First, a plurality of computing blocks are used to construct a super network corresponding to the target hardware as a first super network. In each hidden layer of the first super network, the model parameters and the architecture parameters of the plurality of computing blocks are trained to obtain a second super network with updated model parameters and architecture parameters.

[0110] It should be noted that for the constructed super network, as shown in Figure 5 each hidden layer has n blocks, such as {Block1, Block2, …, Blockj, …, Blockn}. For any hidden layer (referred to as layer), each Block is assigned initial model parameters and initial architecture parameters. The model parameters are used to indicate the computing power of the Block, and the architecture parameters are used to indicate the importance of each Block in the layer for the DNN model operation accuracy.

[0111] In the embodiment of the application, the term "first" is used to identify the super network that has not been optimized and trained, and the term "second" is used to identify the super network that has completed a round of optimization and training.

[0112] For example, the first round of optimization and training of the super network is as follows:

[0113] Assuming that the Block model parameters are w and the architecture parameters are α, there are n Blocks in each layer. For example, there are four Blocks in a layer, such as {Block1, Block2, Block3, Block4}. Block1 is assigned initial values w1 and α1, Block2 is assigned initial values w2 and α2, Block3 is assigned initial values w3 and α3, and Block4 is assigned initial values w4 and α4.

[0114] Further, the rule of training the Block model parameters first and then training the architecture parameters is followed, and the principle of activating only one Block in each layer is also followed, to optimize the super network. By comparing the initial values of a1, a2, a3, and a4, it is assumed that the value of a1 is the largest, so that Block1 in the layer is activated, and the model parameters w1 of Block1 are trained to obtain w11, at this time, the ws of other Blocks are not changed; then, w11 and other unchanged ws2, ws3, and ws4 are used to train the architecture parameters a of each Block, to obtain the changed a11, a22, a33, and a44 corresponding to the four Blocks. The above is an example of training the model parameters and architecture parameters of the Blocks in a layer, but it needs to be explained that each layer in the super network is trained simultaneously based on the same operation, and accordingly, the first round of optimization training of the super network is completed.

[0115] For example, for the second round of optimization training of the super network, the changed model parameters and architecture parameters of the Blocks in each layer in the first round of optimization training are used to repeat the above first round of optimization training operation, and if the third round of optimization training is needed, the optimization training is performed based on the super network obtained in the second round of optimization training, and if the third round, the fourth round, and so on are needed, the specific implementation steps are not described here.

[0116] Secondly, for the second super network completed by each round of optimization training, two constraint conditions need to be verified, and the specific implementation process includes the following:

[0117] In the embodiment of the application, the pre-set model accuracy is used as the constraint condition, for example: the second super network completed by the first round of optimization training is used to construct a DNN model, and then the DNN model is deployed on a target device to run, to obtain a model accuracy, which is compared with the pre-set accuracy, if the model accuracy reaches the pre-set accuracy, the second super network of the first round of optimization training is obtained as the target super network; but if the model accuracy does not reach the pre-set accuracy, the second round of optimization training of the super network is performed, that is, the one round of optimization training of the super network is repeated.

[0118] Further, the preset maximum number of training cycles is used as a constraint condition. For example, if the accuracy of the DNN model constructed by the second super network obtained through multiple rounds of optimization training of the super network still fails to reach the preset target accuracy, then the maximum number of training cycles is used to stop the unlimited iterative training of the super network. That is, when the preset maximum number of training cycles is met, the second super network corresponding to the round of optimization training is obtained as the target super network, and the next round of optimization training of the super network is stopped.

[0119] 205、Through the initial population formed by sampling the target super network, the initial population is evolved to derive the target subnetwork by using the hardware index table, and the target subnetwork is used to construct the target deep neural network model suitable for the target device.

[0120] It should be noted that, for the target super network obtained through multiple rounds of optimization training, since each layer is optimized independently, the model parameters and architecture parameters of each Block, after multiple rounds of optimization training, each layer still has the same number of Blocks, but the actual content of each layer includes different calculation blocks. The model parameters and architecture parameters of each Block in the target super network obtained through multiple rounds of optimization training are as close to optimal as possible. Therefore, the quality of the initial population formed by sampling the subnetwork in such a super network is also better.

[0121] In the embodiment of the present application, a target score value (as the main) and a maximum number of evolutions (as the auxiliary) can be preset as two constraint conditions. The specific implementation steps of obtaining the target subnetwork satisfying the two constraint conditions can include the following:

[0122] The first step is to form an initial population by sampling a subnetwork from the target super network, to include the subnetwork in the initial population as the first batch of subnetworks, to analyze the Block included in each subnetwork from the initial population, to obtain the index data corresponding to each Block from the hardware index table according to the mapping relationship between the Block identifier, the accuracy index, and the delay index included in the hardware index table, and to further statistically obtain the index data corresponding to each subnetwork.

[0123] For example, it is assumed that a super network layer is 5 layers and each layer contains 4 blocks {Block1, Block2, Block3, Block4}. It should be noted that after multi-round optimization training, although {Block1, Block2, Block3, Block4} are still included in different layers, Block1 in any two layers is not equivalent, and Block2 is not equivalent, Block3 is not equivalent, and Block4 is not equivalent. For example, taking Block1 as an example, for the optimized target super network obtained after multi-round optimization training, the model parameters and architecture parameters of Block1 in different layers are different, so Block1 in different layers cannot be equivalent. Here, the number 1 after Block is only used to indicate that the position of the corresponding calculation block in each layer is "first".

[0124] Then, a block is randomly extracted from each layer to form a subnet, which can be

Block1, Block1, Block2, Block2, Block2

[0125] In the second step, the index data corresponding to each subnet is input into a preset subnet scoring formula, and the scoring value corresponding to each subnet is output.

[0126] In the embodiment of the present application, the preset subnet scoring formula is used to realize the scoring of the subnet by comprehensively considering the accuracy index and the latency index (i.e., two factors). For example, the preset scoring formula can be, but is not limited to, the following formula (1):

[0127] target value=Accuracy+λ(Latency-target(Latency));

[0128] Wherein, the target value is the output score value; the Accuracy is the accuracy value, and the Latency is the latency value, and the values of the two indexes can be directly obtained from the hardware index table; the target (Latency) is a preset target latency value, and for each target hardware, a target latency value can be preset in advance; λ is an empirical value parameter artificially set and takes a value between 0 and 1, and is used to indicate the proportion of the required accuracy index and the latency index. Then, the score value obtained by using formula (1) is mainly used to indicate that the balance between the model accuracy index and the running efficiency index on the target hardware and the degree of the subnetwork effect.

[0129] Further, if the score value of the subnetwork reaches the preset target score value, the subnetwork is selected as the target subnetwork; but if the score values of the subnetworks in the initial population (i.e. the first batch of subnetworks) do not reach the preset target score value, the subnetwork with the highest score value is selected from the first batch of subnetworks as a basic subnetwork, and the second batch of subnetworks is derived from the basic subnetwork.

[0130] For example, it is assumed that the subnetwork with the highest score value selected from the initial population (i.e. the first batch of subnetworks) is

Block1, Block1, Block1, Block2, Block2

[0131] Based on the subnetwork with the highest score value selected from the initial population, the Blocks in the subnetwork are replaced each time, for example: five hidden layers are obtained from the subnetwork

Block1, Block1, Block1, Block2, Block2

[0132] For example, the first Block1 in the subnetwork

Block1, Block1, Block1, Block2, Block2

[0133] The above replacement operation of the Block in the subnet is used to evolve and mutate more subnets. Furthermore, the number of evolved and mutated subnets in each batch can be preset (for example, M subnets and M is a positive integer), so as to avoid too many generated subnets and too large operation amount of subsequent calculation of the scoring values, and to avoid reduction of subsequent processing efficiency.

[0134] After obtaining the second batch of subnets, the calculation of the index data corresponding to each subnet and the calculation of the corresponding scoring values are repeatedly performed, and the scoring values are compared with the preset target scoring value. If there is still no subnet satisfying the constraint condition, the subnet with the highest scoring value in the second batch of subnets is used to evolve and derive the next batch of subnets, until there is a mutated subnet whose corresponding scoring value satisfies the constraint condition, and the subnet satisfying the constraint condition is taken as the target subnet.

[0135] In the third step, another constraint condition (satisfying the preset maximum evolution times) is used to avoid too many times of repeated execution of whether the preset target scoring value is satisfied. Specifically, the following is performed:

[0136] If there is no subnet in the batch of subnets corresponding to the current evolution derivation times whose corresponding scoring value reaches the preset target scoring value, but the current evolution derivation times reaches the preset evolution derivation times threshold, the subnet with the highest scoring value in the batch of subnets corresponding to the current evolution derivation times is selected as the target subnet and the operation of evolving and deriving the next batch of subnets is stopped.

[0137] In the above embodiment of the present application, after multiple rounds of training, the target subnet with the best scoring value can be found. This subnet is a better target subnet that can balance the model processing accuracy and the execution efficiency on the target hardware. Therefore, the target DNN model can be constructed using the target subnet, so that the target DNN model is also suitable for the target device.

[0138] 206, using the running target deep neural network model, outputting the corresponding accuracy.

[0139] 207, determining the loss value by comparing the accuracy with the preset accuracy threshold.

[0140] 208, according to the loss value, performing back propagation of the model parameters of the target deep neural network model, updating the model parameters of the calculation block in the target subnet corresponding to the target deep neural network model, and obtaining the adjusted deep neural network model.

[0141] In the embodiment of the present application, in combination with steps 206-208, the target DNN model constructed with the target subnetwork can also be fine-tuned for X times to obtain a better DNN model. This fine-tuning is mainly fine-tuning the model parameters of the Block in the target DNN model. Specifically, an accuracy threshold and a maximum number of fine-tuning can be preset as two constraint conditions to implement the model fine-tuning process, which can specifically include the following:

[0142] First, the target DNN model is run to output the corresponding accuracy. By comparing the accuracy with the preset accuracy threshold, the loss value is determined. According to the loss value, the model parameters of the target DNN model are back-propagated to update the model parameters of the Block in the target subnetwork of the target DNN model, and an adjusted deep neural network model is obtained. This operation is a model fine-tuning operation.

[0143] Second, the fine-tuned DNN model is run to output the accuracy, and then it is judged whether the accuracy reaches the preset accuracy threshold. If it does, the fine-tuning of the target DNN model is ended. But if it does not, the above operation is repeated to update the model parameters of the Block in the target subnetwork of the target DNN model again as a second model fine-tuning operation, until the model accuracy after multiple fine-tuning reaches the target accuracy, and the next model fine-tuning operation is stopped.

[0144] It should be noted that the embodiment of the present application uses another constraint condition to avoid the problem of waste of processing resources caused by too many model fine-tuning times. That is, if the cumulative number of model fine-tuning reaches the maximum number of fine-tuning, the next round of model fine-tuning operation is stopped regardless of whether the accuracy of the fine-tuned model reaches the preset accuracy threshold or not, and the current fine-tuned model is obtained as the final target DNN model. This model is a DNN model suitable for the target device.

[0145] Further, in the embodiment of the present application, after the target DNN model is fine-tuned for X times, the model can also be optimized in relation to hardware, such as parallelism setting, I / O optimization, layer fusion, resource allocation optimization, and clock frequency. The specific operation is to run the fine-tuned target DNN model on the target hardware while performing manual parameter optimization.

[0146] In the embodiment of the present application, the final target DNN model is used to realize specific applications, including but not limited to image classification, target detection, target tracking, and image super-resolution. Specifically, according to different application requirements, the number of layers and Blocks is set when training the target DNN model in steps 201-208, which is not limited here.

[0147] Further, as an implementation of the method shown in the above Figure 1 , Figure 6 , the embodiment of the present application provides a hardware-aware differentiable neural network structure search device. The device embodiment corresponds to the foregoing method embodiment, and for the sake of easy reading, the details of the foregoing method embodiment will not be described one by one, but it should be clear that the device in this embodiment can correspondingly implement all the contents in the foregoing method embodiment. The device is applied to search for a DNN network structure suitable for a specified target device, and specifically as shown in Figure 7 , the device comprises:

[0148] The assembly unit 31 is configured to assemble a plurality of calculation blocks by using calculation components obtained from different deep neural network structures, wherein the calculation blocks at least contain an activation layer calculation component and other layer calculation components, and the other layer calculation components at least contain a convolution layer calculation component and / or a pooling layer calculation component.

[0149] The acquisition unit 32 is configured to obtain a hardware index table corresponding to the target hardware by constructing a plurality of deep neural network models using a plurality of calculation blocks and respectively deploying them to the target hardware for running, wherein the hardware index table at least contains a mapping relationship among the calculation block identifier, the accuracy index, and the time delay index in the deep neural network model.

[0150] The construction unit 33 is configured to construct a super network corresponding to the target hardware using a plurality of calculation blocks, wherein each hidden layer in the super network contains a plurality of calculation blocks, and each calculation block has model parameters and architecture parameters.

[0151] The first training unit 34 is configured to train and optimize the super network according to the model parameters and the architecture parameters of the plurality of calculation blocks to obtain a corresponding target super network.

[0152] The second training unit 35 is configured to form an initial population by sampling subnetworks from the target super network, and to train the initial population to evolve and derive a target subnetwork using the hardware index table, wherein the target subnetwork is used to construct a target deep neural network model suitable for the target device.

[0153] Further, as shown in Figure 8 , the second training unit 35 comprises:

[0154] The analysis module 351 is configured to analyze the calculation blocks contained in each subnetwork from the initial population.

[0155] The obtaining module 352 is configured to obtain the index data corresponding to each computing block in each subnet and count the index data corresponding to each subnet according to a mapping relationship among the computing block identifier, the accuracy index, and the time delay index contained in the hardware index table.

[0156] The processing module 353 is configured to input the index data corresponding to each subnet into a preset subnet scoring formula and output a scoring value corresponding to each subnet.

[0157] The selecting module 354 is configured to select the subnet as a target subnet if the scoring value corresponding to the subnet reaches a preset target scoring value.

[0158] Further, as shown in Figure 8 The second training unit 35 includes:

[0159] The determining module 355 is configured to select the initial population as a first batch of subnets.

[0160] The determining module 355 is further configured to select a subnet with the highest scoring value from the first batch of subnets as a basic subnet when no subnet in the first batch of subnets has a scoring value reaching a preset target scoring value.

[0161] The evolution module 356 is configured to evolve and derive a second batch of subnets according to the basic subnet.

[0162] The selecting module 354 is further configured to select the subnet as a target subnet when at least one subnet in the second batch of subnets has a scoring value reaching a preset target scoring value.

[0163] The first execution module 357 is configured to repeatedly perform the operation of evolving and deriving a next batch of subnets until at least one subnet in a specified batch of subnets has a scoring value reaching a preset target scoring value, and then select the subnet as a target subnet and stop the operation of evolving and deriving a next batch of subnets.

[0164] Further, as shown in Figure 8 The second training unit 35 further includes:

[0165] The obtaining module 352 is further configured to obtain a current evolution derivation frequency before repeatedly performing the operation of evolving and deriving a next batch of subnets.

[0166] The second execution module 358 is configured to: when there is no subnetwork corresponding to a preset target score value in the batch subnetwork corresponding to the current evolution derivation times, but the current evolution derivation times reaches a preset evolution derivation times threshold, select a subnetwork with the highest score value from the batch subnetwork corresponding to the current evolution derivation times as a target subnetwork, and stop performing evolution derivation on the next batch of subnetworks.

[0167] Further, as shown in Figure 8 The acquisition unit 32 includes:

[0168] The selection module 321 is configured to select a target calculation block from the plurality of calculation blocks.

[0169] The insertion module 322 is configured to insert the target calculation block into components in a plurality of hidden layers in a first deep neural network model to obtain a second deep neural network model.

[0170] The execution module 323 is configured to repeatedly perform the operation of inserting the components in the plurality of hidden layers in the first deep neural network model by using other calculation blocks except the target calculation block in the plurality of calculation blocks to obtain a plurality of second deep neural network models.

[0171] The running module 324 is configured to deploy the plurality of second deep neural network models to a target hardware for running respectively to obtain index data corresponding to each second deep neural network model, the index data at least including an accuracy index and a time delay index.

[0172] The composition module 325 is configured to compose a hardware index table corresponding to the target hardware according to a calculation block identifier included in each second deep neural network model, the accuracy index and the time delay index corresponding to the second deep neural network model.

[0173] Further, as shown in Figure 8 The first training unit 34 includes:

[0174] The determination module 341 is configured to construct a super network corresponding to the target hardware by using the plurality of calculation blocks as a first super network.

[0175] The training module 342 is configured to obtain a second super network with updated model parameters and architecture parameters by training the model parameters and the architecture parameters of the plurality of calculation blocks in each hidden layer of the first super network.

[0176] The determination module 341 is further configured to take the second super network as a target super network when the accuracy of a model constructed by the second super network reaches a preset target accuracy.

[0177] The first execution module 343 is configured to perform the next round of training of the model parameters and the architecture parameters of the calculation block of each hidden layer until the accuracy of the model constructed by the next super network reaches the preset target accuracy, and then stop performing the next round of training of the model parameters and the architecture parameters of the calculation block of each hidden layer.

[0178] Further, as shown in Figure 8 The first training unit 34 further includes:

[0179] The accumulation module 344 is configured to accumulate the number of current loop training before performing the next round of training of the model parameters and the architecture parameters of the calculation block of each hidden layer.

[0180] The determination module 341 is further configured to determine the super network corresponding to the number of current loop training.

[0181] The second execution module 345 is configured to obtain the super network as a target super network and stop performing the next round of training of the model parameters and the architecture parameters of the calculation block of each hidden layer when the accuracy of the model constructed by the super network does not reach the preset target accuracy, but the number of current loop training reaches the preset maximum number of loop training.

[0182] Further, as shown in Figure 8 The apparatus further includes:

[0183] The running unit 36 is configured to run the target deep neural network model to output a corresponding accuracy.

[0184] The determination unit 37 is further configured to determine a loss value by comparing the accuracy with a preset accuracy threshold.

[0185] The update unit 38 is configured to perform back propagation of model parameters of the target deep neural network model according to the loss value, update the model parameters of the calculation block in the target subnetwork corresponding to the target deep neural network model, and obtain an adjusted deep neural network model.

[0186] The hardware-aware differentiable neural architecture search apparatus includes a processor and a memory, and the above-mentioned component unit, acquisition unit, construction unit, first training unit, and second training unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0187] The processor comprises a core, and the core retrieves corresponding program units in the memory. The core can be provided with one or more than one, and the core parameters are adjusted to utilize the differentiable neural network structure, train the super network with scalable width and depth, and extract a more optimal subnetwork balancing accuracy and execution efficiency from the super network according to a hardware index table collected on the target hardware, and then utilize the subnetwork to construct a target DNN model suitable for the target hardware, so that the application based on the DNN model can be effectively deployed on the edge device with limited computing and storage resources.

[0188] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the hardware-aware differentiable neural network structure searching method.

[0189] The embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the hardware-aware differentiable neural network structure searching method.

[0190] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for implementing the functions specified in one or more flows and / or blocks Figure 1 The device for implementing the functions specified in one or more flows and / or blocks

[0191] In a typical configuration, the device comprises one or more processors (CPU), memory and bus. The device can also comprise an input / output interface, a network interface and the like.

[0192] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip. The memory is an example of the computer readable medium.

[0193] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0194] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that processes, methods, articles or apparatuses that comprise a list of elements not only include those elements, but also include other elements not expressly listed or inherent to such processes, methods, articles or apparatuses. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0195] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0196] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent insertion, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A hardware-aware, differentiable neural network structure search method, characterized in that, The method includes: By utilizing computational components obtained from different deep neural network structures, multiple computational blocks are constructed. Each computational block contains at least one activation layer computational component and other layer computational components. The other layer computational components contain at least one convolutional layer computational component or one pooling layer computational component. By constructing multiple deep neural network models using multiple computing blocks and deploying them to target hardware for execution, a hardware performance table corresponding to the target hardware is obtained. This includes: selecting a target computing block from the multiple computing blocks; inserting the target computing block into each hidden layer of a first deep neural network model to obtain a second deep neural network model; repeatedly executing the operation inserted into each hidden layer of the first deep neural network model using other computing blocks besides the target computing block to obtain multiple second deep neural network models; deploying the multiple second deep neural network models to the target hardware for execution to obtain performance data corresponding to each second deep neural network model, the performance data including at least an accuracy performance indicator and a latency performance indicator; and constructing a hardware performance table corresponding to the target hardware based on the computing block identifiers included in each second deep neural network model, the accuracy performance indicator, and the latency performance indicator corresponding to the second deep neural network model. The hardware performance table characterizes the performance data corresponding to each computing block on the target hardware, and the hardware performance table includes at least the mapping relationship between the computing block identifiers used to construct the deep neural network model, the accuracy performance indicator, and the latency performance indicator. A super network applied to the target hardware is constructed using multiple computing blocks, wherein each hidden layer of the super network contains multiple computing blocks, and each computing block has model parameters and architecture parameters; Based on the model parameters and architecture parameters of the multiple computational blocks, the super network is trained and optimized to obtain the corresponding target super network; in the target super network, the model parameters and architecture parameters of the computational blocks in each hidden layer are changed to the trained and optimized model parameters and architecture parameters. An initial population is formed by sampling subnets from the target supernetwork, and the initial population is trained using the hardware index table to derive the target subnet. The target subnet is used to construct a target deep neural network model suitable for the target device. The subnet is composed of a computational block extracted from each hidden layer of the target supernetwork, and the initial population is composed of the sampled subnets. The step of forming an initial population by sampling subnets from the target supernetwork and training the initial population to derive the target subnet using the hardware metric table includes: The computational blocks contained in each subnet are resolved from the initial population; Based on the mapping relationship between the computing block identifier, accuracy index, and latency index contained in the hardware index table, obtain the index data corresponding to the computing block in each subnet and count the index data corresponding to each subnet. Input the indicator data corresponding to each subnet into the preset subnet scoring formula, and output the score value corresponding to each subnet. If the score corresponding to the subnet reaches the preset target score, then the subnet is selected as the target subnet; The initial population includes subnets as the first batch of subnets. If no subnet in the first batch of subnets has a score that reaches the preset target score, the method further includes: Select the subnet with the highest score from the first batch of subnets as the base subnet; The second batch of subnets is derived from the basic subnet through evolution; If at least one subnet in the second batch of subnets has a score that reaches the preset target score, then the subnet is selected as the target subnet. If no subnet in the second batch of subnets has a score that reaches the preset target score, the evolutionary derivation of the next batch of subnets operation is repeated until at least one subnet in the specified batch of subnets has a score that reaches the preset target score. Then, the subnet is selected as the target subnet and the evolutionary derivation of the next batch of subnets operation is stopped. Before repeatedly performing the evolutionary derivation of the next batch of subnets, the method further includes: obtaining the current evolutionary derivation count; if no subnet in the batch of subnets corresponding to the current evolutionary derivation count has a score value that reaches the preset target score value, but the current evolutionary derivation count reaches the preset evolutionary derivation count threshold, then the subnet with the highest score value is selected from the batch of subnets corresponding to the current evolutionary derivation count as the target subnet, and the execution of the evolutionary derivation of the next batch of subnets is stopped.

2. The method according to claim 1, characterized in that, The step of training and optimizing the supernetwork based on the model parameters and architecture parameters of the multiple computational blocks to obtain the corresponding target supernetwork includes: A super network corresponding to the target hardware will be constructed using multiple computing blocks, serving as the first super network; Within each hidden layer of the first super network, a second super network with updated model and architecture parameters is obtained by training multiple computational blocks to have model parameters and architecture parameters. If the accuracy of the model built by the second super network reaches the preset target accuracy, then the second super network will be used as the target super network. If the accuracy of the model built by the second super network does not reach the preset target accuracy, then the model parameters and architecture parameters of the computational blocks of each hidden layer in the next round of training are executed until the accuracy of the model built by the next super network reaches the preset target accuracy. Then, the execution of the model parameters and architecture parameters of the computational blocks of each hidden layer in the next round of training is stopped.

3. The method according to claim 2, characterized in that, Before performing the model parameter and architecture parameter operations for each hidden layer computation block in the next round of training, the method further includes: Accumulate the current number of training iterations; Determine the supernetwork corresponding to the current number of training iterations; If the accuracy of the model built by the super network does not reach the preset target accuracy, but the current number of training iterations reaches the preset maximum number of training iterations, then the super network is selected as the target super network, and the operation of model parameters and architecture parameters of each hidden layer's computational block in the next round of training is stopped.

4. The method according to claim 1, characterized in that, The method further includes: By running the target deep neural network model, the corresponding accuracy is output; The loss value is determined by comparing the accuracy rate with a preset accuracy rate threshold; Based on the loss value, backpropagation of model parameters is performed on the target deep neural network model to update the model parameters of the computational blocks in the target subnet corresponding to the target deep neural network model, thereby obtaining the adjusted deep neural network model.

5. A hardware-aware, differentiable neural network structure search device, characterized in that, The device includes: The assembly unit is used to assemble multiple computation blocks by utilizing computational components obtained from different deep neural network structures. Each computation block contains at least one activation layer computational component and other layer computational components, wherein each other layer computational component contains at least one convolutional layer computational component or one pooling layer computational component. The acquisition unit is used to acquire a hardware index table corresponding to the target hardware by constructing multiple deep neural network models using multiple computing blocks and deploying them to run on the target hardware respectively. The acquisition unit includes: A selection module is used to select a target computation block from a plurality of computation blocks; An insertion module is used to insert components into multiple hidden layers of a first deep neural network model using the target computation block to obtain a second deep neural network model. An execution module is configured to repeatedly execute component operations inserted into multiple hidden layers of the first deep neural network model using other computational blocks besides the target computational block in multiple computational blocks, thereby obtaining multiple second deep neural network models; The running module is used to deploy multiple second deep neural network models to the target hardware for running, and obtain the indicator data corresponding to each second deep neural network model. The indicator data includes at least the accuracy indicator and the latency indicator. The component module is used to assemble a hardware indicator table corresponding to the target hardware based on the computation block identifier contained in each second deep neural network model, the accuracy indicator and the latency indicator corresponding to the second deep neural network model. The hardware indicator table is used to characterize the indicator data corresponding to each computation block on the target hardware. The hardware indicator table at least includes the mapping relationship between the computation block identifier, the accuracy indicator and the latency indicator used to construct the deep neural network model. A construction unit is used to construct a super network corresponding to the target hardware using multiple computing blocks, wherein each hidden layer in the super network contains multiple computing blocks, and each computing block has model parameters and architecture parameters; The first training unit is used to train and optimize the super network based on the model parameters and architecture parameters of the multiple computational blocks to obtain the corresponding target super network; in the target super network, the model parameters and architecture parameters of the computational blocks in each hidden layer are changed to the trained and optimized model parameters and architecture parameters. The second training unit is used to form an initial population by sampling subnets from the target supernetwork, and to train the initial population to evolve and derive a target subnet using the hardware index table. The target subnet is used to construct a target deep neural network model suitable for the target device. The subnet is composed of a computational block extracted from each hidden layer of the target supernetwork, and the initial population is composed of the sampled subnets. The second training unit includes: A parsing module is used to parse the computational blocks contained in each subnet from the initial population; The acquisition module is used to acquire the indicator data corresponding to the computing block in each subnet and to count the indicator data corresponding to each subnet based on the mapping relationship between the computing block identifier, accuracy indicator and latency indicator contained in the hardware indicator table. The processing module is used to input the indicator data corresponding to each subnet into the preset subnet scoring formula and output the score value corresponding to each subnet. The selection module is used to select the subnet as the target subnet if the score value corresponding to the subnet reaches the preset target score value. The second training unit further includes: The determination module is used to identify the initial population containing subnets as the first batch of subnets; The determining module is further configured to select the subnet with the highest score from the first batch of subnets as the basic subnet when no subnet in the first batch of subnets has a score that reaches the preset target score. An evolution module is used to derive a second batch of subnets based on the base subnet; The selection module is further configured to select the subnet as the target subnet when at least one subnet in the second batch of subnets has a score that reaches the preset target score. The first execution module is used to repeatedly execute the evolutionary derivation of the next batch of subnets when no subnet in the second batch of subnets has a score value that reaches the preset target score value, until at least one subnet in the specified batch of subnets has a score value that reaches the preset target score value, then select the subnet as the target subnet and stop executing the evolutionary derivation of the next batch of subnets. The second training unit further includes: The acquisition module is also used to acquire the current number of evolutionary derivations before repeatedly performing the evolutionary derivation of the next batch of subnets operation; The second execution module is used to select the subnet with the highest score from the batch of subnets corresponding to the current evolutionary derivation number as the target subnet when no subnet in the batch of subnets corresponding to the current evolutionary derivation number reaches the preset target score, but the current evolutionary derivation number reaches the preset evolutionary derivation number threshold, and then stop executing the evolutionary derivation of the next batch of subnets.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the hardware-aware, differentiable neural network structure search method as described in any one of claims 1-4.

Images